当前位置: > 其它学习 > Elasticsearch >

ELK 日志系统搭建--监控nginx

时间:2020-01-03 10:55来源:linux.it.net.cn 作者:IT
logstash安装
下载路径: https://www.elastic.co/downloads/logstash(安装方法参考官网安装步骤)
要读取nginx日志,配置nginx日志格式
vim nginx.conf
修改nginx记录日志格式,从http模块下
log_format  main  '$remote_addr | $time_local | $request | $uri | '
                      '$status | $body_bytes_sent | $bytes_sent | $gzip_ratio | $http_referer | '
                      '"$http_user_agent" | $http_x_forwarded_for | $upstream_addr | $upstream_response_time | $upstream_status | $request_time';
修改完成后保存,使用./nginx -s reload 重新加载

/etc/logstash/conf.d下创建nginx日志配置文件
touch nginx_access.conf
sudo vim nginx_access.conf
input {
    file {
        path => [ "/usr/local/nginx/logs/adsapi.access.log" ]
         type => "nginx_access"
    }
}
filter {
   grok {
    match => [
                    "message", "%{IPORHOST:clientip} \| %{HTTPDATE:timestamp} \| (?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:http_version})?|-) \| %{URIPATH:uripath} \| %{NUMBER:response} \| (?:%{NUMBER:body_bytes_sent}|-) \| (?:%{NUMBER:bytes_sent}|-) \| (?:%{NOTSPACE:gzip_ratio}|-) \| (?:%{QS:http_referer}|-) \| %{QS:user_agent} \| (?:%{QS:http_x_forwarded_for}|-) \| (%{URIHOST:upstream_addr}|-) \| (%{BASE16FLOAT:upstream_response_time}) \| %{NUMBER:upstream_status} \| (%{BASE16FLOAT:request_time})"
                ]
        }
geoip {
      source => "clientip"
      target => "geoip"
      add_field => [ "[geoip][coordinates]","%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]","%{[geoip][latitude]}" ]
    }
    mutate {
     convert => [ "[geoip][coordinates]","float" ]
    }

    date {
      match => [ "timestamp","dd/MMM/yyyy:HH:mm:ss Z"]

    }
    mutate {
      remove_field => "timestamp"

    }
}


output {
    elasticsearch {
        hosts => ["127.0.*.*:9200"]
        index => "logstash-nginx-access-%{+YYYY.MM.dd}"
        user => "****" //下文安装kibana会设置
                password => "pwd"
    }
    stdout {

       }
}



2 . elasticsearch安装

下载地址:https://www.elastic.co/downloads/elasticsearch,安装步骤参见官网
安装完成后,从etc/elasticsearch/ 目录下
vim elasticsearch.yml

cluster.name: elk
node.name: es2
path.data: /data/elasticsearch(存储目录一定要给elasticsearch账户授权)
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.host: *.*.*.*(服务器ip)
http.port: 9200
启动服务sudo service elasticsearch start
查看启动日志,或直接查看启动后的进程状态是否成功
elasticsearch (pid 19206) is running.
浏览器输入:http://ip地址:9200/,给出响应结果

3 . kibana安装

下载地址:https://www.elastic.co/downloads/kibana
安装x-pack ,下载地址:https://www.elastic.co/downloads/x-pack
自己安装的在/user/share/ 目录下
从etc/kibana/ 目录下修改kibana.yml文件

sudo vim kibana.yml server.name: "*.*.*.*"// (服务器ip地址)
elasticsearch.url: "http://*.*.*.*:9200"
elasticsearch.username: "username"
elasticsearch.password: "pwd"
增加:
tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&z={z}'
配置完成后,三个服务依次启动 elasticsearch–kibana –logstash
service elasticsearch start
service kibana start
initctl start logstash


4. 要外网访问需要配置nginx.conf,访问地址到kibana

upstream elk {
    ip_hash;
    server 127.0.0.1:5601;
}

server {
    listen 80;
    server_name 域名;
    server_tokens off;

    client_body_timeout 5s;
    client_header_timeout 5s;

    location / {
        proxy_pass http://elk/;
        index index.html index.htm;
     proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Host $http_host;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header X-Real-IP $remote_addr;
    }
}
5.  配置完成后,重新加载nginx,浏览器输入域名,填写安装x-pack的用户名和密码
6.  登录成功后,![这里写图片描述](http://img.blog.csdn.net/20171201115255853?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvcWluZ3RpYW4yMDAy/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast)
7.  在Configure an index pattern功能下配置:logstash-nginx-access*
![这里写图片描述](http://img.blog.csdn.net/20171201115318889?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvcWluZ3RpYW4yMDAy/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast)
    8.创建成功后,选择discover模块就能查看到 (责任编辑:IT)
------分隔线----------------------------