日志监控和分析在保障业务稳定运行时,起到了很重要的作用,不过一般情况下日志都分散在各个生产服务器,且开发人员无法登陆生产服务器,这时候就需要一个集中式的日志收集装置,对日志中的关键字进行监控,触发异常时进行报警,并且开发人员能够查看相关日志。logstash+elasticsearch+kibana3就是实现这样功能的一套系统,并且功能更强大。
Logstash:负责日志的收集,处理和储存 1、环境介绍
elkServer elkClient
IP:192.168.31.23 2、下载准备 官网下载最新的安装包:https://www.elastic.co/downloads elasticsearch-1.7.3.noarch.rpm (server上安装) kibana-4.1.2-linux-x64.tar.gz (server上安装) logstash-1.5.4-1.noarch.rpm (server上安装) logstash-forwarder-0.4.0-1.x86_64.rpm (client上安装)
3.1安装jdk1.7 [root@localhost ~]# yum install java-1.7.0-openjdk Loaded plugins: fastestmirror, langpacks base | 3.6 kB 00:00:00 extras | 3.4 kB 00:00:00 updates | 3.4 kB 00:00:00 Loading mirror speeds from cached hostfile * base: mirrors.btte.net * extras: mirrors.163.com * updates: mirrors.163.com Package 1:java-1.7.0-openjdk-1.7.0.91-2.6.2.1.el7_1.x86_64 already installed and latest version Nothing to do
3.2安装elasticsearch [root@localhost elk]# yum localinstall elasticsearch-1.7.3.noarch.rpm (yum 本地安装elasticsearch) Loaded plugins: fastestmirror, langpacks Examining elasticsearch-1.7.3.noarch.rpm: elasticsearch-1.7.3-1.noarch elasticsearch-1.7.3.noarch.rpm: does not update installed package. Nothing to do [root@localhost elk]# systemctl daemon-reload [root@localhost elk]# systemctl enable elasticsearch.service (设置开机自启动) ln -s '/usr/lib/systemd/system/elasticsearch.service' '/etc/systemd/system/multi-user.target.wants/elasticsearch.service' [root@localhost elk]# systemctl start elasticsearch.service (开启服务) [root@localhost elk]# systemctl status elasticsearch.service (查看服务状态) elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled) Active: active (running) since Sun 2015-11-08 11:05:09 CST; 28s ago Docs: http://www.elastic.co Main PID: 15345 (java) CGroup: /system.slice/elasticsearch.service ?..15345 java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+Heap... Nov 08 11:05:09 localhost.localdomain systemd[1]: Started Elasticsearch. [root@localhost elk]# rpm -qc elasticsearch /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/logging.yml /etc/init.d/elasticsearch /etc/sysconfig/elasticsearch /usr/lib/sysctl.d/elasticsearch.conf /usr/lib/systemd/system/elasticsearch.service /usr/lib/tmpfiles.d/elasticsearch.conf [root@localhost elk]# netstat -nltp (查看端口监听状况) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 784/rpcbind tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1457/sshd tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 3213/cupsd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2656/master tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 14407/sshd: root@pt tcp6 0 0 :::111 :::* LISTEN 784/rpcbind tcp6 0 0 :::9200 :::* LISTEN 15345/java tcp6 0 0 :::9300 :::* LISTEN 15345/java tcp6 0 0 :::22 :::* LISTEN 1457/sshd tcp6 0 0 ::1:631 :::* LISTEN 3213/cupsd tcp6 0 0 ::1:25 :::* LISTEN 2656/master tcp6 0 0 ::1:6010 :::* LISTEN 14407/sshd: root@pt [root@localhost elk]# firewall-cmd --permanent --add-port={9200/tcp,9300/tcp} (防火墙添加两个端口) success [root@localhost elk]# firewall-cmd --reload (重载防火墙) success [root@localhost elk]# firewall-cmd --list-all (查看防火墙开发端口) public (default, active) interfaces: ens33 sources: services: dhcpv6-client ssh ports: 9200/tcp 9300/tcp masquerade: no forward-ports: icmp-blocks: rich rules:
[root@localhost elk]# tar zxf kibana-4.1.2-linux-x64.tar.gz -C /usr/local/ (解压缩安装包到指定目录中) [root@localhost elk]# cd /usr/local/ [root@localhost local]# ls bin etc games include kibana-4.1.2-linux-x64 lib lib64 libexec sbin share src [root@localhost local]# mv kibana-4.1.2-linux-x64/ kibana (重命名) [root@localhost local]# cd kibana/ [root@localhost kibana]# ls bin config LICENSE.txt node plugins README.txt src [root@localhost kibana]# cd bin/ [root@localhost bin]# ls (运行./kibana即可开启服务,但我们将其做到service) kibana kibana.bat [root@localhost bin]# cd /etc/systemd/system/ [root@localhost system]# vi kibana.service (编辑kibana服务) [Service] ExecStart=/usr/local/kibana/bin/kibana [Install] WantedBy=multi-user.target [root@localhost system]# systemctl enable kibana.service (设置开机自启动) ln -s '/etc/systemd/system/kibana.service' '/etc/systemd/system/multi-user.target.wants/kibana.service' [root@localhost system]# systemctl start kibana.service (开启服务) [root@localhost system]# systemctl status kibana.service (查看服务运行状态) kibana.service Loaded: loaded (/etc/systemd/system/kibana.service; enabled) Active: active (running) since Sun 2015-11-08 11:16:28 CST; 10s ago Main PID: 16131 (node) CGroup: /system.slice/kibana.service ?..16131 /usr/local/kibana/bin/../node/bin/node /usr/local/kibana/bin/../src/bin/kibana.js Nov 08 11:16:28 localhost.localdomain systemd[1]: Started kibana.service. Nov 08 11:16:34 localhost.localdomain kibana[16131]: {"name":"Kibana","hostname":"localhost.localdomain","pid":16131,"level":30,"msg":"No existing kibana index found","time":"20...43Z","v":0} Nov 08 11:16:34 localhost.localdomain kibana[16131]: {"name":"Kibana","hostname":"localhost.localdomain","pid":16131,"level":30,"msg":"Listening on 0.0.0.0:5601","time":"2015-11...93Z","v":0} Hint: Some lines were ellipsized, use -l to show in full. [root@localhost system]# netstat -nltp (查看端口监听状态) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 16131/node tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 784/rpcbind tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1457/sshd tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 3213/cupsd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2656/master tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 14407/sshd: root@pt tcp6 0 0 :::111 :::* LISTEN 784/rpcbind tcp6 0 0 :::9200 :::* LISTEN 15345/java tcp6 0 0 :::9300 :::* LISTEN 15345/java tcp6 0 0 :::22 :::* LISTEN 1457/sshd tcp6 0 0 ::1:631 :::* LISTEN 3213/cupsd tcp6 0 0 ::1:25 :::* LISTEN 2656/master tcp6 0 0 ::1:6010 :::* LISTEN 14407/sshd: root@pt [root@localhost system]# firewall-cmd --permanent --add-port=5601/tcp (防火墙开启5601端口) success [root@localhost system]# firewall-cmd --reload (重载防火墙) success [root@localhost system]# firewall-cmd --list-all (查看防火墙开放端口) public (default, active) interfaces: ens33 sources: services: dhcpv6-client ssh ports: 9200/tcp 9300/tcp 5601/tcp masquerade: no forward-ports: icmp-blocks: rich rules: [root@localhost system]# firewall-cmd --permanent --add-forward-port=port=80:proto=tcp:toport=5601 (为5601端口添加80端口的映射,这样在浏览器中就可以不用输入端口了) success [root@localhost system]# firewall-cmd --reload (重载防火墙) success [root@localhost system]# firewall-cmd --list-all (查看防火墙开放端口) public (default, active) interfaces: ens33 sources: services: dhcpv6-client ssh ports: 9200/tcp 9300/tcp 5601/tcp masquerade: no forward-ports: port=80:proto=tcp:toport=5601:toaddr= icmp-blocks: rich rules: 3.4安装logstash [root@localhost system]# cd /home/elk/ [root@localhost elk]# ls elasticsearch-1.7.3.noarch.rpm kibana-4.1.2-linux-x64.tar.gz logstash-1.5.4-1.noarch.rpm logstash-forwarder-0.4.0-1.x86_64.rpm [root@localhost elk]# yum localinstall logstash-1.5.4-1.noarch.rpm (yum本地安装logstash) Loaded plugins: fastestmirror, langpacks Examining logstash-1.5.4-1.noarch.rpm: 1:logstash-1.5.4-1.noarch Marking logstash-1.5.4-1.noarch.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package logstash.noarch 1:1.5.4-1 will be installed --> Finished Dependency Resolution base/7/x86_64 | 3.6 kB 00:00:00 extras/7/x86_64 | 3.4 kB 00:00:00 extras/7/x86_64/primary_db | 116 kB 00:00:00 updates/7/x86_64 | 3.4 kB 00:00:00 updates/7/x86_64/primary_db | 4.7 MB 00:00:03 Dependencies Resolved =============================================================================================================================================================================================== Package Arch Version Repository Size =============================================================================================================================================================================================== Installing: logstash noarch 1:1.5.4-1 /logstash-1.5.4-1.noarch 136 M Transaction Summary =============================================================================================================================================================================================== Install 1 Package Total size: 136 M Installed size: 136 M Is this ok [y/d/N]: y Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : 1:logstash-1.5.4-1.noarch 1/1 Verifying : 1:logstash-1.5.4-1.noarch 1/1 Installed: logstash.noarch 1:1.5.4-1 Complete! [root@localhost tls]# hostname -f (查看当前FQDN,FQDN设置参见http://www.cnblogs.com/zhenyuyaodidiao/p/4947930.html) elk.server.com [root@localhost ~]# cd /etc/pki/tls/ (进入到/etc/pki/tls/文件夹) [root@localhost tls]# ls cert.pem certs misc openssl.cnf private (以下生成openssl key用于客户端上传日志文件用,在客户端配置时会用到) [root@localhost tls]# openssl req -subj '/CN=elk.server.com/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt Generating a 2048 bit RSA private key ..............+++ .............+++ writing new private key to 'private/logstash-forwarder.key' ----- [root@localhost tls]# ls cert.pem certs misc openssl.cnf private [root@localhost tls]# cd private/ [root@localhost private]# ll total 4 -rw-r--r--. 1 root root 1704 Nov 8 17:20 logstash-forwarder.key [root@localhost private]# cd ../certs/ [root@localhost certs]# ll total 16 lrwxrwxrwx. 1 root root 49 Apr 14 2015 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem lrwxrwxrwx. 1 root root 55 Apr 14 2015 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt -rw-r--r--. 1 root root 1107 Nov 8 17:20 logstash-forwarder.crt -rwxr-xr-x. 1 root root 610 Mar 24 2015 make-dummy-cert -rw-r--r--. 1 root root 2388 Mar 24 2015 Makefile -rwxr-xr-x. 1 root root 829 Mar 24 2015 renew-dummy-cert [root@localhost ~]# cd /etc/logstash/conf.d/ [root@localhost conf.d]# vi 01-logstash-initial.conf (编辑logstash配置文件) input { lumberjack { port => 5000 type => "logs" ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" } } filter { if [type] == "syslog" { grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" } add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } syslog_pri { } date { match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] } } } output { elasticsearch { host => localhost } stdout { codec => rubydebug } } [root@localhost conf.d]# systemctl enable logstash (设置开机自启动) logstash.service is not a native service, redirecting to /sbin/chkconfig. Executing /sbin/chkconfig logstash on The unit files have no [Install] section. They are not meant to be enabled using systemctl. Possible reasons for having this kind of units are: 1) A unit may be statically enabled by being symlinked from another unit's .wants/ or .requires/ directory. 2) A unit's purpose may be to act as a helper for some other unit which has a requirement dependency on it. 3) A unit may be started when needed via activation (socket, path, timer, D-Bus, udev, scripted systemctl call, ...). [root@localhost conf.d]# systemctl start logstash.service (开启logstash服务) [root@localhost conf.d]# systemctl status logstash.service (查看服务运行状态) logstash.service - LSB: Starts Logstash as a daemon. Loaded: loaded (/etc/rc.d/init.d/logstash) Active: active (running) since Sun 2015-11-08 17:28:34 CST; 14s ago Process: 20799 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS) CGroup: /system.slice/logstash.service ?..20805 java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib... Nov 08 17:28:34 elk logstash[20799]: logstash started. Nov 08 17:28:34 elk systemd[1]: Started LSB: Starts Logstash as a daemon.. [root@localhost conf.d]# netstat -nltp (查看端口占用) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 16131/node tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 784/rpcbind tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1457/sshd tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 3213/cupsd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2656/master tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 14407/sshd: root@pt tcp 0 0 127.0.0.1:6012 0.0.0.0:* LISTEN 17715/sshd: root@pt tcp6 0 0 :::5000 :::* LISTEN 20805/java tcp6 0 0 :::111 :::* LISTEN 784/rpcbind tcp6 0 0 :::9200 :::* LISTEN 15345/java tcp6 0 0 :::9300 :::* LISTEN 15345/java tcp6 0 0 :::9301 :::* LISTEN 20805/java tcp6 0 0 :::22 :::* LISTEN 1457/sshd tcp6 0 0 ::1:631 :::* LISTEN 3213/cupsd tcp6 0 0 ::1:25 :::* LISTEN 2656/master tcp6 0 0 ::1:6010 :::* LISTEN 14407/sshd: root@pt tcp6 0 0 ::1:6012 :::* LISTEN 17715/sshd: root@pt [root@localhost conf.d]# cd /var/log/logstash/ [root@localhost logstash]# ls (日志文件) logstash.err logstash.log logstash.stdout [root@localhost logstash]# firewall-cmd --permanent --add-port=5000/tcp (防火墙开放5000端口) success [root@localhost logstash]# firewall-cmd --reload (重载防火墙) success [root@localhost logstash]# firewall-cmd --list-all (查看端口开放情况) public (default, active) interfaces: ens33 sources: services: dhcpv6-client ssh ports: 9200/tcp 9300/tcp 5000/tcp 5601/tcp masquerade: no forward-ports: port=80:proto=tcp:toport=5601:toaddr= icmp-blocks: rich rules: 4、Client端安装 [root@localhost elk]# vi /etc/hosts (编辑hosts文件) 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.7.27 elk.server.com [root@localhost elk]# service network restart Restarting network (via systemctl): [ OK ] [root@localhost elk]# ping elk.server.com (测试连接) PING elk.server.com (192.168.7.27) 56(84) bytes of data. 64 bytes from elk.server.com (192.168.7.27): icmp_seq=1 ttl=63 time=0.754 ms 64 bytes from elk.server.com (192.168.7.27): icmp_seq=2 ttl=63 time=0.477 ms ^C --- elk.server.com ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1000ms rtt min/avg/max/mdev = 0.477/0.615/0.754/0.140 ms [root@localhost laizy]# mkdir elk [root@localhost laizy]# cd elk/ [root@localhost elk]# ls [root@localhost elk]# scp root@192.168.7.27:/home/elk/logstash-forwarder-0.4.0-1.x86_64.rpm . (拷贝logstash-forwarder到本地) The authenticity of host '192.168.7.27 (192.168.7.27)' can't be established. ECDSA key fingerprint is 49:b9:53:89:55:f2:93:87:9b:81:bb:23:a5:24:f1:f9. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.7.27' (ECDSA) to the list of known hosts. root@192.168.7.27's password: logstash-forwarder-0.4.0-1.x86_64.rpm 100% 1692KB 1.7MB/s 00:00 [root@localhost elk]# ls logstash-forwarder-0.4.0-1.x86_64.rpm [root@localhost elk]# scp root@192.168.7.27:/etc/pki/tls/certs/logstash-forwarder.crt . (拷贝Server端的key到本地) root@192.168.7.27's password: logstash-forwarder.crt 100% 1107 1.1KB/s 00:00 [root@localhost elk]# ll total 1700 -rw-r--r--. 1 root root 1732758 Nov 8 17:36 logstash-forwarder-0.4.0-1.x86_64.rpm -rw-r--r--. 1 root root 1107 Nov 8 17:37 logstash-forwarder.crt [root@localhost elk]# cp logstash-forwarder.crt /etc/pki/tls/certs/ (将key拷贝到/etc/pki/tls/certs/下) [root@localhost elk]# cd /etc/pki/tls/certs/ [root@localhost certs]# ls ca-bundle.crt ca-bundle.trust.crt logstash-forwarder.crt make-dummy-cert Makefile renew-dummy-cert [root@localhost certs]# cd /home/laizy/elk/ [root@localhost elk]# ls logstash-forwarder-0.4.0-1.x86_64.rpm logstash-forwarder.crt [root@localhost elk]# yum localinstall logstash-forwarder-0.4.0-1.x86_64.rpm (yum本地安装logstash-forwarder) Loaded plugins: fastestmirror, langpacks Examining logstash-forwarder-0.4.0-1.x86_64.rpm: logstash-forwarder-0.4.0-1.x86_64 Marking logstash-forwarder-0.4.0-1.x86_64.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package logstash-forwarder.x86_64 0:0.4.0-1 will be installed --> Finished Dependency Resolution base/7/x86_64 | 3.6 kB 00:00:00 extras/7/x86_64 | 3.4 kB 00:00:00 updates/7/x86_64 | 3.4 kB 00:00:00 Dependencies Resolved =============================================================================================================================================================================================== Package Arch Version Repository Size =============================================================================================================================================================================================== Installing: logstash-forwarder x86_64 0.4.0-1 /logstash-forwarder-0.4.0-1.x86_64 5.7 M Transaction Summary =============================================================================================================================================================================================== Install 1 Package Total size: 5.7 M Installed size: 5.7 M Is this ok [y/d/N]: y Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : logstash-forwarder-0.4.0-1.x86_64 1/1 Logs for logstash-forwarder will be in /var/log/logstash-forwarder/ Verifying : logstash-forwarder-0.4.0-1.x86_64 1/1 Installed: logstash-forwarder.x86_64 0:0.4.0-1 Complete! [root@localhost elk]# systemctl enable logstash-forwarder (设置开机自启动) logstash-forwarder.service is not a native service, redirecting to /sbin/chkconfig. Executing /sbin/chkconfig logstash-forwarder on The unit files have no [Install] section. They are not meant to be enabled using systemctl. Possible reasons for having this kind of units are: 1) A unit may be statically enabled by being symlinked from another unit's .wants/ or .requires/ directory. 2) A unit's purpose may be to act as a helper for some other unit which has a requirement dependency on it. 3) A unit may be started when needed via activation (socket, path, timer, D-Bus, udev, scripted systemctl call, ...). [root@localhost elk]# systemctl start logstash-forwarder.service (开启服务) [root@localhost elk]# cd /var/log/logstash-forwarder/ (日志目录) [root@localhost logstash-forwarder]# ls logstash-forwarder.err logstash-forwarder.log [root@localhost elk]# vi /etc/logstash-forwarder.conf (编辑配置文件) { "network": { "servers": [ "elk.server.com:5000" ], "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt", "timeout": 15 }, "files": [ { "paths": [ "/var/log/messages", "/var/log/secure" ], "fields": { "type": "syslog" } } ] } [root@localhost elk]# systemctl restart logstash-forwarder.service (重启服务) [root@localhost elk]# systemctl status logstash-forwarder.service (查看服务运行状态) logstash-forwarder.service - LSB: no description given Loaded: loaded (/etc/rc.d/init.d/logstash-forwarder) Active: active (running) since Sun 2015-11-08 18:30:51 CST; 18s ago Process: 10788 ExecStop=/etc/rc.d/init.d/logstash-forwarder stop (code=exited, status=0/SUCCESS) Process: 10794 ExecStart=/etc/rc.d/init.d/logstash-forwarder start (code=exited, status=0/SUCCESS) CGroup: /system.slice/logstash-forwarder.service ?..10798 /opt/logstash-forwarder/bin/logstash-forwarder -config /etc/logstash-forwarder.conf Nov 08 18:30:51 localhost.localdomain systemd[1]: Starting LSB: no description given... Nov 08 18:30:51 localhost.localdomain /etc/init.d/logstash-forwarder[10799]: logstash-forwarder started Nov 08 18:30:51 localhost.localdomain logstash-forwarder[10794]: logstash-forwarder started Nov 08 18:30:51 localhost.localdomain systemd[1]: Started LSB: no description given. 5、界面验证 首先在client中手动增加一条日志: [root@localhost elk]# logger zhenyuLogtest 界面登录 http://192.168.7.27/ ,做如下操作
从图中可以看到,手动添加的日志已经在界面中被搜索到了。
(责任编辑:IT) |