|
1．环境准备 a) 启动四台linux虚拟机 b) 安装JDK1.8 2. 下载并安装ES(x.x.x.130) 下载地址： curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.5.2.tar.gz 安装ES： tar -xvf elasticsearch-6.5.2.tar.gz 配置ES： cd elasticsearch-6.5.1/config/ vim elasticsearch.yml cluster.name: my-test-es #es集群的名称 path.data: /application/es/data #自定义数据存放的文件夹 path.logs: /application/es/logs #自定义日志存放的文件夹 network.host: x.x.x.130 #es所在主机的ip地址 http.port: 9200 #es的端口号 启动ES： cd elasticsearch-6.5.2/bin ./elasticsearch 3. 下载并安装kibana(x.x.x.129) 下载地址： https://www.elastic.co/downloads 选择kibana 下载 安装kibana： tar -xvf kibana-6.5.1-linux-x86_64.tar.gz 配置kibana server.port: 5601 #kibana的端口号 server.host: "x.x.x.129" #kibana 所在服务器的ip server.name: "centos001-kibana" #kibana 的名称 elasticsearch.url: http://x.x.x.130:9200 #es的地址 启动Kibana cd kibana-6.5.1-linux-x86_64/bin/ ./kibana 4. 下载安装logStash(x.x.x.131) 下载地址： https://www.elastic.co/downloads 选择logStash下载 安装logStash tar -xvf logstash-6.5.1.tar.gz 1 安装logStash插件 cd logstash-6.5.1 vim Gemfile 修改：source "https://ruby.taobao.org/"，替代source "https://rubygems.org" . /bin/logstash-plugin install --no-verify 配置logStash ------------------------使用logStash采集日志信息配置开始----------------------------------------- 新建配置文件test.conf cd config vim test.conf 输入以下内容： input{ file{ path => "/application/logstash/testLog/test.log" start_position => beginning } } filter{ grok{ match=>{ "message"=>"%{IP:client} %{WORD:method} %{URIPATHPARAM:reuqest} %{NUMBER:bytes} %{NUMBER:duration}" } } } output{ elasticsearch { hosts => ["http://x.x.x.130:9200"] index => "logstash-testlog-%{+YYYY.MM.dd}" #user => "elastic" #password => "changeme" } #stdout{codec=>rubydebug} } 新建test.log 55.3.244.1 GET /index.html 15824 0.043 ------------------------使用logStash采集日志信息配置结束----------------------------------------- ------------------------使用filebeat采集日志信息，logStash过滤日志信息配置开始-------------- 新建配置文件test.conf cd config vim test.conf 输入以下内容： input{ beats { port => 5044 } } filter{ grok{ match=>{ "message"=>"%{IP:client} %{WORD:method} %{URIPATHPARAM:reuqest} %{NUMBER:bytes} %{NUMBER:duration}" } } } output{ elasticsearch { hosts => ["http://x.x.x.130:9200"] index => "logstash-testlog-%{+YYYY.MM.dd}" #user => "elastic" #password => "changeme" } #stdout{codec=>rubydebug} } -------------------------使用filebeat采集日志信息，logStash过滤日志信息配---------------------- 启动logStash cd logstash-6.5.1/ ./bin/logstash -f config/test.conf 5. 下载安装filebeat 下载： curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.5.2-linux-x86_64.tar.gz 安装： tar xzvf filebeat-6.5.2-linux-x86_64.tar.gz 配置：修改文件 filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /var/log/test.log #- c:\programdata\elasticsearch\logs\* #-------------------------- Elasticsearch output ------------------------------ #output.elasticsearch: # Array of hosts to connect to. #hosts: ["localhost:9200"] # Optional protocol and basic auth credentials. #protocol: "https" #username: "elastic" #password: "changeme" #----------------------------- Logstash output -------------------------------- output.logstash: # The Logstash hosts hosts: ["x.x.x.131:5044"] # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key" 新建test.log 55.3.244.1 GET /index.html 15824 0.043 启动filebeat sudo chown root filebeat.yml sudo ./filebeat -e 6.filebeat读取多个log文件，采用不同的过滤方式，输出到es的不同index中 Filebeat中的配置，filebeat.yml修改为： filebeat.inputs: - type: log enabled: true paths: #- /var/log/log1.log fields: # level: debug # review: 1 log_source: log1 - type: log enabled: true paths: #- /var/log/log2.log fields: # level: debug # review: 1 log_source: log2 #-------------------------- Elasticsearch output ------------------------------ #output.elasticsearch: # Array of hosts to connect to. #hosts: ["localhost:9200"] # Optional protocol and basic auth credentials. #protocol: "https" #username: "elastic" #password: "changeme" #----------------------------- Logstash output -------------------------------- output.logstash: # The Logstash hosts hosts: ["192.168.6.131:5044"] # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key" logStash中的配置文件修改为： input{ beats { port => 5044 } } filter{ if [fields][log_source] == log1" { grok { match => { "message"=>""#这里写自己的处理 } } } if [fields][log_source] == "log2" { grok { match=>{ "message"=>""# 这里写自己的处理方式 } } } mutate { #这里删除一些信息 可选 remove_field =>["message"] remove_field =>["beat"] remove_field =>["host"] #remove_field =>["fields"] remove_field =>["input"] remove_field =>["prospector"] } } output{ if [fields][log_source] == log1" { elasticsearch { hosts => ["http://x.x.x.130:9200"] index => "log1" #user => "elastic" #password => "changeme" } } if [fields][log_source] == "log2" { elasticsearch { hosts => ["http://x.x.x.130:9200"] index => "log2" #user => "elastic" #password => "changeme" } } } (责任编辑：IT) |