elk7.5 之 filebeat部署
时间:2020-01-03 11:28 来源:csdn.net 作者:GeekXuShuo
原理篇请查看https://blog.csdn.net/xiaokangtongxue410/article/details/82838872
这里我就不copy了
部署、配置过程：
安装略。。。
配置目录放在：/etc/filebeat
$ cd /etc/filebeat
$ ll
-rw-r--r-- 1 root root 938 Oct 14 10:46 filebeat-htjfweb.yml
drwxr-xr-x 2 root root 4096 Nov 8 14:39 prospectors.d
$ cat filebeat-htjfweb.yml
path.config: /etc/filebeat
path.logs: /var/log/filebeat-kafka-htjfweb
path.data: /var/lib/filebeat/data-kafka-htjfweb
filebeat.registry_file: /var/lib/filebeat/registry-kafka-htjfweb
filebeat.shutdown_timeout: 0
logging.level: info
logging.metrics.enabled: false
logging.files.rotateeverybytes: 104857600
logging.files.keepfiles: 10
logging.files.permissions: 0600
setup.template.name: "filebeat"
setup.template.pattern: "filebeat-*"
filebeat.config:
prospectors:
enabled: true
path: ${path.config}/prospectors.d/product-htjfweb*.yml ##热加载配置
reload.enabled: true
reload.period: 10s
output.kafka:
enabled: true
hosts: ["10.100.202.177:9092","10.100.202.191:9092","10.100.202.192:9092"]
topic: logstash-htjfweb
partition.round_robin: # 集群是否开启kafka的partition分区
reachable_only: false
compression: gzip
max_message_bytes: 10000000
required_acks: 1
我的配置文件放在prospectors.d目录中
$ cat prospectors.d/product-htjfweb-account-web.yml
- type: log
enabled: true
paths:
- /data/WEBLOG/product-htjfweb-account-web/*/*.log
scan_frequency: 10s
fields_under_root: true
multiline.pattern: '^\d{4}-\d{1,2}-\d{1,2}'
multiline.negate: true
multiline.match: after
multiline.max_lines: 500
multiline.timeout: 5s
exclude_lines: ['WARN']
fields:
env: product
index: logstash-htjf
logtype: product-htjfweb-account-web
topic: logstash-htjf
k8s_host: node
tail_files: false
close_inactive: 2h
close_eof: false
close_removed: true
clean_removed: true
close_renamed: false
启动filebeat
$ /usr/local/bin/filebeat -c /etc/filebeat/filebeat-htjfweb.yml
如有多个配置文件 需要将以下配置变更成不同的文件，不然启动报错。
path.config: /etc/filebeat
path.logs: /var/log/filebeat-kafka-htjfweb
path.data: /var/lib/filebeat/data-kafka-htjfweb
filebeat.registry_file: /var/lib/filebeat/registry-kafka-htjfweb
通过查看kafka 的topic里是否有数据来判断收集是否成功。
$ cd /usr/local/kafka/bin/
$ ./kafka-console-consumer.sh --bootstrap-server 10.100.202.191:9092 --topic logstash-htjfweb --from-beginning ##从头查看收集到的数据
$ ./kafka-consumer-groups.sh --group logstash-htjfweb --describe --bootstrap-server 10.100.202.177:9092 ##查看消费情况
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
logstash-htjfweb logstash-htjfweb 0 13965817 13965822 5 logstash-0-e16abafa-1976-4677-a520-ec95c9ce38b8 /10.100.202.47 logstash-0
logstash-htjfweb logstash-htjfweb 1 13965595 13965599 4 logstash-0-e16abafa-1976-4677-a520-ec95c9ce38b8 /10.100.202.47 logstash-0
logstash-htjfweb logstash-htjfweb 2 13965756 13965761 5 logstash-0-e16abafa-1976-4677-a520-ec95c9ce38b8 /10.100.202.47 logstash-0
(责任编辑：IT)
| 原理篇请查看https://blog.csdn.net/xiaokangtongxue410/article/details/82838872 这里我就不copy了 部署、配置过程： 安装略。。。 配置目录放在：/etc/filebeat $ cd /etc/filebeat $ ll -rw-r--r-- 1 root root 938 Oct 14 10:46 filebeat-htjfweb.yml drwxr-xr-x 2 root root 4096 Nov 8 14:39 prospectors.d $ cat filebeat-htjfweb.yml path.config: /etc/filebeat path.logs: /var/log/filebeat-kafka-htjfweb path.data: /var/lib/filebeat/data-kafka-htjfweb filebeat.registry_file: /var/lib/filebeat/registry-kafka-htjfweb filebeat.shutdown_timeout: 0 logging.level: info logging.metrics.enabled: false logging.files.rotateeverybytes: 104857600 logging.files.keepfiles: 10 logging.files.permissions: 0600 setup.template.name: "filebeat" setup.template.pattern: "filebeat-*" filebeat.config: prospectors: enabled: true path: ${path.config}/prospectors.d/product-htjfweb*.yml ##热加载配置 reload.enabled: true reload.period: 10s output.kafka: enabled: true hosts: ["10.100.202.177:9092","10.100.202.191:9092","10.100.202.192:9092"] topic: logstash-htjfweb partition.round_robin: # 集群是否开启kafka的partition分区 reachable_only: false compression: gzip max_message_bytes: 10000000 required_acks: 1 我的配置文件放在prospectors.d目录中 $ cat prospectors.d/product-htjfweb-account-web.yml - type: log enabled: true paths: - /data/WEBLOG/product-htjfweb-account-web/*/*.log scan_frequency: 10s fields_under_root: true multiline.pattern: '^\d{4}-\d{1,2}-\d{1,2}' multiline.negate: true multiline.match: after multiline.max_lines: 500 multiline.timeout: 5s exclude_lines: ['WARN'] fields: env: product index: logstash-htjf logtype: product-htjfweb-account-web topic: logstash-htjf k8s_host: node tail_files: false close_inactive: 2h close_eof: false close_removed: true clean_removed: true close_renamed: false 启动filebeat $ /usr/local/bin/filebeat -c /etc/filebeat/filebeat-htjfweb.yml 如有多个配置文件 需要将以下配置变更成不同的文件，不然启动报错。 path.config: /etc/filebeat path.logs: /var/log/filebeat-kafka-htjfweb path.data: /var/lib/filebeat/data-kafka-htjfweb filebeat.registry_file: /var/lib/filebeat/registry-kafka-htjfweb 通过查看kafka 的topic里是否有数据来判断收集是否成功。 $ cd /usr/local/kafka/bin/ $ ./kafka-console-consumer.sh --bootstrap-server 10.100.202.191:9092 --topic logstash-htjfweb --from-beginning ##从头查看收集到的数据 $ ./kafka-consumer-groups.sh --group logstash-htjfweb --describe --bootstrap-server 10.100.202.177:9092 ##查看消费情况 GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID logstash-htjfweb logstash-htjfweb 0 13965817 13965822 5 logstash-0-e16abafa-1976-4677-a520-ec95c9ce38b8 /10.100.202.47 logstash-0 logstash-htjfweb logstash-htjfweb 1 13965595 13965599 4 logstash-0-e16abafa-1976-4677-a520-ec95c9ce38b8 /10.100.202.47 logstash-0 logstash-htjfweb logstash-htjfweb 2 13965756 13965761 5 logstash-0-e16abafa-1976-4677-a520-ec95c9ce38b8 /10.100.202.47 logstash-0 (责任编辑：IT) |