**环境配置** |服务器| IP地址| | ------------ | ------------ | |Logstash+Nginx服务器|10.60.60.60| |Elasticsearch服务器|10.60.60.9| |Kibana服务器|10.60.60.27| ------------ ------------ # #一、Logstash概述 Logstash是一个用来搜集、分析、过滤日志的工具。它支持几乎任何类型的日志,包括系统日志、错误日志和自定义应用程序日志。它可以从许多来源接收日志,这些来源包括 syslog、消息传递(例如 RabbitMQ)和JMX,它能够以多种方式输出数据,包括电子邮件、websockets和Elasticsearch。 ![Logstash概述.png](https://www.xiaoleizhang.com/usr/uploads/2022/01/1694532525.png) # 二、Logstash工作原理 Logstash使用管道方式进行日志的搜集处理和输出。 Logstash需要安装在有日志的服务器上。 有点类似Linux系统的管道命令 aaa| bbb | ccc,aaa执行完了会执行bbb,然后执行ccc。 在logstash中,包括了三个阶段: 输入input --> 处理filter(不是必须的) --> 输出output Inputs:通过配置input使其获取数据,常用的有:file、syslog(port:514)、redis等。 Filters:筛选器是Logstash管道中的中间处理层。 如果事件符合特定条件,则可以将过滤器与条件语句结合使用以对事件执行操作。 一些有用的过滤器包括: ```shell grok: parse and structure arbitrary text. Grok is currently the best way in Logstash to parse unstructured log data into something structured and queryable. With 120 patterns built-in to Logstash, it’s more than likely you’ll find one that meets your needs! mutate: perform general transformations on event fields. You can rename, remove, replace, and modify fields in your events. drop: drop an event completely, for example, debug events. clone: make a copy of an event, possibly adding or removing fields. geoip: add information about geographical location of IP addresses (also displays amazing charts in Kibana!) ``` Outputs:输出是Logstash管道的最后阶段。 一个事件可以通过多个输出,但是一旦完成所有输出处理,该事件就完成了执行。 一些常用的输出包括: ```shell elasticsearch: send event data to Elasticsearch. If you’re planning to save your data in an efficient, convenient, and easily queryable format… Elasticsearch is the way to go. Period. Yes, we’re biased :) file: write event data to a file on disk. graphite: send event data to graphite, a popular open source tool for storing and graphing metrics. http://graphite.readthedocs.io/en/latest/ statsd: send event data to statsd, a service that "listens for statistics, like counters and timers, sent over UDP and sends aggregates to one or more pluggable backend services". If you’re already using statsd, this could be useful for you! ``` 配置文件也是按这个顺序进行配置的。 ![Logstash工作原理.png](https://www.xiaoleizhang.com/usr/uploads/2022/01/3442466104.png) # 三、安装Logstash ## 1、Logstash下载 可以去官网下载Logstash,下载后上传至服务器内 下载链接:https://www.elastic.co/cn/downloads/logstash ![Logstash官网下载.png](https://www.xiaoleizhang.com/usr/uploads/2022/01/1188851901.png) ```shell [root@10-60-60-60 ~]# rz [root@10-60-60-60 ~]# ll total 357056 -rw-r--r-- 1 root root 365618045 Jan 4 10:50 logstash-7.16.2-linux-x86_64.tar.gz [root@10-60-60-60 ~]# ``` ## 2、解压并重命名 ```shell tar -zxvf logstash-7.16.2-linux-x86_64.tar.gz mv /root/logstash-7.16.2 /usr/local/logstash ``` **需要注意的是,Logstash最低需要jdk1.8的支持** 7.8,最低java8,支持java11 java14 6.8,最低jdk8,支持jdk11 6.0,最低jdk8,支持jdk9 5.0,最低jdk8,支持jdk9 官网下载压缩包内已经有了jdk ```shell [root@10-60-60-60 ~]# /usr/local/logstash/jdk/bin/java --version openjdk 11.0.13 2021-10-19 OpenJDK Runtime Environment Temurin-11.0.13+8 (build 11.0.13+8) OpenJDK 64-Bit Server VM Temurin-11.0.13+8 (build 11.0.13+8, mixed mode) [root@10-60-60-60 ~]# ``` ## 3、修改配置文件 复制配置文件 ```shell cp /usr/local/logstash/config/logstash-sample.conf /usr/local/logstash/config/syslog.conf ``` 修改配置文件 ```shell vim /usr/local/logstash/config/syslog.conf ``` 修改成内容如下 ```shell input # 定义日志源 { syslog { type => "system-syslog" # 定义类型 port => 10514 # 定义监听端口 } } output # 定义日志输出 { stdout { codec => rubydebug # 将日志输出到当前的终端上显示 } } ``` ## 4、验证配置文件 验证命令如下 ```shell /usr/local/logstash/bin/logstash --path.settings /usr/local/logstash/config/ -f /usr/local/logstash/config/syslog.conf --config.test_and_exit ``` **命令说明:** - --path.settings 用于指定logstash的配置文件所在的目录 - -f 指定需要被检测的配置文件的路径 - --config.test_and_exit 指定检测完之后就退出,不然就会直接启动了 **正确输出如下:** ```shell [root@10-60-60-60 ~]# /usr/local/logstash/bin/logstash --path.settings /usr/local/logstash/config/ -f /usr/local/logstash/config/syslog.conf --config.test_and_exit Using bundled JDK: /usr/local/logstash/jdk OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. Sending Logstash logs to /usr/local/logstash/logs which is now configured via log4j2.properties [2022-01-04T15:02:57,492][INFO ][logstash.runner ] Log4j configuration path used is: /usr/local/logstash/config/log4j2.properties [2022-01-04T15:02:57,500][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.16.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"} [2022-01-04T15:02:57,777][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2022-01-04T15:02:58,236][INFO ][org.reflections.Reflections] Reflections took 71 ms to scan 1 urls, producing 119 keys and 417 values Configuration OK [2022-01-04T15:02:59,152][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash [root@10-60-60-60 ~]# ``` 看到 Configuration OK 可以知道我们的配置没有问题。 ## 5、设置数据源日志输出 配置服务器的ip以及配置的监听端口 ```shell vim /etc/rsyslog.conf ``` 在末尾按照如下增加自己服务器IP ```shell *.* @@10.60.60.60:10514 ``` 重启rsyslog,让配置生效 ```shell systemctl restart rsyslog.service ``` ## 6、启动Logstash 指定配置文件,启动Logstash ```shell /usr/local/logstash/bin/logstash --path.settings /usr/local/logstash/config/ -f /usr/local/logstash/config/syslog.conf ``` 打开新终端检查一下10514端口是否已被监听: ```shell [root@10-60-60-60 ~]# netstat -tuplan |grep 10514 tcp 0 0 10.60.60.60:54522 10.60.60.60:10514 ESTABLISHED 6675/rsyslogd tcp6 0 0 :::10514 :::* LISTEN 6685/java tcp6 0 0 10.60.60.60:10514 10.60.60.60:54522 ESTABLISHED 6685/java udp 0 0 0.0.0.0:10514 0.0.0.0:* 6685/java [root@10-60-60-60 ~]# ``` 然后在别的机器ssh登录到这台机器上,测试一下有没有日志输出: ```shell { "priority" => 38, "message" => "New session 78 of user root.\n", "timestamp" => "Jan 4 16:06:10", "@timestamp" => 2022-01-04T08:06:10.000Z, "facility" => 4, "type" => "system-syslog", "@version" => "1", "host" => "10.60.60.60", "logsource" => "10-60-60-60", "severity" => 6, "program" => "systemd-logind", "facility_label" => "security/authorization", "severity_label" => "Informational" } { "priority" => 86, "message" => "Accepted password for root from 10.60.60.13 port 36528 ssh2\n", "timestamp" => "Jan 4 16:06:10", "@timestamp" => 2022-01-04T08:06:10.000Z, "facility" => 10, "type" => "system-syslog", "@version" => "1", "host" => "10.60.60.60", "logsource" => "10-60-60-60", "severity" => 6, "program" => "sshd", "pid" => "9875", "facility_label" => "security/authorization", "severity_label" => "Informational" } { "priority" => 86, "message" => "pam_unix(sshd:session): session opened for user root by (uid=0)\n", "timestamp" => "Jan 4 16:06:10", "@timestamp" => 2022-01-04T08:06:10.000Z, "facility" => 10, "type" => "system-syslog", "@version" => "1", "host" => "10.60.60.60", "logsource" => "10-60-60-60", "severity" => 6, "program" => "sshd", "pid" => "9875", "facility_label" => "security/authorization", "severity_label" => "Informational" } { "priority" => 30, "message" => "Started Session 78 of user root.\n", "timestamp" => "Jan 4 16:06:10", "@timestamp" => 2022-01-04T08:06:10.000Z, "facility" => 3, "type" => "system-syslog", "@version" => "1", "host" => "10.60.60.60", "logsource" => "10-60-60-60", "severity" => 6, "program" => "systemd", "facility_label" => "system", "severity_label" => "Informational" } ``` 如上,可以看到,终端中以JSON的格式打印了收集到的日志,表示测试成功了。 ## 7、输出日志到Elasticsearch 以上只是测试的配置,这一步我们需要重新改一下配置文件,让收集的日志信息输出到Elasticsearch服务器中,而不是当前终端 ```shell vim /usr/local/logstash/config/syslog.conf ``` 按照如下修改输出源 ```shell input { syslog { type => "system-syslog" port => 10514 } } output { elasticsearch { hosts => ["10.60.60.9:9200"] # 定义es服务器的ip index => "system-syslog-%{+YYYY.MM}" # 定义索引 } } ``` 修改完配置文件,检测一下配置文件是否正确 ```shell [root@10-60-60-60 ~]# /usr/local/logstash/bin/logstash --path.settings /usr/local/logstash/config/ -f /usr/local/logstash/config/syslog.conf --config.test_and_exit Using bundled JDK: /usr/local/logstash/jdk OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. Sending Logstash logs to /usr/local/logstash/logs which is now configured via log4j2.properties [2022-01-05T17:07:40,198][INFO ][logstash.runner ] Log4j configuration path used is: /usr/local/logstash/config/log4j2.properties [2022-01-05T17:07:40,207][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.16.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"} [2022-01-05T17:07:40,494][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2022-01-05T17:07:41,088][INFO ][org.reflections.Reflections] Reflections took 76 ms to scan 1 urls, producing 119 keys and 417 values Configuration OK [2022-01-05T17:07:42,044][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash [root@10-60-60-60 ~]# ``` 启动服务 ```shell /usr/local/logstash/bin/logstash --path.settings /usr/local/logstash/config/ -f /usr/local/logstash/config/syslog.conf ``` 查看端口的监听情况 ```shell [root@10-60-60-60 ~]# netstat -tuplan | grep 9600 tcp6 0 0 127.0.0.1:9600 :::* LISTEN 6669/java [root@10-60-60-60 ~]# ``` 9600端口有监听 ## 8、Logstash做成Service自启 编辑配置文件 ```shell vim /etc/systemd/system/logstash.service ``` 内容如下 ```shell [Unit] Description=logstash After=network-online.target [Service] Restart=on-failure ExecStart=/usr/local/logstash/bin/logstash --path.settings /usr/local/logstash/config/ -f /usr/local/logstash/config/syslog.conf [Install] WantedBy=multi-user.target ``` 启动服务以及设置开机自启(这里注意start服务之前把先前启动的进程先关了) ```shell systemctl daemon-reload systemctl start logstash.service systemctl enable logstash.service ``` ```shell [root@10-60-60-60 ~]# systemctl status logstash.service ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2022-01-05 17:34:51 CST; 2min 25s ago Main PID: 8824 (java) CGroup: /system.slice/logstash.service └─8824 /usr/local/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.h... Jan 05 17:35:47 10-60-60-60 logstash[8824]: [2022-01-05T17:35:47,326][INFO ][logstash.inputs.syslog ][main][636fb47e671dbe995b2da7e4d1538455caa7765bbc5fcfb26f04f...0.0:10514"} Jan 05 17:35:47 10-60-60-60 logstash[8824]: [2022-01-05T17:35:47,332][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], ...elines=>[]} Jan 05 17:35:51 10-60-60-60 logstash[8824]: [2022-01-05T17:35:51,219][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http:...0.9:9200/"} Jan 05 17:36:21 10-60-60-60 logstash[8824]: [2022-01-05T17:36:21,232][ERROR][logstash.outputs.elasticsearch][main] Unable to get license information {:url=>"http://10.60.60.9... Jan 05 17:36:21 10-60-60-60 logstash[8824]: [2022-01-05T17:36:21,233][ERROR][logstash.outputs.elasticsearch][main] Could not connect to a compatible version of Ela...0.9:9200/"} Jan 05 17:36:21 10-60-60-60 logstash[8824]: [2022-01-05T17:36:21,250][INFO ][logstash.inputs.syslog ][main][636fb47e671dbe995b2da7e4d1538455caa7765bbc5fcfb26f04f....60:51178"} Jan 05 17:36:22 10-60-60-60 logstash[8824]: [2022-01-05T17:36:22,239][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http:...0.9:9200/"} Jan 05 17:36:52 10-60-60-60 logstash[8824]: [2022-01-05T17:36:52,247][ERROR][logstash.outputs.elasticsearch][main] Unable to get license information {:url=>"http://10.60.60.9... Jan 05 17:36:52 10-60-60-60 logstash[8824]: [2022-01-05T17:36:52,248][ERROR][logstash.outputs.elasticsearch][main] Could not connect to a compatible version of Ela...0.9:9200/"} Jan 05 17:36:53 10-60-60-60 logstash[8824]: [2022-01-05T17:36:53,252][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http:...0.9:9200/"} Hint: Some lines were ellipsized, use -l to show in full. [root@10-60-60-60 ~]# ``` **就这样Logstash安装完成了!!!** 最后修改:2022 年 01 月 25 日 © 允许规范转载 赞 0 如果觉得我的文章对你有用,请随意赞赏