Loading... <p><span style="font-family:宋体">网海过客<br /></span>www.chinasa.net</p> <p>ELK 日志分析实例<br />一、ELK-web日志分析<br />二、ELK-MySQL 慢查询日志分析<br />三、ELK-SSH登陆日志分析<br />四、ELK-vsftpd 日志分析</p> <p></p> <p><span style="font-family: 宋体"></span></p> <h2><span style="font-family:宋体">一、</span>ELK-web<span style="font-family:宋体">日志分析</span></h2> <p style="border:none;padding:0;padding:0 0 0 12px"><span style="font-family:宋体">通过</span>logstash grok<span style="font-family:宋体">正则将</span>web<span style="font-family:宋体">日志过滤出来,输出到</span>Elasticsearch <span style="font-family:宋体">搜索引擎里,通过</span>Kibana<span style="font-family:宋体">前端展示。</span></p> <p> <img src="//cto.wang/usr/uploads/2016/07/20160703184430-15.png" title="1464839185717065.png" alt="1.png" /></p> <p><strong><span style="font-size:16px">1.1</span><span style="font-size:16px;font-family:宋体">、创建</span><span style="font-size:16px">logstash grok </span><span style="font-size:16px;font-family:宋体">过滤规则</span></strong></p> <p>#cat ./logstahs/patterns/nginx</p> <p><span style="font-size: 14px">NGINXACCESS %{IPORHOST:remote_addr} – – \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}</span></p> <p></p> <p><strong>1.2、创建logstash web日志配置文件</strong></p> <p>#cat ./logstash/conf/ngx_log.conf</p> <pre class="brush:html;toolbar:false">input { file { type => "nginx_log" path => "/opt/nginx/logs/access.log" } } filter { if [type] == "nginx_log" { grok { match => { "message" => "%{NGINXACCESS}" } } geoip { source => "remote_addr" target => "geoip" database => "/opt/logstash-2.0.0/conf/GeoLiteCity.dat" add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ] add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ] } mutate { convert => [ "[geoip][coordinates]","float", "body_bytes_sent","float", \ "body_bytes_sent.raw","float"] } } } output { stdout { codec => rubydebug } elasticsearch { hosts => "elk.test.com:9200" index => "ngx_log-%{+YYYY.MM}" } }</pre> <p></p> <p><strong><span style="font-size:16px">1.3</span><span style="font-size:16px;font-family: 宋体">、创建</span><span style="font-size: 16px">Kibana</span><span style="font-size:16px;font-family:宋体">图形</span></strong></p> <p><span style="font-family: 宋体">统计</span>httpcode<span style="font-family:宋体">状态码</span></p> <p><span style="font-family: 宋体">选择【</span>Visualize<span style="font-family:宋体">】菜单,选择</span> <span style="font-family: 宋体">【</span>Pie chart<span style="font-family:宋体">】选项。字段选择</span>status.raw<span style="font-family:宋体">,如下图所示</span>:</p> <p><img src="//cto.wang/usr/uploads/2016/07/20160703184430-52.png" title="1464841431235759.png" alt="2.png" /></p> <p><strong><span style="font-family: 宋体">统计访问前</span>50 IP</strong></p> <p><span style="font-family: 宋体">选择【</span>Visualize<span style="font-family:宋体">】菜单,选择</span> <span style="font-family: 宋体">【</span>Vertical bar chart<span style="font-family:宋体">】选项。字段选择</span>remote_addr.raw<span style="font-family:宋体">,如下图所示</span>:</p> <p></p> <p><img src="//cto.wang/usr/uploads/2016/07/20160703184431-5.png" title="1464841459364774.png" alt="3.png" /></p> <p><span style="font-family: 宋体"><br /></span></p> <p><strong><span style="font-family: 宋体">统计</span> 403-405 <span style="font-family:宋体">状态码</span></strong></p> <p><span style="font-family: 宋体">选择【</span>Visualize<span style="font-family:宋体">】菜单,选择</span> <span style="font-family: 宋体">【</span>Line chart<span style="font-family:宋体">】选项。字段选择</span>status.raw<span style="font-family:宋体">,如下图所示</span>:</p> <p><img src="//cto.wang/usr/uploads/2016/07/20160703184431-78.png" title="1464841491850854.png" alt="4.png" /></p> <p><span style="font-family: 宋体">其它图形统计,就不详细举例了。</span></p> <p></p> <p><strong><span style="font-size: 16px;font-family:宋体">详细图形展示如下</span><span style="font-size:16px">:</span></strong></p> <p><img src="//cto.wang/usr/uploads/2016/07/20160703184432-39.png" title="1464841538310714.png" alt="5.png" /><span style="font-family: 宋体"></span></p> <p><img src="//cto.wang/usr/uploads/2016/07/20160703184432-75.png" title="1464841541866268.png" alt="6.png" /></p> <p><img src="//cto.wang/usr/uploads/2016/07/20160703184433-80.png" title="1464841544471220.png" alt="7.png" /></p> <p></p> <h2><span style="font-family:宋体">二、</span>ELK-MySQL <span style="font-family:宋体">慢查询日志分析</span></h2> <p><strong>2.1、创建logstash grok 过滤规则</strong></p> <p>#cat ./logstahs/patterns/mysql_slow</p> <pre class="brush:html;toolbar:false;">MYSQLSLOW "# User@Host: %{WORD:user}\[%{WORD}\] @ (%{HOST:client_hostname}|) \[(%{IP:client_ip}|)\]", "# Thread_id: %{NUMBER:thread_id:int} \s*Schema: (%{WORD:schema}| ) \s*Last_errno: \ %{NUMBER:last_errno:int} \s*Killed: %{NUMBER:killed:int}", "# Query_time: %{NUMBER:query_time:float} \s*Lock_time: %{NUMBER:lock_time:float} \ \s*Rows_sent: %{NUMBER:rows_sent:int} \s*Rows_examined: %{NUMBER:rows_examined:int}", "# Bytes_sent: %{NUMBER:bytes_sent:int}", "(?m)SET timestamp=%{NUMBER:timestamp};%{GREEDYDATA:mysql_query}"</pre> <p></p> <p><strong>2.2、创建logstash MySQL-Slow慢查询配置文件</strong></p> <p>#cat ./logstash/conf/MySQL-Slow.conf</p> <pre class="brush:html;toolbar:false">input { file { type => "mysql-slow" path => "/var/log/mysql_slow_log.log" } } filter { if [type] == "mysql-slow" { multiline { pattern => "^#|^SET" negate => true what => "previous" } grok { match => { "message" => "%{MYSQLSLOW}" } } mutate { gsub => [ "mysql_query", "\n", " " ] gsub => [ "mysql_query", " ", " " ] add_tag => "mutated_mysql_query" } multiline { pattern => "(# User|# Thread|# Query|# Time|# Bytes)" negate => false what => "next" } date { match => [ "timestamp","UNIX" ] } mutate { remove_field => [ "timestamp" ] } } } output { stdout { codec => rubydebug } elasticsearch { hosts => "elk.test.com:9200" index => "mysql_slow_log-%{+YYYY.MM}" } }</pre> <p><strong>2.3<span style="font-family: 宋体">、详细图形展示如下</span>:</strong></p> <p><strong><span style="font-size:16px"><img src="//cto.wang/usr/uploads/2016/07/20160703184433-24.png" title="1464841994645026.png" alt="8.png" /></span></strong></p> <p></p> <h2><span style="font-family:宋体">三、</span>ELK-SSH<span style="font-family:宋体">登陆日志分析</span></h2> <p><strong>3.1、创建logstash grok 过滤规则</strong></p> <p>#cat ./logstahs/patterns/ssh</p> <p><span style="font-family:宋体"></span></p> <p><span style="font-size: 14px">SECURELOG %{WORD:program}\[%{DATA:pid}\]: %{WORD:status} password for ?(invalid user)? %{WORD:USER} from %{DATA:IP} port</span></p> <p><span style="font-size: 14px">SYSLOGPAMSESSION %{SYSLOGBASE} (?=%{GREEDYDATA:message})%{WORD:pam_module}\(%{DATA:pam_caller}\): session %{WORD:pam_session_state} for user %{USERNAME:username}(?: by %{GREEDYDATA:pam_by})?</span></p> <p><span style="font-size: 14px">SYSLOGBASE2 (?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:</span></p> <p><strong><span style="font-family:宋体"><br /></span></strong></p> <p><strong>3.2、创建logstash ssh配置文件</strong></p> <p>#cat ./logstash/conf/ssh.conf</p> <pre class="brush:html;toolbar:false">input { file { type => "seclog" path => "/var/log/secure" } } filter { if [type] == "seclog" { grok { match => { "message" => "%{SYSLOGPAMSESSION}" } match => { "message" => "%{SECURELOG}" } match => { "message" => "%{SYSLOGBASE2}" } } geoip { source => "IP" fields => ["city_name"] database => "/opt/logstash-2.0.0/conf/GeoLiteCity.dat" } if ([status] == "Accepted") { mutate { add_tag => ["Success"] } } else if ([status] == "Failed") { mutate { add_tag => ["Failed"] } } } output { stdout { codec => rubydebug } elasticsearch { hosts => "elk.test.com:9200" index => "sshd_log-%{+YYYY.MM}" } }</pre> <p><span style="color: red">PS:</span><span style="font-family: 宋体">添加状态标签,便于</span>Kibana <span style="font-family: 宋体">统计</span><br /><span style="font-family: 宋体"></span></p> <pre class="brush:html;toolbar:false">if ([status] == "Accepted") { #判断字段[status]值,匹配[Accepted] mutate { add_tag => ["Success"] #添加标签[Success] } } else if ([status] == "Failed") { #判断字段[status]值,匹配[Failed] mutate { add_tag => ["Failed"] #添加标签[Failed] } }</pre> <p><span style="font-size:16px"></span></p> <p><strong><span style="font-size:16px"><br /></span></strong></p> <p><strong><span style="font-size:16px">3.3</span><span style="font-size:16px;font-family:宋体">、详细图形展示如下</span><span style="font-size:16px">:</span></strong></p> <p><strong><span style="font-size:16px"><img src="//cto.wang/usr/uploads/2016/07/20160703184433-33.png" title="1464842162381708.png" alt="9.png" /></span></strong></p> <p></p> <h2><span style="font-family:宋体">四、</span>ELK-vsftpd <span style="font-family:宋体">日志分析</span></h2> <p><strong><span style="font-size:16px"></span></strong></p> <p>4.1、创建logstash grok 过滤规则</p> <p>#cat ./logstahs/patterns/vsftpd</p> <p><span style="font-size: 14px">VSFTPDCONNECT \[pid %{WORD:pid}\] %{WORD:action}: Client \"%{DATA:IP}\"<br /></span><span style="font-size: 14px">VSFTPDLOGIN \[pid %{WORD:pid}\] \[%{WORD:user}\] %{WORD:status} %{WORD:action}: Client \"%{DATA:IP}\"VSFTPDACTION \[pid %{DATA:pid}\] \[%{DATA:user}\] %{WORD:status} %{WORD:action}: Client \"%{DATA:IP}\", \"%{DATA:file}\", %{DATA:bytes} bytes, %{DATA:Kbyte_sec}Kbyte/sec </span></p> <p><span style="font-size:16px"></span></p> <p>4.2、创建logstash vsftpd配置文件</p> <p>#cat ./logstash/conf/vsftpd.conf</p> <pre class="brush:html;toolbar:false">input { file { type => "vsftpd_log" path => "/var/log/vsftpd.log" } } filter { if [type] == "vsftpd_log" { grok { match => { "message" => "%{VSFTPDACTION}" } match => { "message" => "%{VSFTPDLOGIN}" } match => { "message" => "%{VSFTPDCONNECT}" } } } } output { stdout { codec => rubydebug } elasticsearch { hosts => "elk.test.com:9200" index => "vsftpd_log-%{+YYYY.MM}" } }</pre> <p><span style="font-family:宋体"></span></p> <p><span style="font-family:宋体"></span></p> <p><strong><span style="font-size:16px">4.3</span><span style="font-size:16px;font-family:宋体">、详细图形展示如下</span><span style="font-size:16px">:</span></strong></p> <p><strong><span style="font-size:16px"></span></strong></p> <p><img src="//cto.wang/usr/uploads/2016/07/20160703184434-66.png" title="1464842264820223.png" alt="10.png" /></p> 最后修改:2021 年 12 月 10 日 10 : 53 AM © 允许规范转载 赞赏 如果觉得我的文章对你有用,请随意赞赏 赞赏作者 支付宝微信