重点参考:

http://blog.csdn.net/qq1032355091/article/details/52953837

logstash的精髓:

grok插件原理

date插件原理

kv插件原理

日志默认情况

默认将日志内容赋给了message字段, logstash附加了@timestamp @version host 3个字段

{    "@timestamp" => 2017-11-30T06:09:09.625Z,      "@version" => "1",          "host" => "lb-212-222.above.com",       "message" => "sad"}

match匹配原则

参考: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html

date插件匹配过程解析

input { stdin { } } filter {  grok { match => [ "message", "%{HTTPDATE:[@metadata][timestamp]}" ] }  date { match => [ "[@metadata][timestamp]", "dd/MMM/yyyy:HH:mm:ss Z" ] }} output {  stdout { codec => rubydebug }}

##用正则HTTPDATE匹配message,将结果赋给[@metadata][timestamp]字段grok { match => [ "message", "%{HTTPDATE:[@metadata][timestamp]}" ] }##date插件将[@metadata][timestamp]的值赋给 @timestamp字段date { match => [ "[@metadata][timestamp]", "dd/MMM/yyyy:HH:mm:ss Z" ] }

下面是一个完整例子:

参考: http://blog.csdn.net/xiaoyu_bd/article/details/52531051

input  {     stdin{}}filter {    grok {        match => ["message", "%{TIMESTAMP_ISO8601:logdate}"]    }    date {        match => ["logdate", "yyyy-MM-dd HH:mm:ss,SSS"]        target => "@timestamp"  ## 默认target就是"@timestamp    }}output{    stdout{        codec=>rubydebug{}    }}

date {    match => [“timestamp”, “dd/MMM/yyyy:HH:mm:ss Z”]    #默认目标就是@timestamp    target => "@timestamp"    "locale" => "en"}

mutate插件

• 修改字段类型
  参考(修改时间格式): http://blog.csdn.net/wang_zhenwei/article/details/49760975

mutate {          convert => { "dest_Port" => "integer" }        convert => { "source_Port" => "integer" }     }

• 添加字段

input { stdin { } } filter {  mutate { add_field => { "show" => "This data will be in the output" } }} output {    stdout { codec => rubydebug }}

• 还可以转换字段大小写

kibana 查询结果csv导出

table类型的导出:

饼图统计结果导出