Wednesday, June 21, 2017

[Logstash] How to process Amavis logs via filebeat and logstash for elasticsearch

First activate JSON logging for amavis by following this:

in rsyslog.conf add:
$MaxMessageSize 32k
in amavisd.conf add: 
$logline_maxlen = ( 32*1024 ) - 50; # 32k max message size, keep 50 bytes for syslog
$log_templ = <<'EOD';
[:report_json]
EOD

Now the thing is amavis logs via syslog to file so the line structure is syslog + json message.
For Filebeat add the following to your config:

  fields:
        tags: ['json']

  fields_under_root: true

Now you do not need the json tag per se but I am using it because I got json and non-json logs. (can't use hilite me, will break page :D in case you have issues use this https://pastebin.com/raw/DtRJ5mDX )
filter
{
if "json" in [tags] {

grok {
    match => { "message" => "%{SYSLOGTIMESTAMP:logtime} %{SYSLOGHOST} %{DATA:program}(?:\[%{POSINT}\])?: (?:\((.*?)\))? %{GREEDYDATA:message}" }
    overwrite => "message"
}
}
}
Now in your logstash config you need the following to process the message part properly and use the json as source:

You can add the following to it or keep it in a separate config but what you need then is:

    if "json" in [tags] { 
    json {
      source => "message"
       }   
    }

No comments:

Post a Comment