Open Stack

Hi,

We currently use the following grok patterns for openstack with logstash:

AUDITLOGLEVEL ([C|c]ritical|CRITICAL[A|a]udit|AUDIT|[D|d]ebug|DEBUG|[N|n]otice|NOTICE|[I|i]nfo|INFO|[W|w]arn?(?:ing)?|WARN?(?:ING)?|[E|e]rr?(?:or)?|ERR?(?:OR)?|[C|c]rit?(?:ical)?|CRIT?(?:ICAL)?|[F|f]atal|FATAL|[S|s]evere|SEVERE)
RSYSLOG <%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:timestamp}
(?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:
%{GREEDYDATA:message}
OPENSTACK <%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:timestamp}
(%{SYSLOGHOST:logsource} )?(%{TIMESTAMP_ISO8601} )?(%{POSINT:pid} )?([
]*%{AUDITLOGLEVEL:auditlevel}[
]*)?\[*?%{WORD:program}\.?%{PROG:program_path}?\]*
%{GREEDYDATA:message}

along with the following rsyslog template:

# log formats
$template NectarForwardFormat,"<%pri%>%timestamp:::date-rfc3339%
%hostname% %syslogtag%%msg%\n"
$ActionForwardDefaultTemplate NectarForwardFormat

If your format doesn't match the above, you can probably pick out the
bits that are relevant to you.

If you want some further information beyond the grok filters....

Rsyslog on clients forwards to a central log server (over openvpn, but
that's not necessary).

Rsyslog on the central server outputs to logstash:

# define our own raw format for logstash and debuging
$template raw,"<%pri%>%timestamp:::date-rfc3339% %hostname% %syslogtag%%msg%\n"
# Ouptput to tcp port 20515 in raw format for logstash
*.* @@127.0.0.1:20515;raw

And our logstash.conf is as follows (F5 is custom format for parsing F5 logs):

input {
    tcp {
        type => "rsyslog"
        port => "20515"
        host => "127.0.0.1"
    }
}

filter {

    grok {
        patterns_dir => "/etc/logstash/patterns"
        match        => [ "message", "%{F5}" ]
        overwrite   => "message"
        add_tag      => [ "f5" ]
    }
    if "_grokparsefailure" in [tags] {
        grok {
            patterns_dir => "/etc/logstash/patterns"
            match        => [ "message", "%{RSYSLOG}" ]
            overwrite   => "message"
            add_tag      => [ "rsyslog" ]
            remove_tag   => "_grokparsefailure"
        }
    }
    if "_grokparsefailure" in [tags] {
        grok {
            patterns_dir => "/etc/logstash/patterns"
            match        => [ "message", "%{OPENSTACK}" ]
            add_field    => [ "fullprogram", "%{program}.%{program_path}" ]
            overwrite    => "message"
            add_tag      => [ "openstack" ]
            remove_tag   => "_grokparsefailure"
        }
    }
}

output {
    elasticsearch {
        cluster   => "logstash"
        host      => "127.0.0.1"
        bind_host => "127.0.0.1"
    }
}

Hope this helps,

Marcus.

On 22 May 2014 04:19, Mathieu Gagné <mgagne at iweb.com> wrote:
> Hi operators,
>
> It appears that most people are using or planning to use logstash to handle
> and manage their OpenStack logs.
>
> Is there any places, repositories or documentations where I can find grok
> patterns for all the OpenStack services?
>
> If such place does not exist, can we create one so we can search them?
>
> --
> Mathieu
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



-- 
Marcus Furlong