[openstack-dev] [kolla-ansible] how do I unify log data format

Mark Goddard mark at stackhpc.com
Mon Jul 16 11:09:30 UTC 2018

Hi Sergey,

We are using Kolla Ansible with monasca log API, and have added support for
customising the fluentd configuration [1][2]. Doug Szumski (dougsz) made
some changes in Queens to try to standardise the log message format. I
think Kolla Ansible would benefit from some better documentation on what
this format is.

On the monasca log API side there is support for transforming logs using



On 14 July 2018 at 14:29, Sergey Glazyrin <sergey.glazyrin.dev at gmail.com>

> Hello guys!
> We are migrating our product to kolla-ansible and as far as probably you
> know, it uses fluentd to control logs, etc. In non containerized openstack
> we use rsyslog to send data to logstash. We get data from syslog events. It
> looks like it's impossible to use syslog in kolla-ansible. Unfortunately
> external_syslog_server option doesn't work. Is there anyone who was able to
> use it ? But, nevermind, we may use fluentd BUT.. we have one problem -
> different data format for each service/container.
> So, probably the most optimal solution is to use default logging idea in
> kolla-ansible. (to be honest, I am not sure... but I've no found better
> option). But even with default logging idea in kolla - ansible we have one
> serious problem. Fluentd has different data format for each service, for
> instance, you may see this link with explanation how its designed in
> kolla-ansible
> https://github.com/openstack/kolla-ansible/commit/
> 3026cef7cfd1828a27e565d4211692f0ab0ce22e
> there are grok patterns which parses log messages, etc
> so, we managed to put data to elasticsearch but we need to solve two
> problems:
> 1. unify data format for log events. We may solve it using logstash to
> unify it before putting it to elasticsearch (or should we change fluentd
> configs in our own version of kolla-ansible repository ? )
> For instance, we may do it using this logstash plugin
> https://www.elastic.co/guide/en/logstash/2.4/plugins-
> filters-mutate.html#plugins-filters-mutate-rename
> What's your suggestion ?
> --
> Best, Sergey
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180716/9894ef61/attachment.html>

More information about the OpenStack-dev mailing list