[openstack-dev] [kolla-ansible] how do I unify log data format

Rich Megginson rmeggins at redhat.com
Sat Jul 14 16:36:27 UTC 2018

On 07/14/2018 07:29 AM, Sergey Glazyrin wrote:
> Hello guys!
> We are migrating our product to kolla-ansible and as far as probably 
> you know, it uses fluentd to control logs, etc. In non containerized 
> openstack we use rsyslog to send data to logstash.

Why not use rsyslog in containerized openstack too?
Why not use rsyslog to mutate/unify the records?  Why use logstash? Note 
that rsyslog can send records to elasticsearch, and the latest rsyslog 
8.36 has enhanced the elasticsearch plugin to do client cert auth as 
well as handle bulk index retries more efficiently.

> We get data from syslog events. It looks like it's impossible to use 
> syslog in kolla-ansible. Unfortunately external_syslog_server option 
> doesn't work. Is there anyone who was able to use it ? But, nevermind, 
> we may use fluentd BUT.. we have one problem - different data format 
> for each service/container.
> So, probably the most optimal solution is to use default logging idea 
> in kolla-ansible. (to be honest, I am not sure... but I've no found 
> better option). But even with default logging idea in kolla - ansible 
> we have one serious problem. Fluentd has different data format for 
> each service, for instance, you may see this link with explanation how 
> its designed in kolla-ansible
> https://github.com/openstack/kolla-ansible/commit/3026cef7cfd1828a27e565d4211692f0ab0ce22e
> there are grok patterns which parses log messages, etc
> so, we managed to put data to elasticsearch but we need to solve two 
> problems:
> 1. unify data format for log events. We may solve it using logstash to 
> unify it before putting it to elasticsearch (or should we change 
> fluentd configs in our own version of kolla-ansible repository ? )
> For instance, we may do it using this logstash plugin
> https://www.elastic.co/guide/en/logstash/2.4/plugins-filters-mutate.html#plugins-filters-mutate-rename
> What's your suggestion ?
> -- 
> Best, Sergey
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

More information about the OpenStack-dev mailing list