[openstack-dev] [kolla-ansible] how do I unify log data format

Sergey Glazyrin sergey.glazyrin.dev at gmail.com
Sat Jul 14 13:29:38 UTC 2018

Hello guys!
We are migrating our product to kolla-ansible and as far as probably you
know, it uses fluentd to control logs, etc. In non containerized openstack
we use rsyslog to send data to logstash. We get data from syslog events. It
looks like it's impossible to use syslog in kolla-ansible. Unfortunately
external_syslog_server option doesn't work. Is there anyone who was able to
use it ? But, nevermind, we may use fluentd BUT.. we have one problem -
different data format for each service/container.

So, probably the most optimal solution is to use default logging idea in
kolla-ansible. (to be honest, I am not sure... but I've no found better
option). But even with default logging idea in kolla - ansible we have one
serious problem. Fluentd has different data format for each service, for
instance, you may see this link with explanation how its designed in
there are grok patterns which parses log messages, etc

so, we managed to put data to elasticsearch but we need to solve two
1. unify data format for log events. We may solve it using logstash to
unify it before putting it to elasticsearch (or should we change fluentd
configs in our own version of kolla-ansible repository ? )
For instance, we may do it using this logstash plugin

What's your suggestion ?

Best, Sergey
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180714/eb3b0063/attachment.html>

More information about the OpenStack-dev mailing list