Open Stack

Hi Sergey,

We are using Kolla Ansible with monasca log API, and have added support for
customising the fluentd configuration [1][2]. Doug Szumski (dougsz) made
some changes in Queens to try to standardise the log message format. I
think Kolla Ansible would benefit from some better documentation on what
this format is.

On the monasca log API side there is support for transforming logs using
logstash.

[1]
https://docs.openstack.org/kolla-ansible/latest/reference/central-logging-guide.html#custom-log-forwarding
[2]
https://docs.openstack.org/kolla-ansible/latest/reference/central-logging-guide.html#custom-log-filtering

Cheers,
Mark

On 14 July 2018 at 14:29, Sergey Glazyrin <sergey.glazyrin.dev at gmail.com>
wrote:

> Hello guys!
> We are migrating our product to kolla-ansible and as far as probably you
> know, it uses fluentd to control logs, etc. In non containerized openstack
> we use rsyslog to send data to logstash. We get data from syslog events. It
> looks like it's impossible to use syslog in kolla-ansible. Unfortunately
> external_syslog_server option doesn't work. Is there anyone who was able to
> use it ? But, nevermind, we may use fluentd BUT.. we have one problem -
> different data format for each service/container.
>
> So, probably the most optimal solution is to use default logging idea in
> kolla-ansible. (to be honest, I am not sure... but I've no found better
> option). But even with default logging idea in kolla - ansible we have one
> serious problem. Fluentd has different data format for each service, for
> instance, you may see this link with explanation how its designed in
> kolla-ansible
> https://github.com/openstack/kolla-ansible/commit/
> 3026cef7cfd1828a27e565d4211692f0ab0ce22e
> there are grok patterns which parses log messages, etc
>
> so, we managed to put data to elasticsearch but we need to solve two
> problems:
> 1. unify data format for log events. We may solve it using logstash to
> unify it before putting it to elasticsearch (or should we change fluentd
> configs in our own version of kolla-ansible repository ? )
> For instance, we may do it using this logstash plugin
> https://www.elastic.co/guide/en/logstash/2.4/plugins-
> filters-mutate.html#plugins-filters-mutate-rename
>
> What's your suggestion ?
>
>
> --
> Best, Sergey
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180716/9894ef61/attachment.html>