Open Stack

Dear Stackers,

While working on my locally deployed Openstack (Pike using TripleO), I
was a bit struggling with the logging part. Currently, all logs are
pushed to per-service files, in the standard format "one line per
entry", plain flat text.

It's nice, but if one is wanting to push and index those logs in an ELK,
the current, default format isn't really good.

After some discussions about oslo.log, it appears it provides a nice
JSONFormatter handler¹ one might want to use for each (python) service
using oslo central library.

A JSON format is really cool, as it's easy to parse for machines, and it
can be on a multi-line without any bit issue - this is especially
important for stack traces, as their output is multi-line without real
way to have a common delimiter so that we can re-format it and feed it
to any log parser (logstash, fluentd, …).

After some more talks, olso.log will not provide a unified interface in
order to output all received logs as JSON - this makes sens, as that
would mean "rewrite almost all the python logging management
interface"², and that's pretty useless, since (all?) services have their
own "logging.conf" file.

That said… to the main purpose of that mail:

- Default format for logs
A first question would be "are we all OK with the default output format"
- I'm pretty sure "humans" are happy with that, as it's really
convenient to read and grep. But on a "standard" Openstack deploy, I'm
pretty sure one does not have only one controller, one ceph node and one
compute. Hence comes the log centralization, and with that, the log
indexation and treatments.

For that, one might argue "I'm using plain files on my logger, and
grep-ing -r in them". That's a way to do things, and for that, plain,
flat logs are great.

But… I'm pretty sure I'm not the only one wanting to use some kind of
ELK cluster for that kind of purpose. So the right question is: what
about switching the default log format to JSON? On my part, I don't see
"cons", only "pros", but my judgment is of course biased, as I'm "alone
in my corner". But what about you, Community?

- Provide a way to configure the output format/handler
While poking around in the puppet modules code, I didn't find any way to
set the output handler for the logs. For example, in puppet-nova³ we can
set a lot of things, but not the useful handler for the output.

It would be really cool to get, for each puppet module, the capability
to set the handler so that one can just push some stuff in hiera, and
Voilà, we have JSON logs.

Doing so would allow people to chose between the default  (current)
output, and something more "computable".

Of course, either proposal will require a nice code change in all puppet
modules (add a new parameter for the foo::logging class, and use that
new param in the configuration file, and so on), but at least people
will be able to actually chose.

So, before opening an issue on each launchpad project (that would be…
long), I'd rather open the discussion in here and, eventually, come to
some nice, acceptable and accepted solution that would make the
Openstack Community happy :).

Any thoughts?

Thank you for your attention, feedback and wonderful support for that
monster project :).

Cheers,

C.


¹
https://github.com/openstack/oslo.log/blob/master/oslo_log/formatters.py#L166-L235
²
http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/%23openstack-oslo.2017-11-01.log.html#t2017-11-01T13:23:14
³ https://github.com/openstack/puppet-nova/blob/master/manifests/logging.pp


-- 
Cédric Jeanneret
Senior Linux System Administrator
Infrastructure Solutions

Camptocamp SA
PSE-A / EPFL, 1015 Lausanne

Phone: +41 21 619 10 32
Office: +41 21 619 10 02
Email: cedric.jeanneret at camptocamp.com


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 878 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20171101/5ff299cd/attachment.sig>