Open Stack

Hi. Few comments inline.

> Dear Stackers,
> While working on my locally deployed Openstack (Pike using TripleO), I
> was a bit struggling with the logging part. Currently, all logs are
> pushed to per-service files, in the standard format "one line per
> entry", plain flat text.
> 
> It's nice, but if one is wanting to push and index those logs in an ELK,
> the current, default format isn't really good.
> 
> After some discussions about oslo.log, it appears it provides a nice
> JSONFormatter handler¹ one might want to use for each (python) service
> using oslo central library.
> 
> A JSON format is really cool, as it's easy to parse for machines, and it
> can be on a multi-line without any bit issue - this is especially
> important for stack traces, as their output is multi-line without real
> way to have a common delimiter so that we can re-format it and feed it
> to any log parser (logstash, fluentd, …).
> 
> After some more talks, olso.log will not provide a unified interface in
> order to output all received logs as JSON - this makes sens, as that
> would mean "rewrite almost all the python logging management
> interface"², and that's pretty useless, since (all?) services have their
> own "logging.conf" file.
> 
> That said… to the main purpose of that mail:
> 
> - Default format for logs
> A first question would be "are we all OK with the default output format"
> - I'm pretty sure "humans" are happy with that, as it's really
> convenient to read and grep. But on a "standard" Openstack deploy, I'm
> pretty sure one does not have only one controller, one ceph node and one
> compute. Hence comes the log centralization, and with that, the log
> indexation and treatments.
> 
> For that, one might argue "I'm using plain files on my logger, and
> grep-ing -r in them". That's a way to do things, and for that, plain,
> flat logs are great.
> 
> But… I'm pretty sure I'm not the only one wanting to use some kind of
> ELK cluster for that kind of purpose. So the right question is: what
> about switching the default log format to JSON? On my part, I don't see
> "cons", only "pros", but my judgment is of course biased, as I'm "alone
> in my corner". But what about you, Community?
> 
> - Provide a way to configure the output format/handler
> While poking around in the puppet modules code, I didn't find any way to
> set the output handler for the logs. For example, in puppet-nova³ we can
> set a lot of things, but not the useful handler for the output.
> 
> It would be really cool to get, for each puppet module, the capability
> to set the handler so that one can just push some stuff in hiera, and
> Voilà, we have JSON logs.
> 
> Doing so would allow people to chose between the default  (current)
> output, and something more "computable".

This is being implemented (for Q) in the scope of this blueprint [0].
By default, containers will keep the current logging behaviour, which is 
  writing logs into bind-mounted host-path directories, and for those 
services that can syslog only, sending the logs to the host journald via 
syslog (bind-mounted /dev/log). And if the stdout/stderr feature 
enabled, side-car containers will make sure everything is captured in 
journald and tagged properly, with multiline entries as well. And 
journald can dump messages as JSON as well. I hope that works! Please 
send you comments to the aforementioned bp's spec.

[0] https://blueprints.launchpad.net/tripleo/+spec/logging-stdout-rsyslog

> 
> Of course, either proposal will require a nice code change in all puppet
> modules (add a new parameter for the foo::logging class, and use that
> new param in the configuration file, and so on), but at least people
> will be able to actually chose.
> 
> So, before opening an issue on each launchpad project (that would be…
> long), I'd rather open the discussion in here and, eventually, come to
> some nice, acceptable and accepted solution that would make the
> Openstack Community happy :).
> 
> Any thoughts?
> 
> Thank you for your attention, feedback and wonderful support for that
> monster project :).
> 
> Cheers,
> 
> C.
> 
> 
> ¹
> https://github.com/openstack/oslo.log/blob/master/oslo_log/formatters.py#L166-L235
> ²
> http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/%23openstack-oslo.2017-11-01.log.html#t2017-11-01T13:23:14
> ³ https://github.com/openstack/puppet-nova/blob/master/manifests/logging.pp
> 
> 
> -- 
> Cédric Jeanneret
> Senior Linux System Administrator
> Infrastructure Solutions
> 
> Camptocamp SA
> PSE-A / EPFL, 1015 Lausanne
> 
> Phone: +41 21 619 10 32
> Office: +41 21 619 10 02
> Email: cedric.jeanneret at camptocamp.com
> 
> 
> 

-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando