Open Stack

Excerpts from Juan Antonio Osorio's message of 2017-11-04 00:11:47 +0200:
> > On 3 Nov 2017 19:59, "Doug Hellmann" <doug at doughellmann.com> wrote:
> > 
> > Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 +0100:
> > > Dear Stackers,
> > >
> > > While working on my locally deployed Openstack (Pike using TripleO), I
> > > was a bit struggling with the logging part. Currently, all logs are
> > > pushed to per-service files, in the standard format "one line per
> > > entry", plain flat text.
> > >
> > > It's nice, but if one is wanting to push and index those logs in an ELK,
> > > the current, default format isn't really good.
> > >
> > > After some discussions about oslo.log, it appears it provides a nice
> > > JSONFormatter handler¹ one might want to use for each (python) service
> > > using oslo central library.
> > >
> > > A JSON format is really cool, as it's easy to parse for machines, and it
> > > can be on a multi-line without any bit issue - this is especially
> > > important for stack traces, as their output is multi-line without real
> > > way to have a common delimiter so that we can re-format it and feed it
> > > to any log parser (logstash, fluentd, …).
> > >
> > > After some more talks, olso.log will not provide a unified interface in
> > > order to output all received logs as JSON - this makes sens, as that
> > > would mean "rewrite almost all the python logging management
> > > interface"², and that's pretty useless, since (all?) services have their
> > > own "logging.conf" file.
> > >
> > > That said… to the main purpose of that mail:
> > >
> > > - Default format for logs
> > > A first question would be "are we all OK with the default output format"
> > > - I'm pretty sure "humans" are happy with that, as it's really
> > > convenient to read and grep. But on a "standard" Openstack deploy, I'm
> > > pretty sure one does not have only one controller, one ceph node and one
> > > compute. Hence comes the log centralization, and with that, the log
> > > indexation and treatments.
> > >
> > > For that, one might argue "I'm using plain files on my logger, and
> > > grep-ing -r in them". That's a way to do things, and for that, plain,
> > > flat logs are great.
> > >
> > > But… I'm pretty sure I'm not the only one wanting to use some kind of
> > > ELK cluster for that kind of purpose. So the right question is: what
> > > about switching the default log format to JSON? On my part, I don't see
> > > "cons", only "pros", but my judgment is of course biased, as I'm "alone
> > > in my corner". But what about you, Community?
> > >
> > > - Provide a way to configure the output format/handler
> > > While poking around in the puppet modules code, I didn't find any way to
> > > set the output handler for the logs. For example, in puppet-nova³ we can
> > > set a lot of things, but not the useful handler for the output.
> > >
> > > It would be really cool to get, for each puppet module, the capability
> > > to set the handler so that one can just push some stuff in hiera, and
> > > Voilà, we have JSON logs.
> > >
> > > Doing so would allow people to chose between the default  (current)
> > > output, and something more "computable".
> > 
> > Using the JSON formatter currently requires setting a logging
> > configuration file using the standard library configuration format
> > and fully specifying things like log levels, handlers, and output
> > destination. Would it make sense to add an option in oslo.log to
> > give deployers an easier way to enable the JSON formatter?
> > 
>
> This would actually be very useful.

Great! Let me know if you would like to help figuring out where to do
that in the library code.

Doug