Open Stack

On 11/02/2017 05:18 PM, Ben Nemec wrote:
> Adding tags for the relevant projects.  I _think_ this is mostly
> directed at Puppet/TripleO, but Oslo is obviously relevant as well.

Thank you! First mail in there, wasn't really sure how to do that.

> 
> On 11/01/2017 08:54 AM, Cédric Jeanneret wrote:
>> Dear Stackers,
>>
>> While working on my locally deployed Openstack (Pike using TripleO), I
>> was a bit struggling with the logging part. Currently, all logs are
>> pushed to per-service files, in the standard format "one line per
>> entry", plain flat text.
>>
>> It's nice, but if one is wanting to push and index those logs in an ELK,
>> the current, default format isn't really good.
>>
>> After some discussions about oslo.log, it appears it provides a nice
>> JSONFormatter handler¹ one might want to use for each (python) service
>> using oslo central library.
>>
>> A JSON format is really cool, as it's easy to parse for machines, and it
>> can be on a multi-line without any bit issue - this is especially
>> important for stack traces, as their output is multi-line without real
>> way to have a common delimiter so that we can re-format it and feed it
>> to any log parser (logstash, fluentd, …).
>>
>> After some more talks, olso.log will not provide a unified interface in
>> order to output all received logs as JSON - this makes sens, as that
>> would mean "rewrite almost all the python logging management
>> interface"², and that's pretty useless, since (all?) services have their
>> own "logging.conf" file.
>>
>> That said… to the main purpose of that mail:
>>
>> - Default format for logs
>> A first question would be "are we all OK with the default output format"
>> - I'm pretty sure "humans" are happy with that, as it's really
>> convenient to read and grep. But on a "standard" Openstack deploy, I'm
>> pretty sure one does not have only one controller, one ceph node and one
>> compute. Hence comes the log centralization, and with that, the log
>> indexation and treatments.
>>
>> For that, one might argue "I'm using plain files on my logger, and
>> grep-ing -r in them". That's a way to do things, and for that, plain,
>> flat logs are great.
>>
>> But… I'm pretty sure I'm not the only one wanting to use some kind of
>> ELK cluster for that kind of purpose. So the right question is: what
>> about switching the default log format to JSON? On my part, I don't see
>> "cons", only "pros", but my judgment is of course biased, as I'm "alone
>> in my corner". But what about you, Community?
> 
> The major con I see is that we don't require an ELK stack and reading
> JSON logs if you don't have one of those is probably worse, although I
> will admit I haven't spent much time reading our JSON-formatted logs. My
> experience with things that log in JSON has not generally been positive
> though.  It's just not as human-readable.

We're on the same line on that - hence the following proposal. I was
pretty sure switching the default format was a bad thing, but I had to
submit it in order to be complete ;).
Let's focus on the other one, as it's more friendly for everyone.

> 
> The other problem with changing log format defaults is that many people
> have already set up filters and processing based on the existing log
> format.  There are significant user impacts to changing that default.
> 
>>
>> - Provide a way to configure the output format/handler
>> While poking around in the puppet modules code, I didn't find any way to
>> set the output handler for the logs. For example, in puppet-nova³ we can
>> set a lot of things, but not the useful handler for the output.
>>
>> It would be really cool to get, for each puppet module, the capability
>> to set the handler so that one can just push some stuff in hiera, and
>> Voilà, we have JSON logs.
>>
>> Doing so would allow people to chose between the default  (current)
>> output, and something more "computable".
> 
> I think we should do this regardless.  There are valid arguments for
> people to want both log formats IMHO and we should allow people to use
> what they want.

If I understand the things correctly, that would require a "small"
change in every puppet modules so that it configures the service with
the proper log output. Any way to automate something on that? It might
be worth to do some PoC on a specific module maybe?


> 
>>
>> Of course, either proposal will require a nice code change in all puppet
>> modules (add a new parameter for the foo::logging class, and use that
>> new param in the configuration file, and so on), but at least people
>> will be able to actually chose.
>>
>> So, before opening an issue on each launchpad project (that would be…
>> long), I'd rather open the discussion in here and, eventually, come to
>> some nice, acceptable and accepted solution that would make the
>> Openstack Community happy :).
>>
>> Any thoughts?
>>
>> Thank you for your attention, feedback and wonderful support for that
>> monster project :).
>>
>> Cheers,
>>
>> C.
>>
>>
>> ¹
>> https://github.com/openstack/oslo.log/blob/master/oslo_log/formatters.py#L166-L235
>>
>> ²
>> http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/%23openstack-oslo.2017-11-01.log.html#t2017-11-01T13:23:14
>>
>> ³
>> https://github.com/openstack/puppet-nova/blob/master/manifests/logging.pp
>>
>>
>>
>>
>> __________________________________________________________________________
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 878 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20171103/a0b6f935/attachment.sig>