Open Stack

On Fri, Nov 3, 2017 at 12:21 AM, Cédric Jeanneret
<cedric.jeanneret at camptocamp.com> wrote:
> On 11/02/2017 05:18 PM, Ben Nemec wrote:
>> Adding tags for the relevant projects.  I _think_ this is mostly
>> directed at Puppet/TripleO, but Oslo is obviously relevant as well.
>
> Thank you! First mail in there, wasn't really sure how to do that.
>
>>
>> On 11/01/2017 08:54 AM, Cédric Jeanneret wrote:
>>> Dear Stackers,
>>>
>>> While working on my locally deployed Openstack (Pike using TripleO), I
>>> was a bit struggling with the logging part. Currently, all logs are
>>> pushed to per-service files, in the standard format "one line per
>>> entry", plain flat text.
>>>
>>> It's nice, but if one is wanting to push and index those logs in an ELK,
>>> the current, default format isn't really good.
>>>
>>> After some discussions about oslo.log, it appears it provides a nice
>>> JSONFormatter handler¹ one might want to use for each (python) service
>>> using oslo central library.
>>>
>>> A JSON format is really cool, as it's easy to parse for machines, and it
>>> can be on a multi-line without any bit issue - this is especially
>>> important for stack traces, as their output is multi-line without real
>>> way to have a common delimiter so that we can re-format it and feed it
>>> to any log parser (logstash, fluentd, …).
>>>
>>> After some more talks, olso.log will not provide a unified interface in
>>> order to output all received logs as JSON - this makes sens, as that
>>> would mean "rewrite almost all the python logging management
>>> interface"², and that's pretty useless, since (all?) services have their
>>> own "logging.conf" file.
>>>
>>> That said… to the main purpose of that mail:
>>>
>>> - Default format for logs
>>> A first question would be "are we all OK with the default output format"
>>> - I'm pretty sure "humans" are happy with that, as it's really
>>> convenient to read and grep. But on a "standard" Openstack deploy, I'm
>>> pretty sure one does not have only one controller, one ceph node and one
>>> compute. Hence comes the log centralization, and with that, the log
>>> indexation and treatments.
>>>
>>> For that, one might argue "I'm using plain files on my logger, and
>>> grep-ing -r in them". That's a way to do things, and for that, plain,
>>> flat logs are great.
>>>
>>> But… I'm pretty sure I'm not the only one wanting to use some kind of
>>> ELK cluster for that kind of purpose. So the right question is: what
>>> about switching the default log format to JSON? On my part, I don't see
>>> "cons", only "pros", but my judgment is of course biased, as I'm "alone
>>> in my corner". But what about you, Community?
>>
>> The major con I see is that we don't require an ELK stack and reading
>> JSON logs if you don't have one of those is probably worse, although I
>> will admit I haven't spent much time reading our JSON-formatted logs. My
>> experience with things that log in JSON has not generally been positive
>> though.  It's just not as human-readable.
>
> We're on the same line on that - hence the following proposal. I was
> pretty sure switching the default format was a bad thing, but I had to
> submit it in order to be complete ;).
> Let's focus on the other one, as it's more friendly for everyone.
>
>>
>> The other problem with changing log format defaults is that many people
>> have already set up filters and processing based on the existing log
>> format.  There are significant user impacts to changing that default.
>>
>>>
>>> - Provide a way to configure the output format/handler
>>> While poking around in the puppet modules code, I didn't find any way to
>>> set the output handler for the logs. For example, in puppet-nova³ we can
>>> set a lot of things, but not the useful handler for the output.
>>>
>>> It would be really cool to get, for each puppet module, the capability
>>> to set the handler so that one can just push some stuff in hiera, and
>>> Voilà, we have JSON logs.
>>>
>>> Doing so would allow people to chose between the default  (current)
>>> output, and something more "computable".
>>
>> I think we should do this regardless.  There are valid arguments for
>> people to want both log formats IMHO and we should allow people to use
>> what they want.
>
> If I understand the things correctly, that would require a "small"
> change in every puppet modules so that it configures the service with
> the proper log output. Any way to automate something on that? It might
> be worth to do some PoC on a specific module maybe?
>

So for the puppet modules, the available logging configuration lives
in puppet-oslo so if there's a configuration around that that needs to
be updated[0] then that is the first change. From there you would need
to go through each module to expose this new configuration in the
logging classes. For example puppet-nova[1].  You could probably write
a script to modify each but I'm not sure they are completely
consistent between classes. I think they are but it would require some
validation.

[0] https://github.com/openstack/puppet-oslo/blob/master/manifests/log.pp
[1] https://github.com/openstack/puppet-nova/blob/master/manifests/logging.pp

>
>>
>>>
>>> Of course, either proposal will require a nice code change in all puppet
>>> modules (add a new parameter for the foo::logging class, and use that
>>> new param in the configuration file, and so on), but at least people
>>> will be able to actually chose.
>>>
>>> So, before opening an issue on each launchpad project (that would be…
>>> long), I'd rather open the discussion in here and, eventually, come to
>>> some nice, acceptable and accepted solution that would make the
>>> Openstack Community happy :).
>>>
>>> Any thoughts?
>>>
>>> Thank you for your attention, feedback and wonderful support for that
>>> monster project :).
>>>
>>> Cheers,
>>>
>>> C.
>>>
>>>
>>> ¹
>>> https://github.com/openstack/oslo.log/blob/master/oslo_log/formatters.py#L166-L235
>>>
>>> ²
>>> http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/%23openstack-oslo.2017-11-01.log.html#t2017-11-01T13:23:14
>>>
>>> ³
>>> https://github.com/openstack/puppet-nova/blob/master/manifests/logging.pp
>>>
>>>
>>>
>>>
>>> __________________________________________________________________________
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>