Open Stack

On Mon, Nov 6, 2017 at 8:04 AM, Cédric Jeanneret <
cedric.jeanneret at camptocamp.com> wrote:

> On 11/04/2017 07:08 AM, Doug Hellmann wrote:
> > Excerpts from Juan Antonio Osorio's message of 2017-11-04 00:11:47 +0200:
> >>> On 3 Nov 2017 19:59, "Doug Hellmann" <doug at doughellmann.com> wrote:
> >>>
> >>> Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 +0100:
> >>>> Dear Stackers,
> >>>>
> >>>> While working on my locally deployed Openstack (Pike using TripleO), I
> >>>> was a bit struggling with the logging part. Currently, all logs are
> >>>> pushed to per-service files, in the standard format "one line per
> >>>> entry", plain flat text.
> >>>>
> >>>> It's nice, but if one is wanting to push and index those logs in an
> ELK,
> >>>> the current, default format isn't really good.
> >>>>
> >>>> After some discussions about oslo.log, it appears it provides a nice
> >>>> JSONFormatter handler¹ one might want to use for each (python) service
> >>>> using oslo central library.
> >>>>
> >>>> A JSON format is really cool, as it's easy to parse for machines, and
> it
> >>>> can be on a multi-line without any bit issue - this is especially
> >>>> important for stack traces, as their output is multi-line without real
> >>>> way to have a common delimiter so that we can re-format it and feed it
> >>>> to any log parser (logstash, fluentd, …).
> >>>>
> >>>> After some more talks, olso.log will not provide a unified interface
> in
> >>>> order to output all received logs as JSON - this makes sens, as that
> >>>> would mean "rewrite almost all the python logging management
> >>>> interface"², and that's pretty useless, since (all?) services have
> their
> >>>> own "logging.conf" file.
> >>>>
> >>>> That said… to the main purpose of that mail:
> >>>>
> >>>> - Default format for logs
> >>>> A first question would be "are we all OK with the default output
> format"
> >>>> - I'm pretty sure "humans" are happy with that, as it's really
> >>>> convenient to read and grep. But on a "standard" Openstack deploy, I'm
> >>>> pretty sure one does not have only one controller, one ceph node and
> one
> >>>> compute. Hence comes the log centralization, and with that, the log
> >>>> indexation and treatments.
> >>>>
> >>>> For that, one might argue "I'm using plain files on my logger, and
> >>>> grep-ing -r in them". That's a way to do things, and for that, plain,
> >>>> flat logs are great.
> >>>>
> >>>> But… I'm pretty sure I'm not the only one wanting to use some kind of
> >>>> ELK cluster for that kind of purpose. So the right question is: what
> >>>> about switching the default log format to JSON? On my part, I don't
> see
> >>>> "cons", only "pros", but my judgment is of course biased, as I'm
> "alone
> >>>> in my corner". But what about you, Community?
> >>>>
> >>>> - Provide a way to configure the output format/handler
> >>>> While poking around in the puppet modules code, I didn't find any way
> to
> >>>> set the output handler for the logs. For example, in puppet-nova³ we
> can
> >>>> set a lot of things, but not the useful handler for the output.
> >>>>
> >>>> It would be really cool to get, for each puppet module, the capability
> >>>> to set the handler so that one can just push some stuff in hiera, and
> >>>> Voilà, we have JSON logs.
> >>>>
> >>>> Doing so would allow people to chose between the default  (current)
> >>>> output, and something more "computable".
> >>>
> >>> Using the JSON formatter currently requires setting a logging
> >>> configuration file using the standard library configuration format
> >>> and fully specifying things like log levels, handlers, and output
> >>> destination. Would it make sense to add an option in oslo.log to
> >>> give deployers an easier way to enable the JSON formatter?
> >>>
> >>
> >> This would actually be very useful.
> >
> > Great! Let me know if you would like to help figuring out where to do
> > that in the library code.
>
> There's already some (not really documented) feature allowing oslo.log
> to load a configuration file. In fact, there's already one existing in
> the keystone tree (/etc/keystone/logging.conf - altougn not loaded) - we
> might base something "casual" and "generic" on that one.
>
> I think all wsgi/python services should configure the logging through
> that file so that it's clearer and cleaner. But that part is maybe a bit
> too big and "aggressive" :). And the logging configuration isn't that
> friendly, to be honest, at least I have some issues with its doc ;).
>
> But I think we might come to something nice and cool. It would allow
> anyone to push  their own log "formatter", in the end.
> So you (oslo.log) wouldn't need to work with format output requests
> anymore, just provide two basics (plain and json), and let users play
> with the logging configuration (and even output things in XML if they
> want) ;).
>
> Regarding oslo.log: apparently, adding the following entry in the
> service configuration file should do it:
> log-config-append¹
>
> Can anyone confirm that?
>

It seems to be the case at least from the docs in the option [1]. But if we
use this file (in TripleO and puppet) we really need to make it backwards
compatible. Would folks be OK with taking it into use? I guess what it
would take would be to document better the usage of this advanced
configuration. Taking it into use doesn't look that bad; If you look at the
file where the options are being set (_options.py), you will see that there
are several options that get ignored once we start using this file [2]. To
see all the options that get ignored you can look at the instances of
_IGNORE_MESSAGE. So, if we would start taking that into use, we would need
to change the parameters of the oslo::log resource in puppet [3] to also
configure an equivalent option to that advanced logging file.

[1]
https://github.com/openstack/oslo.log/blob/master/oslo_log/_options.py#L44
[2]
https://github.com/openstack/oslo.log/blob/master/oslo_log/_options.py#L32
[3] https://github.com/openstack/puppet-oslo/blob/master/manifests/log.pp

>
>
> ¹ https://github.com/openstack/oslo.log/blob/master/oslo_log/
> _options.py#L44
>
> >
> > Doug
> >
> > ____________________________________________________________
> ______________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:
> unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Juan Antonio Osorio R.
e-mail: jaosorior at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20171106/54320bf6/attachment.html>