[openstack-dev] [oslo][tripleo][puppet] Logging format: let's discuss a bit about default format, format configuration and so on

Juan Antonio Osorio jaosorior at gmail.com
Mon Nov 6 13:18:33 UTC 2017


Proposed a patch to be able to enable the JSON formatter via an oslo.log
configuration parameter:

https://review.openstack.org/#/c/517882/

On Mon, Nov 6, 2017 at 9:43 AM, Cédric Jeanneret <
cedric.jeanneret at camptocamp.com> wrote:

> On 11/06/2017 08:36 AM, Juan Antonio Osorio wrote:
> > Giving this a bit more thought; I'm slightly more inclined on merely
> adding
> > the JSON formatter option to the standard oslo.log configuration options.
>
> Fine for me.
>
> >
> > This is because we right now have the ability to pass oslo.cfg options
> via
> > the CLI, and we would loose that with the advanced logging configuration
> > file. The aforementioned option is something that we're using for
> > containerized openstack services.
>
> OK - I'll check on my own if I can get something nice with the
> logging.conf file. But that won't be for "tomorrow", as I'm not
> full-time on openstack (unfortunately :(). Both solutions have their
> pros and cons in the end.
>
> >
> > I'll look into adding the ability to turn that handler on/off from
> oslo.log.
>
> Ping me if I can help :). And big thanks for that coming change!
>
> >
> >
> >
> > On Mon, Nov 6, 2017 at 8:34 AM, Juan Antonio Osorio <jaosorior at gmail.com
> >
> > wrote:
> >
> >>
> >>
> >> On Mon, Nov 6, 2017 at 8:04 AM, Cédric Jeanneret <
> >> cedric.jeanneret at camptocamp.com> wrote:
> >>
> >>> On 11/04/2017 07:08 AM, Doug Hellmann wrote:
> >>>> Excerpts from Juan Antonio Osorio's message of 2017-11-04 00:11:47
> >>> +0200:
> >>>>>> On 3 Nov 2017 19:59, "Doug Hellmann" <doug at doughellmann.com> wrote:
> >>>>>>
> >>>>>> Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34
> +0100:
> >>>>>>> Dear Stackers,
> >>>>>>>
> >>>>>>> While working on my locally deployed Openstack (Pike using
> TripleO),
> >>> I
> >>>>>>> was a bit struggling with the logging part. Currently, all logs are
> >>>>>>> pushed to per-service files, in the standard format "one line per
> >>>>>>> entry", plain flat text.
> >>>>>>>
> >>>>>>> It's nice, but if one is wanting to push and index those logs in an
> >>> ELK,
> >>>>>>> the current, default format isn't really good.
> >>>>>>>
> >>>>>>> After some discussions about oslo.log, it appears it provides a
> nice
> >>>>>>> JSONFormatter handler¹ one might want to use for each (python)
> >>> service
> >>>>>>> using oslo central library.
> >>>>>>>
> >>>>>>> A JSON format is really cool, as it's easy to parse for machines,
> >>> and it
> >>>>>>> can be on a multi-line without any bit issue - this is especially
> >>>>>>> important for stack traces, as their output is multi-line without
> >>> real
> >>>>>>> way to have a common delimiter so that we can re-format it and feed
> >>> it
> >>>>>>> to any log parser (logstash, fluentd, …).
> >>>>>>>
> >>>>>>> After some more talks, olso.log will not provide a unified
> interface
> >>> in
> >>>>>>> order to output all received logs as JSON - this makes sens, as
> that
> >>>>>>> would mean "rewrite almost all the python logging management
> >>>>>>> interface"², and that's pretty useless, since (all?) services have
> >>> their
> >>>>>>> own "logging.conf" file.
> >>>>>>>
> >>>>>>> That said… to the main purpose of that mail:
> >>>>>>>
> >>>>>>> - Default format for logs
> >>>>>>> A first question would be "are we all OK with the default output
> >>> format"
> >>>>>>> - I'm pretty sure "humans" are happy with that, as it's really
> >>>>>>> convenient to read and grep. But on a "standard" Openstack deploy,
> >>> I'm
> >>>>>>> pretty sure one does not have only one controller, one ceph node
> and
> >>> one
> >>>>>>> compute. Hence comes the log centralization, and with that, the log
> >>>>>>> indexation and treatments.
> >>>>>>>
> >>>>>>> For that, one might argue "I'm using plain files on my logger, and
> >>>>>>> grep-ing -r in them". That's a way to do things, and for that,
> plain,
> >>>>>>> flat logs are great.
> >>>>>>>
> >>>>>>> But… I'm pretty sure I'm not the only one wanting to use some kind
> of
> >>>>>>> ELK cluster for that kind of purpose. So the right question is:
> what
> >>>>>>> about switching the default log format to JSON? On my part, I don't
> >>> see
> >>>>>>> "cons", only "pros", but my judgment is of course biased, as I'm
> >>> "alone
> >>>>>>> in my corner". But what about you, Community?
> >>>>>>>
> >>>>>>> - Provide a way to configure the output format/handler
> >>>>>>> While poking around in the puppet modules code, I didn't find any
> >>> way to
> >>>>>>> set the output handler for the logs. For example, in puppet-nova³
> we
> >>> can
> >>>>>>> set a lot of things, but not the useful handler for the output.
> >>>>>>>
> >>>>>>> It would be really cool to get, for each puppet module, the
> >>> capability
> >>>>>>> to set the handler so that one can just push some stuff in hiera,
> and
> >>>>>>> Voilà, we have JSON logs.
> >>>>>>>
> >>>>>>> Doing so would allow people to chose between the default  (current)
> >>>>>>> output, and something more "computable".
> >>>>>>
> >>>>>> Using the JSON formatter currently requires setting a logging
> >>>>>> configuration file using the standard library configuration format
> >>>>>> and fully specifying things like log levels, handlers, and output
> >>>>>> destination. Would it make sense to add an option in oslo.log to
> >>>>>> give deployers an easier way to enable the JSON formatter?
> >>>>>>
> >>>>>
> >>>>> This would actually be very useful.
> >>>>
> >>>> Great! Let me know if you would like to help figuring out where to do
> >>>> that in the library code.
> >>>
> >>> There's already some (not really documented) feature allowing oslo.log
> >>> to load a configuration file. In fact, there's already one existing in
> >>> the keystone tree (/etc/keystone/logging.conf - altougn not loaded) -
> we
> >>> might base something "casual" and "generic" on that one.
> >>>
> >>> I think all wsgi/python services should configure the logging through
> >>> that file so that it's clearer and cleaner. But that part is maybe a
> bit
> >>> too big and "aggressive" :). And the logging configuration isn't that
> >>> friendly, to be honest, at least I have some issues with its doc ;).
> >>>
> >>> But I think we might come to something nice and cool. It would allow
> >>> anyone to push  their own log "formatter", in the end.
> >>> So you (oslo.log) wouldn't need to work with format output requests
> >>> anymore, just provide two basics (plain and json), and let users play
> >>> with the logging configuration (and even output things in XML if they
> >>> want) ;).
> >>>
> >>> Regarding oslo.log: apparently, adding the following entry in the
> >>> service configuration file should do it:
> >>> log-config-append¹
> >>>
> >>> Can anyone confirm that?
> >>>
> >>
> >> It seems to be the case at least from the docs in the option [1]. But if
> >> we use this file (in TripleO and puppet) we really need to make it
> >> backwards compatible. Would folks be OK with taking it into use? I guess
> >> what it would take would be to document better the usage of this
> advanced
> >> configuration. Taking it into use doesn't look that bad; If you look at
> the
> >> file where the options are being set (_options.py), you will see that
> there
> >> are several options that get ignored once we start using this file [2].
> To
> >> see all the options that get ignored you can look at the instances of
> >> _IGNORE_MESSAGE. So, if we would start taking that into use, we would
> need
> >> to change the parameters of the oslo::log resource in puppet [3] to also
> >> configure an equivalent option to that advanced logging file.
> >>
> >> [1] https://github.com/openstack/oslo.log/blob/master/oslo_log/_
> >> options.py#L44
> >> [2] https://github.com/openstack/oslo.log/blob/master/oslo_log/_
> >> options.py#L32
> >> [3] https://github.com/openstack/puppet-oslo/blob/master/
> manifests/log.pp
> >>
> >>>
> >>>
> >>> ¹ https://github.com/openstack/oslo.log/blob/master/oslo_log/_
> >>> options.py#L44
> >>>
> >>>>
> >>>> Doug
> >>>>
> >>>> ____________________________________________________________
> >>> ______________
> >>>> OpenStack Development Mailing List (not for usage questions)
> >>>> Unsubscribe: OpenStack-dev-request at lists.op
> >>> enstack.org?subject:unsubscribe
> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>
> >>>
> >>>
> >>>
> >>> ____________________________________________________________
> >>> ______________
> >>> OpenStack Development Mailing List (not for usage questions)
> >>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:
> unsubscrib
> >>> e
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>
> >>>
> >>
> >>
> >> --
> >> Juan Antonio Osorio R.
> >> e-mail: jaosorior at gmail.com
> >>
> >>
> >
> >
> >
> >
> > ____________________________________________________________
> ______________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:
> unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Juan Antonio Osorio R.
e-mail: jaosorior at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20171106/fee88441/attachment.html>


More information about the OpenStack-dev mailing list