Open Stack

I've been keeping an eye on this thread, and it seems I actually have a few minutes to spend on a response today.

To first answer the specific question, while there are some minor technical concerns about oslo logging, the bigger concerns are non-technical. Some things I'm concerned about from a technical perspective are that it's not a separate module or package that can be imported, so it would probably currently require copy/paste code into the Swift codebase. My second concern is that there are log line elements that just don't seem to make sense like "instance". I'd be happy to be wrong on both of these items, and I want to make clear that these are not long-term issues. They are both solvable.

My bigger concern with using oslo logging in Swift is simply changing the request log format is something that cannot be done lightly. Request logs are a very real interface into the system, and changing the log format in a breaking way can cause major headaches for people relying on those logs for system health, billing, and other operational concerns.

One possible solution to this is to keep requests logged the same way, but add configuration options for all of the other things that are logged. Having two different logging systems (or multiple configurable log handlers) to do this seems to add a fair bit of complexity to me, especially when I'm not quite sure of the actual problem that's being solved. That said, adding in a different log format into Swift isn't a terrible idea by itself, but migration is a big concern of any implementation (and I know you'll find very strong feelings on this in gerrit if/when something is proposed).




Now back to the original topic of actual logging formats.

Here's (something like) what I'd like to see for a common log standard (ie Sean, what I think you were asking for comments on):

log_line = prefix message
prefix = timestamp project log_level
message = bytestream
timestamp = `eg the output of time.time()`
project = `one of {nova,swift,neutron,cinder,glance,etc}`

Now, there's plenty of opportunity to bikeshed what the actual log line would look like, but the general idea of what I want to see has 2 major parts:

1) Every log message is one line (ends with \n) and the log fields are space-delineated. eg (`log_line = ' '.join(urllib.quote(x) for x in log_fields_list)`)

2) The only definition of a log format is the prefix and the message is a set of fields defined by the service actually doing the logging.


--John




On Jan 30, 2014, at 10:11 AM, Sanchez, Cristian A <cristian.a.sanchez at intel.com> wrote:

> Is there any technical reason of why Swift does not use oslo logging?
> If not, I can work on incorporating that to Swift.
> 
> Thanks
> 
> Cristian
> 
> On 30/01/14 11:12, "Sean Dague" <sean at dague.net> wrote:
> 
>> For all projects that use oslo logging (which is currently everything
>> except swift), this works.
>> 
>> 	-Sean
>> 
>> On 01/30/2014 09:07 AM, Macdonald-Wallace, Matthew wrote:
>>> No idea, I only really work on Nova, but as this is in Oslo I expect so!
>>> 
>>> Matt
>>> 
>>>> -----Original Message-----
>>>> From: Sanchez, Cristian A [mailto:cristian.a.sanchez at intel.com]
>>>> Sent: 30 January 2014 13:44
>>>> To: OpenStack Development Mailing List (not for usage questions)
>>>> Subject: Re: [openstack-dev] Proposed Logging Standards
>>>> 
>>>> Hi Matt,
>>>> What about the rest of the components? Do they also have this
>>>> capability?
>>>> Thanks
>>>> 
>>>> Cristian
>>>> 
>>>> On 30/01/14 04:59, "Macdonald-Wallace, Matthew"
>>>> <matthew.macdonald-wallace at hp.com> wrote:
>>>> 
>>>>> Hi Cristian,
>>>>> 
>>>>> The functionality already exists within Openstack (certainly it's
>>>>> there
>>>>> in Nova) it's just not very well documented (something I keep meaning
>>>>> to
>>>>> do!)
>>>>> 
>>>>> Basically you need to add the following to your nova.conf file:
>>>>> 
>>>>> log_config=/etc/nova/logging.conf
>>>>> 
>>>>> And then create /etc/nova/logging.conf with the configuration you want
>>>>> to use based on the Python Logging Module's "ini" configuration
>>>>> format.
>>>>> 
>>>>> Hope that helps,
>>>>> 
>>>>> Matt
>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: Sanchez, Cristian A [mailto:cristian.a.sanchez at intel.com]
>>>>>> Sent: 29 January 2014 17:57
>>>>>> To: OpenStack Development Mailing List (not for usage questions)
>>>>>> Subject: Re: [openstack-dev] Proposed Logging Standards
>>>>>> 
>>>>>> Hi Matthew,
>>>>>> I¹m interested to help in this switch to python logging framework for
>>>>>> shipping to  logstash/etc. Are you working on a blueprint for this?
>>>>>> Cheers,
>>>>>> 
>>>>>> Cristian
>>>>>> 
>>>>>> On 27/01/14 11:07, "Macdonald-Wallace, Matthew"
>>>>>> <matthew.macdonald-wallace at hp.com> wrote:
>>>>>> 
>>>>>>> Hi Sean,
>>>>>>> 
>>>>>>> I'm currently working on moving away from the "built-in" logging to
>>>>>>> use log_config=<filename> and the python logging framework so that
>>>>>>> we can start shipping to logstash/sentry/<insert other useful tool
>>>>>>> here>.
>>>>>>> 
>>>>>>> I'd be very interested in getting involved in this, especially from
>>>>>>> a "why do we have log messages that are split across multiple lines"
>>>>>>> perspective!
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> 
>>>>>>> Matt
>>>>>>> 
>>>>>>> P.S. FWIW, I'd also welcome details on what the "Audit" level gives
>>>>>>> us that the others don't... :)
>>>>>>> 
>>>>>>>> -----Original Message-----
>>>>>>>> From: Sean Dague [mailto:sean at dague.net]
>>>>>>>> Sent: 27 January 2014 13:08
>>>>>>>> To: OpenStack Development Mailing List
>>>>>>>> Subject: [openstack-dev] Proposed Logging Standards
>>>>>>>> 
>>>>>>>> Back at the beginning of the cycle, I pushed for the idea of doing
>>>>>>>> some log  harmonization, so that the OpenStack logs, across
>>>>>>>> services, made sense.
>>>>>>>> I've
>>>>>>>> pushed a proposed changes to Nova and Keystone over the past
>>>>>>>> couple of days.
>>>>>>>> 
>>>>>>>> This is going to be a long process, so right now I want to just
>>>>>>>> focus on making  INFO level sane, because as someone that spends a
>>>>>>>> lot of time staring at logs in  test failures, I can tell you it
>>>>>>>> currently isn't.
>>>>>>>> 
>>>>>>>> https://wiki.openstack.org/wiki/LoggingStandards is a few things
>>>>>>>> I've written  down so far, comments welcomed.
>>>>>>>> 
>>>>>>>> We kind of need to solve this set of recommendations once and for
>>>>>>>> all up front,  because negotiating each change, with each project,
>>>>>>>> isn't going to work (e.g -
>>>>>>>> https://review.openstack.org/#/c/69218/)
>>>>>>>> 
>>>>>>>> What I'd like to find out now:
>>>>>>>> 
>>>>>>>> 1) who's interested in this topic?
>>>>>>>> 2) who's interested in helping flesh out the guidelines for
>>>>>>>> various log levels?
>>>>>>>> 3) who's interested in helping get these kinds of patches into
>>>>>>>> various projects in  OpenStack?
>>>>>>>> 4) which projects are interested in participating (i.e. interested
>>>>>>>> in prioritizing  landing these kinds of UX improvements)
>>>>>>>> 
>>>>>>>> This is going to be progressive and iterative. And will require
>>>>>>>> lots of folks  involved.
>>>>>>>> 
>>>>>>>> 	-Sean
>>>>>>>> 
>>>>>>>> --
>>>>>>>> Sean Dague
>>>>>>>> Samsung Research America
>>>>>>>> sean at dague.net / sean.dague at samsung.com http://dague.net
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> OpenStack-dev mailing list
>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> OpenStack-dev mailing list
>>>>>> OpenStack-dev at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>> 
>>>>> _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> 
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>> 
>> 
>> -- 
>> Sean Dague
>> Samsung Research America
>> sean at dague.net / sean.dague at samsung.com
>> http://dague.net
>> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140130/2a4daa13/attachment.pgp>