Open Stack

On 09/17/2014 08:48 PM, John Dickinson wrote:
> 
> On Sep 17, 2014, at 8:43 PM, Jay Faulkner <jay at jvf.cc> wrote:
> 
>> Comments inline.
>>
>>> -----Original Message-----
>>> From: Monty Taylor [mailto:mordred at inaugust.com]
>>> Sent: Wednesday, September 17, 2014 7:34 PM
>>> To: openstack-dev at lists.openstack.org
>>> Subject: Re: [openstack-dev] Log Rationalization -- Bring it on!
>>>
>>> On 09/17/2014 04:42 PM, Rochelle.RochelleGrober wrote:
>>>> TL;DR:  I consider the poor state of log consistency a major
>>>> impediment for more widespread adoption of OpenStack and would like to
>>>> volunteer to own this cross-functional process to begin to unify and
>>>> standardize logging messages and attributes for Kilo while dealing
>>>> with the most egregious issues as the community identifies them.
>>>>
>>>
>>> I fully support this, and I, for one, welcome our new log-standardization
>>> overlords.
>>>
>>
>> Something that could be interesting is to see if we can emit metrics everytime a loggable event happens. There's already a spec+code being drafted for Ironic in Kilo (https://review.openstack.org/#/c/100729/ &https://review.openstack.org/#/c/103202/) that we're using downstream to emit metrics from Ironic.
> 
> You may be interested to see how Swift has integrated StatsD events into a log adapter.
> 
> https://github.com/openstack/swift/blob/master/swift/common/utils.py#L1197
> 
> See also the StatsdClient class in that same file.

+1000

I'd so far as to say that everything should really have statsd
instrumentation.

> 
>>
>> If we have good organization of logging events, and levels, perhaps there's possibly a way to make it easy for metrics to be emitted at that time as well.
>>
>> -
>> Jay Faulkner
>>
>>>
>>>>
>>>> Recap from some mail threads:
>>>>
>>>>
>>>>
>>>> From Sean Dague on Kilo cycle goals:
>>>>
>>>> 2. Consistency in southbound interfaces (Logging first)
>>>>
>>>>
>>>>
>>>> Logging and notifications are south bound interfaces from OpenStack
>>>> providing information to people, or machines, about what is going on.
>>>>
>>>> There is also a 3rd proposed south bound with osprofiler.
>>>>
>>>>
>>>>
>>>> For Kilo: I think it's reasonable to complete the logging standards
>>>> and implement them. I expect notifications (which haven't quite kicked
>>>> off) are going to take 2 cycles.
>>>>
>>>>
>>>>
>>>> I'd honestly *really* love to see a unification path for all the the
>>>> southbound parts, logging, osprofiler, notifications, because there is
>>>> quite a bit of overlap in the instrumentation/annotation inside the
>>>> main code for all of these.
>>>>
>>>>
>>>> And from Doug Hellmann: 1. Sean has done a lot of analysis and started
>>>> a spec on standardizing logging guidelines where he is gathering input
>>>> from developers, deployers, and operators [1].
>>>> Because it is far enough for us to see real progress, it's a good
>>>> place for us to start experimenting with how to drive cross-project
>>>> initiatives involving code and policy changes from outside of a single
>>>> project. We have a couple of potentially related specs in Oslo as part
>>>> of the oslo.log graduation work [2] [3], but I think most of the work
>>>> will be within the applications.
>>>>
>>>> [1] https://review.openstack.org/#/c/91446/ [2]
>>>> https://blueprints.launchpad.net/oslo.log/+spec/app-agnostic-logging-p
>>>> arameters
>>>>
>>>>
>>> [3] https://blueprints.launchpad.net/oslo.log/+spec/remove-context-
>>> adapter
>>>>
>>>>
>>>>
>>>> And from James Blair:
>>>>
>>>> 1) Improve log correlation and utility
>>>>
>>>>
>>>>
>>>> If we're going to improve the stability of OpenStack, we have to be
>>>> able to understand what's going on when it breaks.  That's both true
>>>> as developers when we're trying to diagnose a failure in an
>>>> integration test, and it's true for operators who are all too often
>>>> diagnosing the same failure in a real deployment.  Consistency in
>>>> logging across projects as well as a cross-project request token would
>>>> go a long way toward this.
>>>>
>>>> While I am not currently managing an OpenStack deployment, writing
>>>> tests or code, or debugging the stack, I have spent many years doing
>>>> just that.  Through QA, Ops and Customer support, I have come to revel
>>>> in good logging and log messages and curse the holes and vagaries in
>>>> many systems.
>>>>
>>>> Defining/refining logs to be useful and usable is a cross-functional
>>>> effort that needs to include:
>>>>
>>>> ·         Operators
>>>>
>>>> ·         QA
>>>>
>>>> ·         End Users
>>>>
>>>> ·         Community managers
>>>>
>>>> ·         Tech Pubs
>>>>
>>>> ·         Translators
>>>>
>>>> ·         Developers
>>>>
>>>> ·         TC (which provides the forum and impetus for all the
>>>> projects to cooperate on this)
>>>>
>>>> At the moment, I think this effort may best work under the auspices of
>>>> Oslo (oslo.log), I'd love to hear other proposals.
>>>>
>>>> Here is the beginnings of my proposal of how to attack and subdue the
>>>> painful state of logs:
>>>>
>>>>
>>>> ·         Post this email to the MLs (dev, ops, enduser) to get
>>>> feedback, garner support and participants in the process (Done;-)
>>>>
>>>> ·         In parallel:
>>>>
>>>> o   Collect up problems, issues, ideas, solutions on an etherpad
>>>> https://etherpad.openstack.org/p/Log-Rationalization where anyone in
>>>> the communities can post.
>>>>
>>>> o   Categorize  reported Log issues into classes (already identified
>>>> classes):
>>>>
>>>> §  Format Consistency across projects
>>>>
>>>> §  Log level definition and categorization across classes
>>>>
>>>> §  Time syncing entries across tens of logfiles
>>>>
>>>> §  Relevancy/usefulness of information provided within messages
>>>>
>>>> §  Etc (missing a lot here, but I'm sure folks will speak up)
>>>>
>>>> o   Analyze existing log message formats, standards across integrated
>>>> projects
>>>>
>>>> o   File bugs where issues identified are actual project bugs
>>>>
>>>> o   Build a session outline for F2F working session at the Paris
>>>> Design Summit
>>>>
>>>> ·         At the Paris Design Summit, use a session and/or pod
>>>> discussions to set priorities, recruit contributors, start and/or
>>>> flesh out specs and blueprints
>>>>
>>>> ·         Proceed according to priorities, specs, blueprints,
>>>> contributions and changes as needed as the work progresses.
>>>>
>>>> ·         Keep an active and open rapport and reporting process for
>>>> the user community to comment and participate in the processes.
>>>> Measures of success:
>>>>
>>>> ·         Log messages provide consistency of format enough for
>>>> productive mining through operator writable scripts
>>>>
>>>> ·         Problem debugging is simplified through the ability to
>>>> trust timestamps across all OpenStack logs (and use scripts to get to
>>>> the time you want in any/all of the logfiles)
>>>>
>>>> ·         Standards for format, content, levels and translations have
>>>> been proposed and agreed to be adopted across all OpenStack integrated
>>>> projects
>>>>
>>>> ·         The user communities demonstrate an increased level of
>>>> trust and decreased level of frustration with OpenStack logging
>>>> (surveys, bug reports, other measures?)
>>>>
>>>> ·         The log team can disband
>>>>
>>>> I expect that getting the logs in very good shape will take more than
>>>> just the Kilo timeframe, but once momentum has built, which should
>>>> happened during Kilo, the process should move very quickly.  A lot of
>>>> this could be handled through "while you're in there" or "low hanging
>>>> fruit" once the standards are established.  The bigger win will be if
>>>> we can ensure what we define/design is extensible over the longer life
>>>> of OpenStack.
>>>>
>>>> --Rocky
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________ OpenStack-
>>> dev mailing
>>>> list OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>