[openstack-dev] os-loganalyze, project log parsing, or ...

Andrew Laski andrew at lascii.com
Tue Sep 27 22:19:17 UTC 2016



On Tue, Sep 27, 2016, at 05:54 PM, Sean Dague wrote:
> On 09/27/2016 05:45 PM, Andrew Laski wrote:
> > 
> > 
> > On Tue, Sep 27, 2016, at 04:43 PM, Matthew Treinish wrote:
> > <snip>
> >>>>
> >>>> I definitely can see the value in having machine parsable log stats in
> >>>> our
> >>>> artifacts, but I'm not sure where project specific pieces would come
> >>>> from. But,
> >>>> given that hypothetical I would say as long as you made those pieces
> >>>> configurable (like a yaml syntax to search for patterns by log file or
> >>>> something) and kept a generic framework/tooling for parsing the log stats
> >>>> I
> >>>> think it's still a good fit for a QA or Infra project. Especially if you
> >>>> think
> >>>> whatever pattern you're planning to use is something other projects would
> >>>> want
> >>>> to reuse.
> >>>
> >>> My concern here is that I want to go beyond simple pattern matching. I
> >>> want to be able to maintain state while parsing to associate log lines
> >>> with events that came before. The project specific bits I envision are
> >>> the logic to handle that, but I don't think yaml is expressive enough
> >>> for it. I came up with a quick example at
> >>> http://paste.openstack.org/show/583160/ . That's Nova specific and
> >>> beyond my capability to express in yaml or elastic-recheck.
> >>
> >> That's pretty simple to do with yaml too. Especially since it's tied to a
> >> single
> >> regex. For example, something roughly like:
> >>
> >> http://paste.openstack.org/show/583165/
> >>
> >> would use yaml to make it a generic framework. 
> > 
> > Okay, I see that it can be done for this example. But I look at that
> > yaml and all I see is that after a couple of definitions:
> > 
> > count_api_stuffs:
> >           - name: Instance Boots
> >             regex = '(req-\S+).*Starting instance'
> >             log_file = n-cpu.txt
> >           - name: Volume something
> >             regex = '(req-\S+).*Random Cinder Event to Count'
> >             log_file = c-api.txt
> > time_api_event:
> >            -name: Foo
> >            regex = <bleh>
> > another_thing:
> > yet_another:
> > 
> > there's no context to what each of these does and the logic is competely
> > decoupled. I guess I'm just personally biased to wanting to see it done
> > in code to help my understanding.
> > 
> > If yaml is what makes this acceptable then maybe I'll rework to do it
> > that way eventually. What I was hoping to do was start small, prove the
> > concept in a project while maintaining flexibility, and then look at
> > expanding for general use. Essentially the oslo model. I'm a little
> > surprised that there's so much resistance to that.
> 
> I think the crux of the resistance isn't to starting small. It's to
> doing all the gate plumbing for a thing in one tree, then do the replumb
> later. And it mostly comes from having to connect all those pieces then
> stage them back out in a non breaking way later.
> 
> If this was stuff where the execution control was also embedded in
> tox.ini, so changes in calling were easily addressed there, then it
> could be run in a pretty quiet controlled experiment.
> 
> But it pretty much starts multi project from day one, because it needs
> specific things (and guaruntees) from devstack and devstack-gate from
> the get go. So thinking about this as a multi project effort from the
> get go seems worth while.

I totally understand where you're coming from. I just see it
differently.

The way it was done did not affect any other projects, and the plumbing
used is something that could have easily be left in. And as luck would
have it there's another patch up to use that same plumbing so it's
probably going in anyways.

I completely agree that before expanding this to be in any way cross
project it's worth figuring out a better way to do it. But at this point
I don't feel comfortable enough with a long term vision to tackle that.
I would much prefer to experiment in a small way before moving forward.

> 
> 	-Sean
> 
> -- 
> Sean Dague
> http://dague.net
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



More information about the OpenStack-dev mailing list