Open Stack

On Tue, Sep 27, 2016 at 01:03:35PM -0400, Andrew Laski wrote:
> 
> 
> On Tue, Sep 27, 2016, at 12:39 PM, Matthew Treinish wrote:
> > On Tue, Sep 27, 2016 at 11:36:07AM -0400, Andrew Laski wrote:
> > > Hello all,
> > > 
> > > Recently I noticed that people would look at logs from a Zuul née
> > > Jenkins CI run and comment something like "there seem to be more
> > > warnings in here than usual." And so I thought it might be nice to
> > > quantify that sort of thing so we didn't have to rely on gut feelings.
> > > 
> > > So I threw together https://review.openstack.org/#/c/376531 which is a
> > > script that lives in the Nova tree, gets called from a devstack-gate
> > > post_test_hook, and outputs an n-stats.json file which can be seen at
> > > http://logs.openstack.org/06/375106/8/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/e103612/logs/n-stats.json.
> > > This provides just a simple way to compare two runs and spot large
> > > changes between them. Perhaps later things could get fancy and these
> > > stats could be tracked over time. I am also interested in adding stats
> > > for things that are a bit project specific like how long (max, min, med)
> > > it took to boot an instance, or what's probably better to track is how
> > > many operations that took for some definition of an operation.
> > > 
> > > I received some initial feedback that this might be a better fit in the
> > > os-loganalyze project so I took a look over there. So I cloned the
> > > project to take a look and quickly noticed
> > > http://git.openstack.org/cgit/openstack-infra/os-loganalyze/tree/README.rst#n13.
> > > That makes me think it would not be a good fit there because what I'm
> > > looking to do relies on parsing the full file, or potentially multiple
> > > files, in order to get useful data.
> > > 
> > > So my questions: does this seem like a good fit for os-loganalyze? If
> > > not is there another infra/QA project that this would be a good fit for?
> > > Or would people be okay with a lone project like Nova implementing this
> > > in tree for their own use?
> > > 
> > 
> > I think having this in os-loganalyze makes sense since we use that for
> > visualizing the logs already. It also means we get it for free on all the
> > log
> > files. But, if it's not a good fit for a technical reason then I think
> > creating
> > another small tool under QA or infra would be a good path forward. Since
> > there
> > really isn't anything nova specific in that.
> 
> There's nothing Nova specific atm because I went for low hanging fruit.
> But if the plan is to have Nova specific, Cinder specific, Glance
> specific, etc... things in there do people still feel that a QA/infra
> tool is the right path forward. That's my only hesitation here.

Well I think that raises more questions, what do you envision the nova specific
bits would be. The only thing I could see would be something that looks for
specific log messages or patterns in the logs. Which feels like exactly what
elastic-recheck does?

I definitely can see the value in having machine parsable log stats in our
artifacts, but I'm not sure where project specific pieces would come from. But,
given that hypothetical I would say as long as you made those pieces
configurable (like a yaml syntax to search for patterns by log file or
something) and kept a generic framework/tooling for parsing the log stats I
think it's still a good fit for a QA or Infra project. Especially if you think
whatever pattern you're planning to use is something other projects would want
to reuse.

-Matt Treinish


> 
> > 
> > I would caution against doing it as a one off in a project repo doesn't
> > seem
> > like the best path forward for something like this. We actually tried to
> > do
> > something similar to that in the past inside the tempest repo:
> > 
> > http://git.openstack.org/cgit/openstack/tempest/tree/tools/check_logs.py
> > 
> > and
> > 
> > http://git.openstack.org/cgit/openstack/tempest/tree/tools/find_stack_traces.py
> > 
> > all it did was cause confusion because no one knew where the output was
> > coming
> > from. Although, the output from those tools was also misleading, which
> > was
> > likely a bigger problm. So this probably won't be an issue if you add a
> > json
> > output to the jobs.
> > 
> > I also wonder if the JSONFormatter from oslo.log:
> > 
> > http://docs.openstack.org/developer/oslo.log/api/formatters.html#oslo_log.formatters.JSONFormatter
> > 
> > would be useful here. We can proabbly turn that on if it makes things
> > easier.
> > 
> > -Matt Treinish
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > Email had 1 attachment:
> > + signature.asc
> >   1k (application/pgp-signature)
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160927/7f7aa33c/attachment.pgp>