Open Stack

On 09/27/2016 11:36 AM, Andrew Laski wrote:
> Hello all,
> 
> Recently I noticed that people would look at logs from a Zuul née
> Jenkins CI run and comment something like "there seem to be more
> warnings in here than usual." And so I thought it might be nice to
> quantify that sort of thing so we didn't have to rely on gut feelings.
> 
> So I threw together https://review.openstack.org/#/c/376531 which is a
> script that lives in the Nova tree, gets called from a devstack-gate
> post_test_hook, and outputs an n-stats.json file which can be seen at
> http://logs.openstack.org/06/375106/8/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/e103612/logs/n-stats.json.
> This provides just a simple way to compare two runs and spot large
> changes between them. Perhaps later things could get fancy and these
> stats could be tracked over time. I am also interested in adding stats
> for things that are a bit project specific like how long (max, min, med)
> it took to boot an instance, or what's probably better to track is how
> many operations that took for some definition of an operation.
> 
> I received some initial feedback that this might be a better fit in the
> os-loganalyze project so I took a look over there. So I cloned the
> project to take a look and quickly noticed
> http://git.openstack.org/cgit/openstack-infra/os-loganalyze/tree/README.rst#n13.
> That makes me think it would not be a good fit there because what I'm
> looking to do relies on parsing the full file, or potentially multiple
> files, in order to get useful data.
> 
> So my questions: does this seem like a good fit for os-loganalyze? If
> not is there another infra/QA project that this would be a good fit for?
> Or would people be okay with a lone project like Nova implementing this
> in tree for their own use?

Some things to keep in mind:

post_test_hook was really designed as a way for dedicated project
specific jobs to do something special instead of tempest. It currently
only can exist once in a job. While code in nova taking ownership for
nova only jobs is totally fine, it's a bit weird if the intent is to use
this on the integrated gate jobs that are shared between a bunch of
projects.

If we think that other projects are going to want to do similar things,
starting with a collaboration space for that up front would be useful.
Especially as bunch of it is going to be some shared log parsing.

If we think other projects want to do similar things, we can also move
to putting down json logs in the devstack runs in the gate, which would
make the parsing less guess work.

One of the reasons I had suggested something like dynamicly doing this
with os-loganalyze is that it provides the ability to ask a new question
of old data, on all the data that exists. The experience with Elastic
Recheck has been that odd regressions happen when no one is looking,
then being able to go back through history is really useful. That is one
mechanism to do it on demand.

	-Sean

-- 
Sean Dague
http://dague.net