Open Stack

On 2015-08-31 14:49:33 -0400 (-0400), Ronald Bradford wrote:
[...]
> fungi provided https://review.openstack.org/#/c/192253/ which is a
> draft spec for a much wider ci-watch effort however I see this a
> more specific example. I am seeking feedback from various projects
> on interest, concerns and recommendations before creating a spec.

Agreed, I simply brought up Sean's proposed spec since code coverage
job results fall within this class of (periodic and post pipeline)
result discoverability and analysis problems, so we should make sure
that whatever solution we choose for coverage reports specifically
is compatible with a broader solution and the spec review is a good
place to discuss that.

> I see the initial steps to implement this as.
> 
> 1. Agree on future domain url location for reports (e.g.
> coverage.openstack.org)

Well, we already have a URL location for the reports, and it's
logs.openstack.org. They're published there today. What we need is
some better indexing and perhaps automated analysis tools linking
into there, which could either be hosted from logs.o.o or some other
domain name. I just worry that "what URL do we want?" is descending
into bikeshed territory, since it's putting the _least_ important
decision first.

> 2. Add to the merge committing process a step similar to the
> existing coverage-log publisher
> (project-config/jenkins/jobs/macros.yaml) to agreed location.

Why publish the same file in two places?

> 3. Provide initially a top level index page at url for published
> reports

This seems like 99% of the need, and we've been in agreement for at
least a couple cycles that we'd love for someone to tackle getting
this designed and implemented.

> These steps enable a single current occurrence of code coverage
> per project. It is not the objective to keep the historical HTML
> output.

Though we do (for a few months at the moment, but we could work out
a way to keep summaries longer if there's interest).

> Following this, the next steps are to record coverage for
> historical analysis with the following.
> 
> 1. Alter tox.ini [testenv:cover] to produce a text based version
> (aka coverage report -m). This is much easier to parse and compare
> different versions. I would propose we keep a weekly/daily dated
> file of this output, which would be automatically published as per
> previous publisher.

Things are likely to stall here. We have 291 active repos with a
[testenv:cover] section in their tox.ini at the moment. Might be
worth considering whether we could extend
jenkins/scripts/run-cover.sh in openstack-infra/project-config so
that it does this part (assuming that devs are unlikely to want that
file most of the time anyway).

> 2. Use these dated files to produce basic per project analysis of
> lines/percentage coverage over time, publishing this to a set
> directory structure (e.g. /analysis/) and linked from the top
> level index page. Over time this can be improved to provide simple
> graphical output.

We already have a statsd server at graphite.openstack.org, so poking
values into it might be the simplest way forward. Then we can embed
graphs into the resulting dashboard interface from there similar to
what the http://status.openstack.org/zuul/ page does (all the graphs
on that page are generated real-time from data in graphite).
-- 
Jeremy Stanley