[openstack-dev] [openstack-qa] Post job failures

Joe Gordon joe.gordon0 at gmail.com
Thu Oct 9 22:05:26 UTC 2014


What about using graphite + logstash to power a post-job
/nightly-job/post-merge-periodic (the new thing we talked about in Germany)
dashboard?

There are a few different use cases for a dashboard for jobs that don't
report on gerrit changes.

* Track the success an failure rates over time
  * If I am maintaining a a job that doesn't vote anywhere, I will check
this daily
  * If I am part of the core team of a project where one feature is tested
post-merge, I want to periodically check this to see if that feature is
being maintained.
* Provide links to logs for failed jobs so the cause of the failure can be
investigated


We can do all this with graphite on logstash. Graphite for the tracking the
trends (something like http://jogo.github.io/gate/) and logstash to find
the logs for failed jobs (we can get around the 10 day logstash window by
saving the results instead of overwriting them every time we regenerate the
list of log links)

And if we really want some sort of alerts, there are a lot of graphite
tools (http://graphite.readthedocs.org/en/latest/tools.html) that can give
us alerts on metrics (alert me if the last X runs of job-foo-bar failed).


On Wed, Oct 1, 2014 at 9:46 AM, Jeremy Stanley <fungi at yuggoth.org> wrote:

> On 2014-10-01 10:39:40 -0400 (-0400), Matthew Treinish wrote:
> [...]
> > So I actually think as a first pass this would be the best way to
> > handle it. You can leave comments on a closed gerrit changes,
> [...]
>
> Not so easy as it sounds. Jobs in post are running on an arbitrary
> Git commit (more often than not, a merge commit), and mapping that
> back to a change in Gerrit is nontrivial.
> --
> Jeremy Stanley
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20141009/08a83029/attachment.html>


More information about the OpenStack-dev mailing list