[openstack-dev] Our New Weekly(ish) Test Status Report

Matthew Treinish mtreinish at kortar.org
Tue Feb 28 16:49:53 UTC 2017


We have a few particularly annoying bugs that have been impacting the
reliability of gate testing recently. It would be great if we could get
volunteers to look at these bugs to improve the reliability of our testing as we
start working on Pike.

These two issues have been identified by elastic-recheck as being our biggest

1. SSH Banner bug http://status.openstack.org/elastic-recheck/#1349617

This bug is a longstanding issue that comes and goes and also has lots of very
similar (but subtly different) failure modes. Tempest attempts to ssh into the
cirros guest and is unable to after 18 attempts over the 300 sec timeout window
and fails to login. Paramiko reports that there was an issue reading the banner
returned on port 22 from the guest. This indicates that something is likely
responding on port 22. We're working on trying to get more details on what is
the cause here with:


2. Libvirt crashes: http://status.openstack.org/elastic-recheck/#1643911 and

Libvirt is randomly crashing during the job which causes things to fail (for
obvious reasons). To address this will likely require someone with experience
debugging libvirt since it's most likely a bug isolated to libvirt. Tonyb has
offered to start working on this so talk to him to coordinate efforts around
fixing this.

The other thing to note is the oom-killer bug:
http://status.openstack.org/elastic-recheck/gate.html#1656386 while there aren't
a lot of hits in logstash for this particular bug, it does raise an import issue
about the increased memory pressure on the test nodes. It's likely that a lot of
the instability may be related to the increased load on the nodes. As a starting
point all projects should look at their memory footprint and see where they can
trim things to try and make the situation better.

As a friendly reminder we do track bug rate incidence within our testing using
the elastic-recheck tool. You can find that data at
http://status.openstack.org/elastic-recheck. It can be quite useful to start
there when determining which bugs to fix based on impact. Elastic recheck also
maintains a list of failures that occurred without a known signature:

We also need some people to help maintain the list of existing queries, we have
a lot of queries for closed bugs that have no hits and others which are overly
broad and matching failures which are unrelated to the bug. This would also be
good task for a new person to start getting involved with. Feel free to submit
patches to:
https://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/queries to
track new issues.

Thank you,

mtreinish and clarkb
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170228/f01cae0a/attachment.pgp>

More information about the OpenStack-dev mailing list