Open Stack

Hello,

One of the major sets of issues currently affecting gate testing is
Libvirt stability. Elastic-recheck is tracking Libvirt crashes for us
and they happen frequently [0][1][2]. These issues appear to only affect
Ubuntu Xenial (and not Trusty or CentOS or Fedora) and after talking in
#openstack-nova it is clear that Libvirt isn't interested in debugging
such an old version of Libvirt (1.3.1). And while it isn't entirely
clear to me which exact version would be acceptable to them the Ubuntu
Cloud Archive (UCA) does publish a much newer Libvirt (2.5.0).

I have pushed a change to devstack [3] to enable using UCA which pulls
in new Libvirt and mostly seems to work. I think we should consider
switching to UCA as this may fix our Libvirt problems and if it doesn't,
we will be closer to a version of Libvirt that upstream should be
willing to fix.

This isn't the most straightfoward switch as UCA has a different repo
for each OpenStack release. libvirt-python is sensitve to the underlying
library changing; it is backward compatible but libvirt-python built
against older libvirt won't work against new libvirt. The result is a
libvirt-python wheel built on our wheel mirror does not work with UCA.
On the positive side both the OpenStack puppet modules and OpenStack
Ansible are using UCA with their deployment tooling so this should get
us closer to what people are using in the wild.

After some thought I think my preferred method of rolling this out would
be to blacklist libvirt-python from our wheel mirror building entirely
and force installs to happen from source so that we are base Xenial and
$UCA_version independent (local testing of these builds show it only
takes a few seconds). Then have specific jobs (like devstack) explicitly
opt into the UCA repo appropriate for them (if any). This last bit is
from feedback from OpenStack Ansible that having the base images be
fairly clean is desirable, but it would also be hard to know which
version of UCA is appropriate for our Xenial images (this likely differs
based on the job).

Now its entirely possible that newer Libvirt will be worse than current
(old) Libvirt; however, being closer to upstream should make getting
fixes easier. Would be great if those with a better understanding of
Libvirt could chime in on this if I am completely wrong here.

Finally it is worth noting that we will get newer packages of other
software as well, most notably openvswitch will be version 2.6.1 instead
of 2.5.0.

Any other thoughts or ideas?

[0] http://status.openstack.org/elastic-recheck/#1646779
[1] http://status.openstack.org/elastic-recheck/#1643911
[2] http://status.openstack.org/elastic-recheck/#1638982
[3] https://review.openstack.org/451492

Thank you,
Clark