Open Stack

On Wed, Jul 16, 2014 at 10:15 AM, Sean Dague <sean at dague.net> wrote:

> Recently the main gate updated from Ubuntu 12.04 to 14.04, and in doing
> so we started executing the livesnapshot code in the nova libvirt
> driver. Which fails about 20% of the time in the gate, as we're bringing
> computes up and down while doing a snapshot. Dan Berange did a bunch of
> debug on that and thinks it might be a qemu bug. We disabled these code
> paths, so live snapshot has now been ripped out.
>
> In January we also triggered a libvirt bug, and had to carry a private
> build of libvirt for 6 weeks in order to let people merge code in
> OpenStack.
>
> We never were able to switch to libvirt 1.1.1 in the gate using the
> Ubuntu Cloud Archive during Icehouse development, because it has a
> different set of failures that would have prevented people from merging
> code.
>
> Based on these experiences, libvirt version differences seem to be as
> substantial as major hypervisor differences. There is a proposal here -
> https://review.openstack.org/#/c/103923/ to hold newer versions of
> libvirt to the same standard we hold xen, vmware, hyperv, docker,
> ironic, etc.
>
> I'm somewhat concerned that the -2 pile on in this review is a double
> standard of libvirt features, and features exploiting really new
> upstream features. I feel like a lot of the language being used here
> about the burden of doing this testing is exactly the same as was
> presented by the docker team before their driver was removed, which was
> ignored by the Nova team at the time. It was the concern by the freebsd
> team, which was also ignored and they were told to go land libvirt
> patches instead.
>

For running our own CI, the burden was largely a matter of resource and
time constraints for individual contributors and/or startups to setup and
maintain 3rd-party CI, especially in light of a parallel requirement to
pass the CI itself. I received community responses that equated to, "if you
were serious, you'd dedicate several full-time developers and/or
infrastructure engineers available for OpenStack development, plus several
thousand a month in infrastructure itself".  For Docker, these were simply
not options. Back in January, putting 2-3 engineers fulltime toward
OpenStack would have been a contribution of 10-20% of our engineering
force. OpenStack is not more important to us than Docker itself.

This thread highlights more deeply the problems for the FreeBSD folks.
First, I still disagree with the recommendation that they contribute to
libvirt. It's a classic example of creating two or more problems from one.
Once they have support in libvirt, how long before their code is in a
version of libvirt acceptable to Nova? When they hit edge-cases or bugs,
requiring changes in libvirt, how long before those fixes are accepted by
Nova?

I concur with thoughts in the Gerrit review which suggest there should be a
non-voting gate for testing against the latest libvirt.

I think the ideal situation would be to functionally test against multiple
versions of libvirt. We'd have at least two versions: "trunk,
latest-stable". We might want "trunk, trunk-snapshot-XYZ, latest-stable,
version-in-ubuntu, version-in-rhel", or any number of back-versions
included in the gate. The version-in-rhel and version-in-ubuntu might be
good candidates for 3rd-party CI.


Regards,
Eric Windisch
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140716/557633d4/attachment.html>