Open Stack

On Thu, May 14, 2015 at 02:23:25PM -0500, Matt Riedemann wrote:
> The minimum required version of libvirt in the driver is 0.9.11 still [1].
> We've been gating against 1.2.2 in Ubuntu Trusty 14.04 since Juno.
> 
> The libvirt distro support matrix is here: [2]
> 
> Can we safely assume the people aren't going to be running Libvirt compute
> nodes on RHEL < 7.1 or Ubuntu Precise?

I don't really think so - at the very least Fedora 20 and RHEL 7.0 are still
actively supported platforms by their vendors, which both have older libvirt
versions (1.1.3 and 1.1.1 respectively).

I'm not sure whether SUSE team consider any of the 12.x versions to still be
actively supported platforms or not, likewise which 13.x versions are under
active support.


> Regarding RHEL, I think this is a safe bet because in Kilo nova dropped
> python 2.6 support and RHEL > 6 doesn't have py26 so you'd be in trouble
> running kilo+ nova on RHEL 6.x anyway.

There are add-on repos for RHEL-6 and RHEL-7 that provide newer python
versions so (py27, various py3x), so it is not unreasonable for people
to consider sticking with RHEL-6 if that is what their existing deployment
is based on. Certainly new deployments though I'd expect to be RHEL-7 based.

> There are some workarounds in the code [3] I'd like to see removed by
> bumping the minimum required version.

Sure, its nice to remove workarounds from a cleanliness POV, but I'm generally
pretty conservative about doing so, because in the majority of case (while it
looks ugly) it is not really a significant burden on maintainers to keep it
around.

This example is really just that. It certainly looks ugly, but we have the
code there now, it is doing the job for people who have that problem and it
isn't really having any measurable impact on our ability to maintain the
libvirt code. Removing this code won't lessen our maintainance burden in
any way, but it will unquestionably impact our users by removing support for
the platform they may be currently deployed on.

The reason why we picked 0.9.11 as the current minimum version was that we
needed to be able to switch over to using libvirt-python from PyPi instead
of relying on the version that shipped with the distro. 0.9.11 is the min
version supported by libvirt-python on PyPi. This had significant benefits
to Nova maintainence, as it our gate jobs deploy libvirt-python from PyPi
in common with all other python packages we depend on. It also unlocked the
ability to run libvirt with python3 bindings.  There was some small amount
of pain for people running Ubuntu 12.04 LTS, but this was mitigated by the
fact that Canonical provided the Cloud Archive repositories for that LTS
version which gave users direct access to new enough libvirt. So in the end
users were not negatively impacted in any serious way - certainly not by
enough to outweigh the benefits Nova maintainence saw.


In this case, I just don't see compelling benefits to Nova libvirt maint
to justify increasing the minimum version to the level you suggest, and
it has a clear negative impact on our users which they will not be able
to easily deal with. They will be left with 3 options all of which are
unsatisfactory

 - Upgrade from RHEL-6 to RHEL-7 - a major undertaking for most organizations
 - Upgrade libvirt on RHEL-6 - they essentially take on the support burden
   for the hypervisor themselves, loosing support from the vendor
 - Re-add the code we remove from Nova - we've given users a maint burden,
   to rid ourselves of code that was posing no real maint burden on ourselves.


As a more general point, I think we are lacking clear guidance on our
policies around hypervisor platform support and thus have difficulty
in deciding when it is reasonable for us to drop support for platforms.

I think it is important to distinguish the hypervisor platform, from
the openstack platform because there are different support implications
there for users. With direct python dependancies for openstack, I we
are generally pretty aggressive at updating to newer versions of packages
from PyPi. There are a number of reasons why this is reasonable to do.
Foremost is that many python modules do a very bad job at maintaining
API compatibility, so we are essentially forced to upgrade and drop old
version support whether we like it or not. Second, the provisioning tools
we use (devstack, packstack, triple-o, and many vendor tools) all handle
deployment of arbitrary newer python modules without any trouble. OpenStack
itself and the vendors all do quite comprehensive testing on the python
modules we use, so we have confidence that the versions we're depending
on are functioning correctly.

The hypervisor platform is very different. While OpenStack does achieve
some level of testing coverage of the hypervisor platform version used
in the gate, this testing is inconsequential compared to the level of
testing that vendors put into their hypervisor platforms.

For example, vendors will test countless different guest operating systems.
For windows they will go through Microsoft WHQL certification of the guest
drivers. They will have Microsoft certification of the hypervisor platform
itself, and governemnt certifcations of the hypervisor platforms for Common
Criteria targets. There are countless performance tests run against the
platform. Security flaw scanning with tools like Coverity. Priority testing
and packaging of security updates. Cross functional testing of combinations
of storage and networking package versions that are in the distro.

So the idea that because OpenStack gate has tested libvirt 1.2.2 and QEMU
2.2.0, it is reasonable for users to upgrade their hypervisor platform
to that version does not pass muster. Users are faaaaaar better off sticking
with the versions that their vendor has provided on the platform, as the
level of testing they have had on their platform dwarves what OpenStack
gate has done. They are better off just performing a tempest run on their
own platform to sanity check that openstack works with their versions.

Even if users decide they want to upgrade their hypervisor platform to
a new version provided officially by the vendor, this is not always a
quick or easy task. Many organizations have non-trivial internal
testing and certification requirements before upgrading OS and/or hypervisor
platforms. The hardware they own has to be certified as compatible and
tested. They often have 3rd party monitoring and security auditing tools
that need to be upgraded and integrated with the new platform. They may
need todo long term stress tests to prove the new platform / hardare
combination is reliable at meeting their uptime requirements, so on.
So even with an active desire to upgrade their platform, it may take
anywhere from 3-6 months to actually put that plan into pratice. It may
seem strange, but at the same time they can be perfectly ok upgrading
openstack in a matter of weeks, or even doing continuous deployment,
simply because that is a layer above that does not directly impact on
the hardware or their base platform certification.

So while it is acceptable that we are aggressive at upgrading the python
layer in openstack, we need to be conservative at requiring upgrades of
the hypervisor layer. There needs to be a compelling reason why we must
drop an old platform. Generally we should aim to only drop old platforms
when the vendor themselves has end of lifed them.

Once all platforms with a particular old version of the hypervisor
packages are cut, we should ensure we give users advance notice of our
intent to drop their platform. I'd suggest that for this we should allow
one cycle of deprecation. eg, for liberty we could check version are
startup and print a warning if it was old, and then in Mxxxx we could
drop it.

As above though, I dont see any compelling reason to increase the min
libvirt to 1.2.2 at this time.

I could see us increase libvirt to 0.10.2 and qemu to 0.12.1.2as, assuming
opensuse 12.2 is end of life, then rhel-6 is the next oldest platform that
is still supported. For that I would make liberty print a warning on start
if libvirt was in the range 0.9.9 < 0.10.2, likewise for QEMU, and then
change the min version in  Mxxxx.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|