Open Stack

On 5/15/2015 6:28 AM, Daniel P. Berrange wrote:
> On Fri, May 15, 2015 at 11:51:22AM +0100, Daniel P. Berrange wrote:
>> On Thu, May 14, 2015 at 02:23:25PM -0500, Matt Riedemann wrote:
>>> The minimum required version of libvirt in the driver is 0.9.11 still [1].
>>> We've been gating against 1.2.2 in Ubuntu Trusty 14.04 since Juno.
>>>
>>> The libvirt distro support matrix is here: [2]
>>>
>>> Can we safely assume the people aren't going to be running Libvirt compute
>>> nodes on RHEL < 7.1 or Ubuntu Precise?
>>
>> I don't really think so - at the very least Fedora 20 and RHEL 7.0 are still
>> actively supported platforms by their vendors, which both have older libvirt
>> versions (1.1.3 and 1.1.1 respectively).
>>
>> I'm not sure whether SUSE team consider any of the 12.x versions to still be
>> actively supported platforms or not, likewise which 13.x versions are under
>> active support.
>>
>>
>>> Regarding RHEL, I think this is a safe bet because in Kilo nova dropped
>>> python 2.6 support and RHEL > 6 doesn't have py26 so you'd be in trouble
>>> running kilo+ nova on RHEL 6.x anyway.
>>
>> There are add-on repos for RHEL-6 and RHEL-7 that provide newer python
>> versions so (py27, various py3x), so it is not unreasonable for people
>> to consider sticking with RHEL-6 if that is what their existing deployment
>> is based on. Certainly new deployments though I'd expect to be RHEL-7 based.
>>
>>> There are some workarounds in the code [3] I'd like to see removed by
>>> bumping the minimum required version.
>>
>> Sure, its nice to remove workarounds from a cleanliness POV, but I'm generally
>> pretty conservative about doing so, because in the majority of case (while it
>> looks ugly) it is not really a significant burden on maintainers to keep it
>> around.
>>
>> This example is really just that. It certainly looks ugly, but we have the
>> code there now, it is doing the job for people who have that problem and it
>> isn't really having any measurable impact on our ability to maintain the
>> libvirt code. Removing this code won't lessen our maintainance burden in
>> any way, but it will unquestionably impact our users by removing support for
>> the platform they may be currently deployed on.
>>
>> The reason why we picked 0.9.11 as the current minimum version was that we
>> needed to be able to switch over to using libvirt-python from PyPi instead
>> of relying on the version that shipped with the distro. 0.9.11 is the min
>> version supported by libvirt-python on PyPi. This had significant benefits
>> to Nova maintainence, as it our gate jobs deploy libvirt-python from PyPi
>> in common with all other python packages we depend on. It also unlocked the
>> ability to run libvirt with python3 bindings.  There was some small amount
>> of pain for people running Ubuntu 12.04 LTS, but this was mitigated by the
>> fact that Canonical provided the Cloud Archive repositories for that LTS
>> version which gave users direct access to new enough libvirt. So in the end
>> users were not negatively impacted in any serious way - certainly not by
>> enough to outweigh the benefits Nova maintainence saw.
>>
>>
>> In this case, I just don't see compelling benefits to Nova libvirt maint
>> to justify increasing the minimum version to the level you suggest, and
>> it has a clear negative impact on our users which they will not be able
>> to easily deal with. They will be left with 3 options all of which are
>> unsatisfactory
>>
>>   - Upgrade from RHEL-6 to RHEL-7 - a major undertaking for most organizations
>>   - Upgrade libvirt on RHEL-6 - they essentially take on the support burden
>>     for the hypervisor themselves, loosing support from the vendor
>>   - Re-add the code we remove from Nova - we've given users a maint burden,
>>     to rid ourselves of code that was posing no real maint burden on ourselves.
>>
>>
>> As a more general point, I think we are lacking clear guidance on our
>> policies around hypervisor platform support and thus have difficulty
>> in deciding when it is reasonable for us to drop support for platforms.
>>
>> I think it is important to distinguish the hypervisor platform, from
>> the openstack platform because there are different support implications
>> there for users. With direct python dependancies for openstack, I we
>> are generally pretty aggressive at updating to newer versions of packages
>> from PyPi. There are a number of reasons why this is reasonable to do.
>> Foremost is that many python modules do a very bad job at maintaining
>> API compatibility, so we are essentially forced to upgrade and drop old
>> version support whether we like it or not. Second, the provisioning tools
>> we use (devstack, packstack, triple-o, and many vendor tools) all handle
>> deployment of arbitrary newer python modules without any trouble. OpenStack
>> itself and the vendors all do quite comprehensive testing on the python
>> modules we use, so we have confidence that the versions we're depending
>> on are functioning correctly.
>>
>> The hypervisor platform is very different. While OpenStack does achieve
>> some level of testing coverage of the hypervisor platform version used
>> in the gate, this testing is inconsequential compared to the level of
>> testing that vendors put into their hypervisor platforms.
>>
>> For example, vendors will test countless different guest operating systems.
>> For windows they will go through Microsoft WHQL certification of the guest
>> drivers. They will have Microsoft certification of the hypervisor platform
>> itself, and governemnt certifcations of the hypervisor platforms for Common
>> Criteria targets. There are countless performance tests run against the
>> platform. Security flaw scanning with tools like Coverity. Priority testing
>> and packaging of security updates. Cross functional testing of combinations
>> of storage and networking package versions that are in the distro.
>>
>> So the idea that because OpenStack gate has tested libvirt 1.2.2 and QEMU
>> 2.2.0, it is reasonable for users to upgrade their hypervisor platform
>> to that version does not pass muster. Users are faaaaaar better off sticking
>> with the versions that their vendor has provided on the platform, as the
>> level of testing they have had on their platform dwarves what OpenStack
>> gate has done. They are better off just performing a tempest run on their
>> own platform to sanity check that openstack works with their versions.
>>
>> Even if users decide they want to upgrade their hypervisor platform to
>> a new version provided officially by the vendor, this is not always a
>> quick or easy task. Many organizations have non-trivial internal
>> testing and certification requirements before upgrading OS and/or hypervisor
>> platforms. The hardware they own has to be certified as compatible and
>> tested. They often have 3rd party monitoring and security auditing tools
>> that need to be upgraded and integrated with the new platform. They may
>> need todo long term stress tests to prove the new platform / hardare
>> combination is reliable at meeting their uptime requirements, so on.
>> So even with an active desire to upgrade their platform, it may take
>> anywhere from 3-6 months to actually put that plan into pratice. It may
>> seem strange, but at the same time they can be perfectly ok upgrading
>> openstack in a matter of weeks, or even doing continuous deployment,
>> simply because that is a layer above that does not directly impact on
>> the hardware or their base platform certification.
>>
>> So while it is acceptable that we are aggressive at upgrading the python
>> layer in openstack, we need to be conservative at requiring upgrades of
>> the hypervisor layer. There needs to be a compelling reason why we must
>> drop an old platform. Generally we should aim to only drop old platforms
>> when the vendor themselves has end of lifed them.
>>
>> Once all platforms with a particular old version of the hypervisor
>> packages are cut, we should ensure we give users advance notice of our
>> intent to drop their platform. I'd suggest that for this we should allow
>> one cycle of deprecation. eg, for liberty we could check version are
>> startup and print a warning if it was old, and then in Mxxxx we could
>> drop it.
>>
>> As above though, I dont see any compelling reason to increase the min
>> libvirt to 1.2.2 at this time.
>>
>> I could see us increase libvirt to 0.10.2 and qemu to 0.12.1.2as, assuming
>> opensuse 12.2 is end of life, then rhel-6 is the next oldest platform that
>> is still supported. For that I would make liberty print a warning on start
>> if libvirt was in the range 0.9.9 < 0.10.2, likewise for QEMU, and then
>> change the min version in  Mxxxx.
>
> One other thing I should have mentioned is that we don't actually have
> one single minimum libvirt version. We actually have a couple of different
> minimum versions based on either the architecture or the hypervisor.
>
> For example, the Parallels hypervisor support was set to 1.2.12 and
> the S/390 support was /supposed/ to have been set to 1.0.4, but I see
> the devs failed to actally submit that change so I'll be doing that
> shortly.
>
> I think there is a credible argument that we increase the min required
> libvirt for LXC, because it requires a pretty new libvirt and kernel
> to provide any sensible level of security (ie user namespaces). We're
> rather negligent at the moment to let users deploy LXC with older
> versions as it is trivial for tenants to escape isolation.
>
> So the current MIN_LIBVIRT_VERSION is really talking about x86 + KVM
> combination.
>
>
> We do a really bad job of making this clear anywhere in our docs for
> Nova AFAIK. Likewise we don't make any distinction in our docs about
> the version we have tested with, vs the versions we are capable of
> running with. This is all critical info to people deploying, so they
> have guidance as to how much testing of their specific platform they
> should do at deployment time.
>
> Regards,
> Daniel
>

I've changed https://review.openstack.org/#/c/183220/ to be a min 
version check in init_host() for liberty and say that we're going to 
make 0.10.2 the minimum required version for the 2016.1 'M' release, so 
that will be in the Liberty release notes and people will have some time 
to move up.

I'll push a separate change to add libvirt-python back into 
test-requirements.txt (>=1.2.5 per global-requirements) so we can remove 
the workaround in host.py.

-- 

Thanks,

Matt Riedemann