Open Stack

On 04/04/2017 12:14 PM, Daniel P. Berrange wrote:
> On Tue, Apr 04, 2017 at 09:21:27AM -0700, Clark Boylan wrote:
>> On Tue, Apr 4, 2017, at 06:36 AM, Daniel P. Berrange wrote:
>>> On Mon, Apr 03, 2017 at 04:06:53PM -0700, Clark Boylan wrote:
>>>> I have pushed a change to devstack [3] to enable using UCA which pulls
>>>> in new Libvirt and mostly seems to work. I think we should consider
>>>> switching to UCA as this may fix our Libvirt problems and if it doesn't,
>>>> we will be closer to a version of Libvirt that upstream should be
>>>> willing to fix.
>>>>
>>>> This isn't the most straightfoward switch as UCA has a different repo
>>>> for each OpenStack release. libvirt-python is sensitve to the underlying
>>>> library changing; it is backward compatible but libvirt-python built
>>>> against older libvirt won't work against new libvirt. The result is a
>>>> libvirt-python wheel built on our wheel mirror does not work with UCA.
>>>
>>> I'm surprised about that - could you elaborate on what's broken for you.
>>> The libvirt.so provides a stable public API, and the standalone python
>>> binding only uses public APIs from libvirt.so.  IOW you should be able
>>> to build libvirt-python against 1.3.0 and then use it against 2.5.0 with
>>> no problems.
>>>
>>> NB, *before* libvirt-python lived on pypi, it used some non-public
>>> libvirt.so symbols, so was tied to the exact libvirt.so it was built
>>> against. Assuming you're using the pypi version this should no longer
>>> apply.
>>
>> The specific issue was "AttributeError: 'module' object has no attribute
>> 'VIR_MIGRATE_POSTCOPY'" where module here is libvirt (full log and
>> traceback at [0]). The libvirt-python module here was built against
>> Libvirt 1.3.1 turned into a wheel and copied into our wheel mirror. Then
>> when running against Libvirt 2.5.0 Nova seems to have detected that
>> newer features should be present that are not reflected in the compiled
>> libvirt-python resulting in the error. This crashed nova compute.
>>
>> Problem was easily corrected by preventing devstack from using our wheel
>> mirror for libvirt-python which resulted in a new installation built
>> against Libvirt 2.5.0.
>>
>> It seems like the API is stable enough for backward compatibility but
>> not forward compatibility. Its also possible that Nova is doing version
>> checking in a buggy way and should be checking what the libvirt-python
>> version is and what it supports?
>
> Ok, so yeah your last sentance is the correct interpretation. You've built
> libvirt-python against libvirt v1.3.1, so it only includes support for
> constants & methods that exist in that version. The VIR_MIGRATE_POSTCOPY
> constant was introduced in v1.3.3, so it will not be included in the
> libvirt-python you built.
>
> When checking features Nova calls a libvirt API that returns the version
> of the libvirtd daemon, which is v2.5.0, and then just blindly assumes
> libvirt-python has the same version.
>
> Unfortunately there is no way for Nova to determine what libvirt version
> the python binding was built against, so it can't improve its version
> check in this respect. To deal with this, Nova would two options:
>
>  - Provide a nova.conf parameter to force it to assume an older libvirt
>    version, thus disabling the features regardless of what libvirtd
>    supports
>  - Make nova check for existance of the python constants / APIs it is
>    trying to use, in addition to checking the libvirt version
>
> The first option is pretty trivial to do if needed. The second option would
> be the more correct approach, but a much bigger maint burden, so I'm not
> convinced it is worth it.

I'm not convinced either are worth it - I think if we just stop building 
libvirt wheels in our wheel mirror it'll resolve itself. Most places 
that are actually running OpenStack for real aren't going to be building 
their install packages blind to what version of the OS they're going to 
install on - so it's really only a problem with one of our gate 
optimizations.