Open Stack

On Tue, Apr 04, 2017 at 09:21:27AM -0700, Clark Boylan wrote:
> On Tue, Apr 4, 2017, at 06:36 AM, Daniel P. Berrange wrote:
> > On Mon, Apr 03, 2017 at 04:06:53PM -0700, Clark Boylan wrote:
> > > I have pushed a change to devstack [3] to enable using UCA which pulls
> > > in new Libvirt and mostly seems to work. I think we should consider
> > > switching to UCA as this may fix our Libvirt problems and if it doesn't,
> > > we will be closer to a version of Libvirt that upstream should be
> > > willing to fix.
> > > 
> > > This isn't the most straightfoward switch as UCA has a different repo
> > > for each OpenStack release. libvirt-python is sensitve to the underlying
> > > library changing; it is backward compatible but libvirt-python built
> > > against older libvirt won't work against new libvirt. The result is a
> > > libvirt-python wheel built on our wheel mirror does not work with UCA.
> > 
> > I'm surprised about that - could you elaborate on what's broken for you.
> > The libvirt.so provides a stable public API, and the standalone python
> > binding only uses public APIs from libvirt.so.  IOW you should be able
> > to build libvirt-python against 1.3.0 and then use it against 2.5.0 with
> > no problems.
> > 
> > NB, *before* libvirt-python lived on pypi, it used some non-public
> > libvirt.so symbols, so was tied to the exact libvirt.so it was built
> > against. Assuming you're using the pypi version this should no longer
> > apply.
> 
> The specific issue was "AttributeError: 'module' object has no attribute
> 'VIR_MIGRATE_POSTCOPY'" where module here is libvirt (full log and
> traceback at [0]). The libvirt-python module here was built against
> Libvirt 1.3.1 turned into a wheel and copied into our wheel mirror. Then
> when running against Libvirt 2.5.0 Nova seems to have detected that
> newer features should be present that are not reflected in the compiled
> libvirt-python resulting in the error. This crashed nova compute.
> 
> Problem was easily corrected by preventing devstack from using our wheel
> mirror for libvirt-python which resulted in a new installation built
> against Libvirt 2.5.0.
> 
> It seems like the API is stable enough for backward compatibility but
> not forward compatibility. Its also possible that Nova is doing version
> checking in a buggy way and should be checking what the libvirt-python
> version is and what it supports?

Ok, so yeah your last sentance is the correct interpretation. You've built
libvirt-python against libvirt v1.3.1, so it only includes support for
constants & methods that exist in that version. The VIR_MIGRATE_POSTCOPY
constant was introduced in v1.3.3, so it will not be included in the
libvirt-python you built.

When checking features Nova calls a libvirt API that returns the version
of the libvirtd daemon, which is v2.5.0, and then just blindly assumes
libvirt-python has the same version.

Unfortunately there is no way for Nova to determine what libvirt version
the python binding was built against, so it can't improve its version
check in this respect. To deal with this, Nova would two options:

 - Provide a nova.conf parameter to force it to assume an older libvirt
   version, thus disabling the features regardless of what libvirtd
   supports
 - Make nova check for existance of the python constants / APIs it is
   trying to use, in addition to checking the libvirt version

The first option is pretty trivial to do if needed. The second option would
be the more correct approach, but a much bigger maint burden, so I'm not
convinced it is worth it.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|