On Sun, Oct 25, 2020 at 12:45:57AM -0700, melanie witt wrote:
I've updated subject line to more specifically describe the issue. Details inline.
[...] Hi, Mel
I tried another DNM patch to revert the libvirt-python bump from 6.6.0 to 6.8.0:
https://review.opendev.org/759552
and rechecked my nova patch that Depends-On it:
https://review.opendev.org/759506
And it nova-compute runs successfully. So it's definitely the libvirt-python patch bump that's causing the failure.
Yet, on the upper-constraints bump patch, nova-compute ran fine with the new libvirt-python version 6.8.0. WHY?!
Yeah, that really is puzzling / strange.
I found that the upper-constraints patch ran tempest with python 3.8 and it passed with libvirt-python 6.8.0 [2] BUT our jobs (and some other projects jobs) run with python 3.6 [3] and it fails with libvirt-python 6.8.0.
So there appears to be some kind of incompatibility between python 3.6 and libvirt-python 6.8.0. I'm not sure what we can do here besides revert the libvirt-python bump and go back to 6.6.0. AFAIK we still need to have integration test coverage for python 3.6 still (correct me if I'm wrong). I'm also not sure if there's way we could configure libvirt-python 6.8.0 for python 3.8 and libvirt python 6.6.0 for python 3.6.
Our jobs running python 3.8 are passing: nova-next, nova-ceph-multistore, nova-multi-cell, tempest-ipv6-only.
Our jobs running python 3.6 are failing: tempest-integrated-compute, grenade, nova-live-migration.
I had a look through the libvirt-python repo to see what changed between 6.6.0 and 6.8.0 and can't tell what's the root cause. We get the TypeError raised in nova-compute when we call the libvirt openAuth method but nothing in this diff [4] looks problematic?
Great sleuthing so far. (Not to mention the terribly cryptic orignal error message) As a small addendum, was talking to Dan Berrangé from libvirt and pointed to the openAuth() method ... and he wrote a quick 'libvirt-python' unit test for the openAuth() method: https://gitlab.com/libvirt/libvirt-python/-/merge_requests/28/diffs ... and it seems to be working as expected (based on the CI result): https://gitlab.com/libvirt/libvirt-python/-/merge_requests/28/pipelines ... it seems to validate several operating system platforms (and Python versions), including Ubuntu 18.04: https://gitlab.com/berrange/libvirt-python/-/pipelines/207610554
I've opened a bug to capture the info I found so far:
https://bugs.launchpad.net/nova/+bug/1901383
-melanie
[1] https://zuul.opendev.org/t/openstack/build/20f1d309663347c28112b711b82b9c03/...
[2] https://zuul.opendev.org/t/openstack/build/81f1f400b43b4e22bdae3ce1c2de92a8/... [3] https://zuul.opendev.org/t/openstack/build/96b64d8e869f46d1a88ee0f728f013f6/... [4] https://github.com/libvirt/libvirt-python/compare/v6.6.0...v6.8.0#diff-a3d04...
-- /kashyap