Open Stack

I've updated subject line to more specifically describe the issue. 
Details inline.

On 10/23/20 12:08, melanie witt wrote:
> Hey all,
> 
> This is just a FYI that we've got another gate failure where 
> nova-compute is failing with the following error raised from the libvirt 
> driver [1]:
> 
> "TypeError: Parameterized generics cannot be used with class"
> 
> which was raised from the eventlet.tpool code. (I don't yet understand 
> what this error means exactly.)
> 
> Because of this, we suspect this is related to the eventlet package 
> version bump from 0.26.1 to 0.28.0 that happened in the 
> upper-constraints change to openstack/requirements that merged earlier 
> today:
> 
> https://review.opendev.org/#/c/750084/47/upper-constraints.txt@150
> 
> But we're not certain yet exactly what is happening and whether this ^ 
> is indeed the cause.
> 
> I am currently investigating to find the root cause of failure and how 
> to fix it/confirm which package we should pin for now. I am trying out a 
> DNM nova patch to pin eventlet to 0.26.1 to see what happens, while I 
> look into more depth at what the error means and whether it's something 
> we should fix in the nova code.
Update: I tried a DNM patch to revert the eventlet bump and nova-compute 
still failed with the same error. So it's not the eventlet package version.

I tried another DNM patch to revert the libvirt-python bump from 6.6.0 
to 6.8.0:

https://review.opendev.org/759552

and rechecked my nova patch that Depends-On it:

https://review.opendev.org/759506

And it nova-compute runs successfully. So it's definitely the 
libvirt-python patch bump that's causing the failure.

Yet, on the upper-constraints bump patch, nova-compute ran fine with the 
new libvirt-python version 6.8.0. WHY?!

I found that the upper-constraints patch ran tempest with python 3.8 and 
it passed with libvirt-python 6.8.0 [2] BUT our jobs (and some other 
projects jobs) run with python 3.6 [3] and it fails with libvirt-python 
6.8.0.

So there appears to be some kind of incompatibility between python 3.6 
and libvirt-python 6.8.0. I'm not sure what we can do here besides 
revert the libvirt-python bump and go back to 6.6.0. AFAIK we still need 
to have integration test coverage for python 3.6 still (correct me if 
I'm wrong). I'm also not sure if there's way we could configure 
libvirt-python 6.8.0 for python 3.8 and libvirt python 6.6.0 for python 3.6.

Our jobs running python 3.8 are passing: nova-next, 
nova-ceph-multistore, nova-multi-cell, tempest-ipv6-only.

Our jobs running python 3.6 are failing: tempest-integrated-compute, 
grenade, nova-live-migration.

I had a look through the libvirt-python repo to see what changed between 
6.6.0 and 6.8.0 and can't tell what's the root cause. We get the 
TypeError raised in nova-compute when we call the libvirt openAuth 
method but nothing in this diff [4] looks problematic?

I've opened a bug to capture the info I found so far:

https://bugs.launchpad.net/nova/+bug/1901383

-melanie

> [1] 
> https://zuul.opendev.org/t/openstack/build/20f1d309663347c28112b711b82b9c03/log/controller/logs/screen-n-cpu.txt?severity=4#861 

[2] 
https://zuul.opendev.org/t/openstack/build/81f1f400b43b4e22bdae3ce1c2de92a8/log/job-output.txt#1283
[3] 
https://zuul.opendev.org/t/openstack/build/96b64d8e869f46d1a88ee0f728f013f6/log/job-output.txt#1234
[4] 
https://github.com/libvirt/libvirt-python/compare/v6.6.0...v6.8.0#diff-a3d0475c2dfecd386f36307bc70befd9d5bd69daceadc61a5589ee86413d04c7L91