I've updated subject line to more specifically describe the issue. Details inline. On 10/23/20 12:08, melanie witt wrote:
Hey all,
This is just a FYI that we've got another gate failure where nova-compute is failing with the following error raised from the libvirt driver [1]:
"TypeError: Parameterized generics cannot be used with class"
which was raised from the eventlet.tpool code. (I don't yet understand what this error means exactly.)
Because of this, we suspect this is related to the eventlet package version bump from 0.26.1 to 0.28.0 that happened in the upper-constraints change to openstack/requirements that merged earlier today:
https://review.opendev.org/#/c/750084/47/upper-constraints.txt@150
But we're not certain yet exactly what is happening and whether this ^ is indeed the cause.
I am currently investigating to find the root cause of failure and how to fix it/confirm which package we should pin for now. I am trying out a DNM nova patch to pin eventlet to 0.26.1 to see what happens, while I look into more depth at what the error means and whether it's something we should fix in the nova code. Update: I tried a DNM patch to revert the eventlet bump and nova-compute still failed with the same error. So it's not the eventlet package version.
I tried another DNM patch to revert the libvirt-python bump from 6.6.0 to 6.8.0: https://review.opendev.org/759552 and rechecked my nova patch that Depends-On it: https://review.opendev.org/759506 And it nova-compute runs successfully. So it's definitely the libvirt-python patch bump that's causing the failure. Yet, on the upper-constraints bump patch, nova-compute ran fine with the new libvirt-python version 6.8.0. WHY?! I found that the upper-constraints patch ran tempest with python 3.8 and it passed with libvirt-python 6.8.0 [2] BUT our jobs (and some other projects jobs) run with python 3.6 [3] and it fails with libvirt-python 6.8.0. So there appears to be some kind of incompatibility between python 3.6 and libvirt-python 6.8.0. I'm not sure what we can do here besides revert the libvirt-python bump and go back to 6.6.0. AFAIK we still need to have integration test coverage for python 3.6 still (correct me if I'm wrong). I'm also not sure if there's way we could configure libvirt-python 6.8.0 for python 3.8 and libvirt python 6.6.0 for python 3.6. Our jobs running python 3.8 are passing: nova-next, nova-ceph-multistore, nova-multi-cell, tempest-ipv6-only. Our jobs running python 3.6 are failing: tempest-integrated-compute, grenade, nova-live-migration. I had a look through the libvirt-python repo to see what changed between 6.6.0 and 6.8.0 and can't tell what's the root cause. We get the TypeError raised in nova-compute when we call the libvirt openAuth method but nothing in this diff [4] looks problematic? I've opened a bug to capture the info I found so far: https://bugs.launchpad.net/nova/+bug/1901383 -melanie
[1] https://zuul.opendev.org/t/openstack/build/20f1d309663347c28112b711b82b9c03/...
[2] https://zuul.opendev.org/t/openstack/build/81f1f400b43b4e22bdae3ce1c2de92a8/... [3] https://zuul.opendev.org/t/openstack/build/96b64d8e869f46d1a88ee0f728f013f6/... [4] https://github.com/libvirt/libvirt-python/compare/v6.6.0...v6.8.0#diff-a3d04...