On Thu, 2022-02-03 at 07:57 -0500, Artom Lifshitz wrote: > https://bugs.launchpad.net/nova/+bug/1959899 > > The test times out verifying the tags in the metadata API after > attaching a NIC and volume. > > This is happening almost 100% of the time, but I've seen at least a > couple of passing runs as well. We have yet to determine what the > exact failure mode is - is the metadata API timing out (unlikely, > since every other test passes)? Or are the expected tags just not > present (more likely)? we might want to add it to the exclude list for that job for now since we should have testing without q35 for it in other jobs. that will "unblock" the ci for now coming up to code freeze as we want to avoid soft blockign the ci with lots fo rechecks. > > I'm trying to figure some things out at [1], but other than that this > is very much under investigation with no root cause identified yet. > > [1] https://review.opendev.org/c/openstack/nova/+/827549 have you been able to repoduce it locally? i know you tried addign some waits to the tempest change too i assuem that did not help? we can technially call the metadata api form outside the vm so perhaps we should also have tempst curl the nova metadata api direclty externally form the vm to assert taht its not in the metadata api issue. in terms fo tags we might also want to have tempest do a blkid or lsblk to see if we can see the volume is attached in the vm by looking at the serial. i think there are other tests that do that but before we assert the tag is in the metadata api we shoudl ensrue the volume is present in the vm. we are not currently doing that https://github.com/openstack/tempest/blob/ad8f599b32e875c438bd49b8d81bfcd9d4eb8ead/tempest/api/compute/servers/test_device_tagging.py#L421 so hwere we shoudl wait for the volume to attach https://github.com/openstack/tempest/blob/ad8f599b32e875c438bd49b8d81bfcd9d4eb8ead/tempest/api/compute/servers/test_device_tagging.py#L413 and then verify its attached in the vm with lsblk/blkid. > >