[nova][placement][tempest] Hold your rechecks
Sorry folks, that's kind of an email I hate writing but let's be honest : our gate is busted. Until we figure out a correct path for resolution, I hereby ask you to *NOT* recheck in order to not spill our precious CI resources for tests that are certain to fail. Long story story, there are currently two problems : #1 https://launchpad.net/bugs/1940425 nova-ovs-hybrid-plug and nova-next jobs 100% fail due to a port remaining in down state. #2 https://bugs.launchpad.net/nova/+bug/1960346 nova-lvm job 100% fails due to a volume detach failure probably due to QEMU #1 is currently investigated by the Neutron team meanwhile a patch [1] has been proposed against Zuul to skip the failing tests. Unfortunately, this patch [1] is unable to merge due to #2. #2 has a Tempest patch that's being worked on [2] but the current state of this patch is WIP. We somehow need to have an agreement on the way forward during this afternoon (UTC) to identify whether we can reasonably progress on [2] or skip the failing tests on nova-lvm. Again, sorry about the bad news and I'll keep you informed. -Sylvain [1] https://review.opendev.org/c/openstack/nova/+/865658/ [2] https://review.opendev.org/c/openstack/tempest/+/842240
Le lun. 28 nov. 2022 à 14:28, Sylvain Bauza <sbauza@redhat.com> a écrit :
Sorry folks, that's kind of an email I hate writing but let's be honest : our gate is busted. Until we figure out a correct path for resolution, I hereby ask you to *NOT* recheck in order to not spill our precious CI resources for tests that are certain to fail.
Long story story, there are currently two problems : #1 https://launchpad.net/bugs/1940425 nova-ovs-hybrid-plug and nova-next jobs 100% fail due to a port remaining in down state. #2 https://bugs.launchpad.net/nova/+bug/1960346 nova-lvm job 100% fails due to a volume detach failure probably due to QEMU
Today's update :
#1 is currently investigated by the Neutron team meanwhile a patch [1] has been proposed against Zuul to skip the failing tests. Unfortunately, this patch [1] is unable to merge due to #2.
Good news, kudos to the Neutron team which delivered a bugfix against the rootcause, which is always better than just skipping tests (and lacking then coverage). https://review.opendev.org/c/openstack/neutron/+/837780/18 Accordingly, [1] is no longer necessary and has been abandoned after a recheck to verify the job runs.
#2 has a Tempest patch that's being worked on [2] but the current state of this patch is WIP. We somehow need to have an agreement on the way forward during this afternoon (UTC) to identify whether we can reasonably progress on [2] or skip the failing tests on nova-lvm.
Given [2] is hard to write, gmann proposed a patch [3] for skipping some nova-lvm tests. Reviews of [3] ongoing, should be hopefully merged today around noon UTC. Once [3] is merged, the gate should be unblocked. Again, an email will be sent once we progress on [3]. -S
Again, sorry about the bad news and I'll keep you informed. -Sylvain
[1] https://review.opendev.org/c/openstack/nova/+/865658/ [2] https://review.opendev.org/c/openstack/tempest/+/842240
(early morning, needing a coffee apparently) Le mar. 29 nov. 2022 à 09:32, Sylvain Bauza <sbauza@redhat.com> a écrit :
Le lun. 28 nov. 2022 à 14:28, Sylvain Bauza <sbauza@redhat.com> a écrit :
Sorry folks, that's kind of an email I hate writing but let's be honest : our gate is busted. Until we figure out a correct path for resolution, I hereby ask you to *NOT* recheck in order to not spill our precious CI resources for tests that are certain to fail.
Long story story, there are currently two problems : #1 https://launchpad.net/bugs/1940425 nova-ovs-hybrid-plug and nova-next jobs 100% fail due to a port remaining in down state. #2 https://bugs.launchpad.net/nova/+bug/1960346 nova-lvm job 100% fails due to a volume detach failure probably due to QEMU
Today's update :
#1 is currently investigated by the Neutron team meanwhile a patch [1] has been proposed against Zuul to skip the failing tests. Unfortunately, this patch [1] is unable to merge due to #2.
Good news, kudos to the Neutron team which delivered a bugfix against the rootcause, which is always better than just skipping tests (and lacking then coverage). https://review.opendev.org/c/openstack/neutron/+/837780/18
Accordingly, [1] is no longer necessary and has been abandoned after a recheck to verify the job runs.
#2 has a Tempest patch that's being worked on [2] but the current state of this patch is WIP. We somehow need to have an agreement on the way forward during this afternoon (UTC) to identify whether we can reasonably progress on [2] or skip the failing tests on nova-lvm.
Given [2] is hard to write, gmann proposed a patch [3] for skipping some nova-lvm tests. Reviews of [3] ongoing, should be hopefully merged today around noon UTC.
Once [3] is merged, the gate should be unblocked. Again, an email will be sent once we progress on [3]. -S
Again, sorry about the bad news and I'll keep you informed. -Sylvain
[1] https://review.opendev.org/c/openstack/nova/+/865658/ [2] https://review.opendev.org/c/openstack/tempest/+/842240
Le mar. 29 nov. 2022 à 09:33, Sylvain Bauza <sbauza@redhat.com> a écrit :
(early morning, needing a coffee apparently)
Le mar. 29 nov. 2022 à 09:32, Sylvain Bauza <sbauza@redhat.com> a écrit :
Le lun. 28 nov. 2022 à 14:28, Sylvain Bauza <sbauza@redhat.com> a écrit :
Sorry folks, that's kind of an email I hate writing but let's be honest : our gate is busted. Until we figure out a correct path for resolution, I hereby ask you to *NOT* recheck in order to not spill our precious CI resources for tests that are certain to fail.
Long story story, there are currently two problems : #1 https://launchpad.net/bugs/1940425 nova-ovs-hybrid-plug and nova-next jobs 100% fail due to a port remaining in down state. #2 https://bugs.launchpad.net/nova/+bug/1960346 nova-lvm job 100% fails due to a volume detach failure probably due to QEMU
Today's update :
#1 is currently investigated by the Neutron team meanwhile a patch [1] has been proposed against Zuul to skip the failing tests. Unfortunately, this patch [1] is unable to merge due to #2.
Good news, kudos to the Neutron team which delivered a bugfix against the rootcause, which is always better than just skipping tests (and lacking then coverage). https://review.opendev.org/c/openstack/neutron/+/837780/18
Accordingly, [1] is no longer necessary and has been abandoned after a recheck to verify the job runs.
#2 has a Tempest patch that's being worked on [2] but the current state of this patch is WIP. We somehow need to have an agreement on the way forward during this afternoon (UTC) to identify whether we can reasonably progress on [2] or skip the failing tests on nova-lvm.
Given [2] is hard to write, gmann proposed a patch [3] for skipping some nova-lvm tests. Reviews of [3] ongoing, should be hopefully merged today around noon UTC.
Once [3] is merged, the gate should be unblocked. Again, an email will be sent once we progress on [3].
[3] is merged, so now the gate is back \o/ Thanks all folks who helped on those issues !
-S
Again, sorry about the bad news and I'll keep you informed. -Sylvain
[1] https://review.opendev.org/c/openstack/nova/+/865658/ [2] https://review.opendev.org/c/openstack/tempest/+/842240
Hi, Could this problem affect cinder-tempest-plugin-lvm-tgt-barbican jobs as well? (All the nova+lvm test are failing) https://zuul.opendev.org/t/openstack/build/140630488ea745a69ce3ebadf85a41fa Thanks Sofia On Tue, Nov 29, 2022 at 3:22 PM Sylvain Bauza <sbauza@redhat.com> wrote:
Le mar. 29 nov. 2022 à 09:33, Sylvain Bauza <sbauza@redhat.com> a écrit :
(early morning, needing a coffee apparently)
Le mar. 29 nov. 2022 à 09:32, Sylvain Bauza <sbauza@redhat.com> a écrit :
Le lun. 28 nov. 2022 à 14:28, Sylvain Bauza <sbauza@redhat.com> a écrit :
Sorry folks, that's kind of an email I hate writing but let's be honest : our gate is busted. Until we figure out a correct path for resolution, I hereby ask you to *NOT* recheck in order to not spill our precious CI resources for tests that are certain to fail.
Long story story, there are currently two problems : #1 https://launchpad.net/bugs/1940425 nova-ovs-hybrid-plug and nova-next jobs 100% fail due to a port remaining in down state. #2 https://bugs.launchpad.net/nova/+bug/1960346 nova-lvm job 100% fails due to a volume detach failure probably due to QEMU
Today's update :
#1 is currently investigated by the Neutron team meanwhile a patch [1] has been proposed against Zuul to skip the failing tests. Unfortunately, this patch [1] is unable to merge due to #2.
Good news, kudos to the Neutron team which delivered a bugfix against the rootcause, which is always better than just skipping tests (and lacking then coverage). https://review.opendev.org/c/openstack/neutron/+/837780/18
Accordingly, [1] is no longer necessary and has been abandoned after a recheck to verify the job runs.
#2 has a Tempest patch that's being worked on [2] but the current state of this patch is WIP. We somehow need to have an agreement on the way forward during this afternoon (UTC) to identify whether we can reasonably progress on [2] or skip the failing tests on nova-lvm.
Given [2] is hard to write, gmann proposed a patch [3] for skipping some nova-lvm tests. Reviews of [3] ongoing, should be hopefully merged today around noon UTC.
Once [3] is merged, the gate should be unblocked. Again, an email will be sent once we progress on [3].
[3] is merged, so now the gate is back \o/ Thanks all folks who helped on those issues !
-S
Again, sorry about the bad news and I'll keep you informed. -Sylvain
[1] https://review.opendev.org/c/openstack/nova/+/865658/ [2] https://review.opendev.org/c/openstack/tempest/+/842240
-- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>
Le ven. 2 déc. 2022 à 17:26, Sofia Enriquez <senrique@redhat.com> a écrit :
Hi, Could this problem affect cinder-tempest-plugin-lvm-tgt-barbican jobs as well? (All the nova+lvm test are failing) https://zuul.opendev.org/t/openstack/build/140630488ea745a69ce3ebadf85a41fa
Sounds related AFAICS as this is related to a volume attach/detach. TBC, the root problem is that when you attach or detach a volume earlier than when the guest kernel is fully booted, then detach won't work. In order to not have a volume detach problem, Tempest needs to reliably wait for the guest to be sshable (as the kernel will be booted). HTH, -Sylvain Thanks
Sofia
On Tue, Nov 29, 2022 at 3:22 PM Sylvain Bauza <sbauza@redhat.com> wrote:
Le mar. 29 nov. 2022 à 09:33, Sylvain Bauza <sbauza@redhat.com> a écrit :
(early morning, needing a coffee apparently)
Le mar. 29 nov. 2022 à 09:32, Sylvain Bauza <sbauza@redhat.com> a écrit :
Le lun. 28 nov. 2022 à 14:28, Sylvain Bauza <sbauza@redhat.com> a écrit :
Sorry folks, that's kind of an email I hate writing but let's be honest : our gate is busted. Until we figure out a correct path for resolution, I hereby ask you to *NOT* recheck in order to not spill our precious CI resources for tests that are certain to fail.
Long story story, there are currently two problems : #1 https://launchpad.net/bugs/1940425 nova-ovs-hybrid-plug and nova-next jobs 100% fail due to a port remaining in down state. #2 https://bugs.launchpad.net/nova/+bug/1960346 nova-lvm job 100% fails due to a volume detach failure probably due to QEMU
Today's update :
#1 is currently investigated by the Neutron team meanwhile a patch [1] has been proposed against Zuul to skip the failing tests. Unfortunately, this patch [1] is unable to merge due to #2.
Good news, kudos to the Neutron team which delivered a bugfix against the rootcause, which is always better than just skipping tests (and lacking then coverage). https://review.opendev.org/c/openstack/neutron/+/837780/18
Accordingly, [1] is no longer necessary and has been abandoned after a recheck to verify the job runs.
#2 has a Tempest patch that's being worked on [2] but the current state of this patch is WIP. We somehow need to have an agreement on the way forward during this afternoon (UTC) to identify whether we can reasonably progress on [2] or skip the failing tests on nova-lvm.
Given [2] is hard to write, gmann proposed a patch [3] for skipping some nova-lvm tests. Reviews of [3] ongoing, should be hopefully merged today around noon UTC.
Once [3] is merged, the gate should be unblocked. Again, an email will be sent once we progress on [3].
[3] is merged, so now the gate is back \o/ Thanks all folks who helped on those issues !
-S
Again, sorry about the bad news and I'll keep you informed. -Sylvain
[1] https://review.opendev.org/c/openstack/nova/+/865658/ [2] https://review.opendev.org/c/openstack/tempest/+/842240
--
Sofía Enriquez
she/her
Software Engineer
Red Hat PnT <https://www.redhat.com>
IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>
participants (2)
-
Sofia Enriquez
-
Sylvain Bauza