---- On Wed, 21 Jul 2021 01:53:59 -0500 Slawek Kaplonski <skaplons@redhat.com> wrote ----
Hi,
On środa, 21 lipca 2021 01:30:25 CEST Akihiro Motoki wrote:
Hi,
I checked the recent neutron log and found the same error in neutron-ovn-tempest-slow job in the check queue [1] of a neutron change merged a couple of hours ago [0]. I don't see such errors in other changes merged recently (801068 and 801076). 779310 is a child commit of 776701, so it hit the same error, but these are not directly related. I believe reverting https://review.opendev.org/c/openstack/neutron/+/776701 will fix the issue. I will update https://review.opendev.org/c/openstack/neutron/+/801478.
The reason this issue happened is a configuration mismatch between tempest-slow/full used globally (in most projects) and tempest-slow/full used in neutron gate. The neutron gate has two types of the tempest-slow jobs: neutron-ovs-tempest-slow and neutron-ovn-tempest-slow. The former is voting but the latter is non-voting. tempest-slow used globally is same as the latter (neutron-ovn-tempest-slow) from the point of view of devstack configuration. This is the reason the neutron gate could not detect it before merging the commit which triggered the error.
Thx for analysis and fast approval of that revert. I just proposed https://review.opendev.org/c/openstack/neutron/+/801598 to make that neutron job voting and gating and to avoid such issues in the future.
+1, having the ovn job which is the default in devstack as voting will help. -gmann
[0] https://review.opendev.org/c/openstack/neutron/+/776701 [1] https://zuul.opendev.org/t/openstack/build/5344d9bdec9346738499114c10a81aff/... og/controller/logs/screen-q-svc.txt#1993
Thanks, Akihiro Motoki (amotoki)
On Wed, Jul 21, 2021 at 7:04 AM Ghanshyam Mann <gmann@ghanshyammann.com>
wrote:
Hello Everyone,
Since 2-3 hrs before, tempest-full-py3, tempest-slow-py3 and few more jobs started failing consistently in create_neutron_initial_network() method:
"++ lib/neutron_plugins/services/l3:create_neutron_initial_network:214 : oscwrap --os-cloud devstack-admin --os-region RegionOne network create --project a6b71b541987471595bf5e38f5fbe264 private"
- https://zuul.openstack.org/builds?job_name=tempest-full-py3 - https://zuul.openstack.org/builds?job_name=tempest-slow-py3
Strange is that 'tempest-integrated-storage'|'compute' jobs are passing even the configuration is almost the same.
Slaweq reported the below bug and trying some neutron revert to know the root cause. He will check it tomorrow morning.
- https://bugs.launchpad.net/neutron/+bug/1936983
Until we find the root cause/fix, do not recheck on these failures.
-gmann
-- Slawek Kaplonski Principal Software Engineer Red Hat