[ci][nova][cinder][glance][devstack][swift] openstacksdk-functional-devstack job failing 100%
Hey all, We noticed sometime today the openstacksdk-functional-devstack job appears to be failing 100% in multiple projects: https://zuul.opendev.org/t/openstack/builds?job_name=openstacksdk-functional-devstack&skip=0 The failing openstacksdk test is test_add_remove_router_interface [1]: ft1.1: openstack.tests.functional.cloud.test_router.TestRouter.test_add_remove_router_interfacetesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/openstacksdk/openstack/tests/functional/cloud/test_router.py", line 232, in test_add_remove_router_interface self.assertIn(key, iface) File "/home/zuul/src/opendev.org/openstack/openstacksdk/.tox/functional/lib/python3.12/site-packages/testtools/testcase.py", line 425, in assertIn self.assertThat(haystack, Contains(needle), message) File "/home/zuul/src/opendev.org/openstack/openstacksdk/.tox/functional/lib/python3.12/site-packages/testtools/testcase.py", line 509, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: 'tenant_id' not in {'id': '57671080-6eb2-4c7f-9e64-c61edfa8995e', 'project_id': '1ef2e6c3fc2d4f6b92fdd8d49f0a05a1', 'port_id': '5b39bbfc-6d3d-440e-9c7a-e56e434ef897', 'network_id': '8aa6b6e2-e0c5-4f96-bb87-5509e30d340c', 'subnet_id': '6f52d8be-cf68-4552-b57d-37142cea65cc', 'subnet_ids': ['6f52d8be-cf68-4552-b57d-37142cea65cc']} which adds an interface to a router and expects the response parameter 'tenant_id' to be in the response. I looked for recently merged changes that could have possibly changed the router API response and it might be something similar to this change: https://review.opendev.org/c/openstack/neutron/+/972982 but I don't really understand if/how that could change the API. I'm not sure what the desired fix will be here, whether the API should be more backward compatible and/or if the openstacksdk test should be fixed by s/tenant_id/project_id/ or something else. Please hold your rechecks if CI has failed on your patch due to this. -melwitt [1] https://github.com/openstack/openstacksdk/blob/7afe210333ec236a14fc5fa2d4a58...
Hi Melanie, Comments inline below. On 2/6/26 6:00 PM, melanie witt wrote:
Hey all,
We noticed sometime today the openstacksdk-functional-devstack job appears to be failing 100% in multiple projects:
https://zuul.opendev.org/t/openstack/builds?job_name=openstacksdk- functional-devstack&skip=0
The failing openstacksdk test is test_add_remove_router_interface [1]:
ft1.1: openstack.tests.functional.cloud.test_router.TestRouter.test_add_remove_router_interfacetesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/openstacksdk/openstack/ tests/functional/cloud/test_router.py", line 232, in test_add_remove_router_interface self.assertIn(key, iface) File "/home/zuul/src/opendev.org/openstack/openstacksdk/.tox/ functional/lib/python3.12/site-packages/testtools/testcase.py", line 425, in assertIn self.assertThat(haystack, Contains(needle), message) File "/home/zuul/src/opendev.org/openstack/openstacksdk/.tox/ functional/lib/python3.12/site-packages/testtools/testcase.py", line 509, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: 'tenant_id' not in {'id': '57671080-6eb2-4c7f-9e64-c61edfa8995e', 'project_id': '1ef2e6c3fc2d4f6b92fdd8d49f0a05a1', 'port_id': '5b39bbfc-6d3d-440e-9c7a- e56e434ef897', 'network_id': '8aa6b6e2-e0c5-4f96-bb87-5509e30d340c', 'subnet_id': '6f52d8be-cf68-4552-b57d-37142cea65cc', 'subnet_ids': ['6f52d8be-cf68-4552-b57d-37142cea65cc']}
which adds an interface to a router and expects the response parameter 'tenant_id' to be in the response.
I looked for recently merged changes that could have possibly changed the router API response and it might be something similar to this change:
So I've been going through the neutron code trying to convert everything to use project_id this cycle when I found some TODOs from 12 years ago on removing tenant_id :-/ I tried to make sure we continue to accept tenant_id, but I guess in this case I also removed it from the response. Although I have canary patches in some of the networking repos I didn't have one for the SDK, I'll add it to my list going forward.
but I don't really understand if/how that could change the API.
My guess is it's because of this part of the change: https://review.opendev.org/c/openstack/neutron/+/972982/14/neutron/db/db_bas...
I'm not sure what the desired fix will be here, whether the API should be more backward compatible and/or if the openstacksdk test should be fixed by s/tenant_id/project_id/ or something else.
Please hold your rechecks if CI has failed on your patch due to this.
I've pushed a change to fix it as I think it's time to move forward: https://review.opendev.org/c/openstack/openstacksdk/+/976008 If anyone thinks otherwise please let me know. Thanks for the report and sorry about the breakage, -Brian
[1] https://github.com/openstack/openstacksdk/ blob/7afe210333ec236a14fc5fa2d4a589e481a43745/openstack/tests/ functional/cloud/test_router.py#L205
Hi Melanie, Just an update that my patch solves the issue with the openstacksdk-functional-devstack job, but there is also an error in the -manila functional job as well, which was there before my neutron change merged. If someone could help debug that I'd appreciate it, might have to incorporate it into the same patch to fix the gate. Or we could change to non-voting until it can be addressed. There is also a pkg_resources issue with the pep8 job, but that seems to be happening across more than one repo at the moment :( -Brian On 2/7/26 2:57 PM, Brian Haley wrote:
Hi Melanie,
Comments inline below.
On 2/6/26 6:00 PM, melanie witt wrote:
Hey all,
We noticed sometime today the openstacksdk-functional-devstack job appears to be failing 100% in multiple projects:
https://zuul.opendev.org/t/openstack/builds?job_name=openstacksdk- functional-devstack&skip=0
The failing openstacksdk test is test_add_remove_router_interface [1]:
ft1.1: openstack.tests.functional.cloud.test_router.TestRouter.test_add_remove_router_interfacetesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/openstacksdk/openstack/ tests/functional/cloud/test_router.py", line 232, in test_add_remove_router_interface self.assertIn(key, iface) File "/home/zuul/src/opendev.org/openstack/openstacksdk/.tox/ functional/lib/python3.12/site-packages/testtools/testcase.py", line 425, in assertIn self.assertThat(haystack, Contains(needle), message) File "/home/zuul/src/opendev.org/openstack/openstacksdk/.tox/ functional/lib/python3.12/site-packages/testtools/testcase.py", line 509, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: 'tenant_id' not in {'id': '57671080-6eb2-4c7f-9e64-c61edfa8995e', 'project_id': '1ef2e6c3fc2d4f6b92fdd8d49f0a05a1', 'port_id': '5b39bbfc-6d3d-440e-9c7a- e56e434ef897', 'network_id': '8aa6b6e2- e0c5-4f96-bb87-5509e30d340c', 'subnet_id': '6f52d8be-cf68-4552- b57d-37142cea65cc', 'subnet_ids': ['6f52d8be-cf68-4552- b57d-37142cea65cc']}
which adds an interface to a router and expects the response parameter 'tenant_id' to be in the response.
I looked for recently merged changes that could have possibly changed the router API response and it might be something similar to this change:
So I've been going through the neutron code trying to convert everything to use project_id this cycle when I found some TODOs from 12 years ago on removing tenant_id :-/
I tried to make sure we continue to accept tenant_id, but I guess in this case I also removed it from the response. Although I have canary patches in some of the networking repos I didn't have one for the SDK, I'll add it to my list going forward.
but I don't really understand if/how that could change the API.
My guess is it's because of this part of the change:
https://review.opendev.org/c/openstack/neutron/+/972982/14/neutron/db/ db_base_plugin_common.py
I'm not sure what the desired fix will be here, whether the API should be more backward compatible and/or if the openstacksdk test should be fixed by s/tenant_id/project_id/ or something else.
Please hold your rechecks if CI has failed on your patch due to this.
I've pushed a change to fix it as I think it's time to move forward:
https://review.opendev.org/c/openstack/openstacksdk/+/976008
If anyone thinks otherwise please let me know.
Thanks for the report and sorry about the breakage,
-Brian
[1] https://github.com/openstack/openstacksdk/ blob/7afe210333ec236a14fc5fa2d4a589e481a43745/openstack/tests/ functional/cloud/test_router.py#L205
Thank you Brian for proposing the fix for openstacksdk! Also thanks for the pointer about how the API was affected. I have put a couple of links inline for ML update purposes. On 2/8/26 14:05, Brian Haley wrote:
Just an update that my patch solves the issue with the openstacksdk- functional-devstack job, but there is also an error in the -manila functional job as well, which was there before my neutron change merged. If someone could help debug that I'd appreciate it, might have to incorporate it into the same patch to fix the gate. Or we could change to non-voting until it can be addressed.
The fix for Manila issue is proposed here: https://review.opendev.org/c/openstack/manila/+/976127
There is also a pkg_resources issue with the pep8 job, but that seems to be happening across more than one repo at the moment :(
And for anyone unaware, the thread for the pkg_resources issue is here: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... -melwitt
On 2/7/26 2:57 PM, Brian Haley wrote:
Hi Melanie,
Comments inline below.
On 2/6/26 6:00 PM, melanie witt wrote:
Hey all,
We noticed sometime today the openstacksdk-functional-devstack job appears to be failing 100% in multiple projects:
https://zuul.opendev.org/t/openstack/builds?job_name=openstacksdk- functional-devstack&skip=0
The failing openstacksdk test is test_add_remove_router_interface [1]:
ft1.1: openstack.tests.functional.cloud.test_router.TestRouter.test_add_remove_router_interfacetesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/openstacksdk/openstack/ tests/functional/cloud/test_router.py", line 232, in test_add_remove_router_interface self.assertIn(key, iface) File "/home/zuul/src/opendev.org/openstack/openstacksdk/.tox/ functional/lib/python3.12/site-packages/testtools/testcase.py", line 425, in assertIn self.assertThat(haystack, Contains(needle), message) File "/home/zuul/src/opendev.org/openstack/openstacksdk/.tox/ functional/lib/python3.12/site-packages/testtools/testcase.py", line 509, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: 'tenant_id' not in {'id': '57671080-6eb2-4c7f-9e64-c61edfa8995e', 'project_id': '1ef2e6c3fc2d4f6b92fdd8d49f0a05a1', 'port_id': '5b39bbfc-6d3d-440e-9c7a- e56e434ef897', 'network_id': '8aa6b6e2- e0c5-4f96-bb87-5509e30d340c', 'subnet_id': '6f52d8be-cf68-4552- b57d-37142cea65cc', 'subnet_ids': ['6f52d8be-cf68-4552- b57d-37142cea65cc']}
which adds an interface to a router and expects the response parameter 'tenant_id' to be in the response.
I looked for recently merged changes that could have possibly changed the router API response and it might be something similar to this change:
So I've been going through the neutron code trying to convert everything to use project_id this cycle when I found some TODOs from 12 years ago on removing tenant_id :-/
I tried to make sure we continue to accept tenant_id, but I guess in this case I also removed it from the response. Although I have canary patches in some of the networking repos I didn't have one for the SDK, I'll add it to my list going forward.
but I don't really understand if/how that could change the API.
My guess is it's because of this part of the change:
https://review.opendev.org/c/openstack/neutron/+/972982/14/neutron/db/ db_base_plugin_common.py
I'm not sure what the desired fix will be here, whether the API should be more backward compatible and/or if the openstacksdk test should be fixed by s/tenant_id/project_id/ or something else.
Please hold your rechecks if CI has failed on your patch due to this.
I've pushed a change to fix it as I think it's time to move forward:
https://review.opendev.org/c/openstack/openstacksdk/+/976008
If anyone thinks otherwise please let me know.
Thanks for the report and sorry about the breakage,
-Brian
[1] https://github.com/openstack/openstacksdk/ blob/7afe210333ec236a14fc5fa2d4a589e481a43745/openstack/tests/ functional/cloud/test_router.py#L205
Hi all, Update: I believe all of the fixes have landed and it is now safe to check your changes. -melwitt On 2/9/26 11:18, melanie witt wrote:
Thank you Brian for proposing the fix for openstacksdk! Also thanks for the pointer about how the API was affected.
I have put a couple of links inline for ML update purposes.
On 2/8/26 14:05, Brian Haley wrote:
Just an update that my patch solves the issue with the openstacksdk- functional-devstack job, but there is also an error in the -manila functional job as well, which was there before my neutron change merged. If someone could help debug that I'd appreciate it, might have to incorporate it into the same patch to fix the gate. Or we could change to non-voting until it can be addressed.
The fix for Manila issue is proposed here:
https://review.opendev.org/c/openstack/manila/+/976127
There is also a pkg_resources issue with the pep8 job, but that seems to be happening across more than one repo at the moment :(
And for anyone unaware, the thread for the pkg_resources issue is here:
https://lists.openstack.org/archives/list/openstack- discuss@lists.openstack.org/thread/374GJ6DM6DRPFF6QBWQ6UGMCDJVA4YLA/
-melwitt
On 2/7/26 2:57 PM, Brian Haley wrote:
Hi Melanie,
Comments inline below.
On 2/6/26 6:00 PM, melanie witt wrote:
Hey all,
We noticed sometime today the openstacksdk-functional-devstack job appears to be failing 100% in multiple projects:
https://zuul.opendev.org/t/openstack/builds?job_name=openstacksdk- functional-devstack&skip=0
The failing openstacksdk test is test_add_remove_router_interface [1]:
ft1.1: openstack.tests.functional.cloud.test_router.TestRouter.test_add_remove_router_interfacetesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/openstacksdk/ openstack/ tests/functional/cloud/test_router.py", line 232, in test_add_remove_router_interface self.assertIn(key, iface) File "/home/zuul/src/opendev.org/openstack/openstacksdk/.tox/ functional/lib/python3.12/site-packages/testtools/testcase.py", line 425, in assertIn self.assertThat(haystack, Contains(needle), message) File "/home/zuul/src/opendev.org/openstack/openstacksdk/.tox/ functional/lib/python3.12/site-packages/testtools/testcase.py", line 509, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: 'tenant_id' not in {'id': '57671080-6eb2-4c7f-9e64-c61edfa8995e', 'project_id': '1ef2e6c3fc2d4f6b92fdd8d49f0a05a1', 'port_id': '5b39bbfc-6d3d-440e-9c7a- e56e434ef897', 'network_id': '8aa6b6e2- e0c5-4f96-bb87-5509e30d340c', 'subnet_id': '6f52d8be-cf68-4552- b57d-37142cea65cc', 'subnet_ids': ['6f52d8be-cf68-4552- b57d-37142cea65cc']}
which adds an interface to a router and expects the response parameter 'tenant_id' to be in the response.
I looked for recently merged changes that could have possibly changed the router API response and it might be something similar to this change:
So I've been going through the neutron code trying to convert everything to use project_id this cycle when I found some TODOs from 12 years ago on removing tenant_id :-/
I tried to make sure we continue to accept tenant_id, but I guess in this case I also removed it from the response. Although I have canary patches in some of the networking repos I didn't have one for the SDK, I'll add it to my list going forward.
but I don't really understand if/how that could change the API.
My guess is it's because of this part of the change:
https://review.opendev.org/c/openstack/neutron/+/972982/14/neutron/ db/ db_base_plugin_common.py
I'm not sure what the desired fix will be here, whether the API should be more backward compatible and/or if the openstacksdk test should be fixed by s/tenant_id/project_id/ or something else.
Please hold your rechecks if CI has failed on your patch due to this.
I've pushed a change to fix it as I think it's time to move forward:
https://review.opendev.org/c/openstack/openstacksdk/+/976008
If anyone thinks otherwise please let me know.
Thanks for the report and sorry about the breakage,
-Brian
[1] https://github.com/openstack/openstacksdk/ blob/7afe210333ec236a14fc5fa2d4a589e481a43745/openstack/tests/ functional/cloud/test_router.py#L205
which adds an interface to a router and expects the response parameter 'tenant_id' to be in the response.
I looked for recently merged changes that could have possibly changed the router API response and it might be something similar to this change:
So I've been going through the neutron code trying to convert everything to use project_id this cycle when I found some TODOs from 12 years ago on removing tenant_id :-/
I tried to make sure we continue to accept tenant_id, but I guess in this case I also removed it from the response. Although I have canary patches in some of the networking repos I didn't have one for the SDK, I'll add it to my list going forward.
but I don't really understand if/how that could change the API.
My guess is it's because of this part of the change:
https://review.opendev.org/c/openstack/neutron/+/972982/14/neutron/db/ db_base_plugin_common.py
I'm not sure what the desired fix will be here, whether the API should be more backward compatible and/or if the openstacksdk test should be fixed by s/tenant_id/project_id/ or something else.
Please hold your rechecks if CI has failed on your patch due to this.
I've pushed a change to fix it as I think it's time to move forward:
https://review.opendev.org/c/openstack/openstacksdk/+/976008
Thanks melwitt, Brian. I will leave the API change decision to the neutron team, but this brought up a testing gap in our gate. I remember the past discussion about testing the sdk cross services and I added it in the integrated gate testing templates[1], but it seems Neutron does not run the tempest-defined integrated-gate-networking. I remember that concern was the duplicate testing of grenade jobs (by grenade base jobs defined in integrated-gate-networking and neutron grenade jobs). To solve that, I proposed to remove the grenade base jobs from integrated-gate-networking template and adding this template in neutron gate. This will help to detect these failures in advance. Please check if it makes sense. - https://review.opendev.org/c/openstack/tempest/+/976203 - https://review.opendev.org/c/openstack/neutron/+/976204 [1] https://review.opendev.org/c/openstack/tempest/+/801920 -gmaan
If anyone thinks otherwise please let me know.
Thanks for the report and sorry about the breakage,
-Brian
[1] https://github.com/openstack/openstacksdk/ blob/7afe210333ec236a14fc5fa2d4a589e481a43745/openstack/tests/ functional/cloud/test_router.py#L205
participants (3)
-
Brian Haley
-
Ghanshyam Maan
-
melanie witt