[Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches �C Suspected Regression in Neutron Master Branch
Hi Neutron Team, We are currently facing an issue in Tacker Zuul patch jobs build (with master branch), where the DevStack setup fails during the initial Neutron network creation step. Problem statement: The failure occurs at this point in the stack.sh script, for devstack environment creation in build process. Below command execution is failing in stack.sh [openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc d777d04b-93ff-4e94-b6a9-cc8de3e23921] Error Message: This results in a 500 Internal Server Error, as seen in the logs: [HttpException: 500: Server Error for url: http://10.x.x.x/networking/v2.0/routers/20b264df-5f19-4599-a83d-2e76e7cbf5dc...] Analysis for the issue: Below error is coming in Neutron API logs: [ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on un-acquired lock] 1. Tried overriding the required-projects in Zuul to use stable/2025.1 branches (instead of master) to identify project(master branch) which is causing root cause. 2. Tacker team has created DNM patch (952892) to invetsigate this in detail. https://review.opendev.org/c/openstack/tacker/+/952892 1. In DNM patch Patchset 11 (on June 29), we have taken devstack stable/2025.1. In stable branch the devstack environment created sucessfully. 2. In DNM patch Patchset 25 (on July 8), with the same devstack override (stable/2025.1), the job now fails earlier during DevStack setup. The logs show that Neutron failed to start, causing the job to fail across all nodes. As per observations and anlysis seems the recent changes in the Neutron master branch between June 29 and July 8 might be causing this issue. We’d appreciate your help in reviewing the changes in Neutron's master branch during that window that could cause this behavior and suggesting any debugging approaches or patches we can test on tacker. Please let us know if you'd like additional logs or info from the Zuul job runs. Thanks in advance for your support! Thanks & Regards, Renu Rani (レヌ ラニ) Mobile: +91-9311888962 E-mail: renu.rani@india.nec.com<mailto:renu.rani@india.nec.com> The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Hi Neutron Team, We are currently facing an issue in Tacker Zuul patch jobs build (with master branch), where the DevStack setup fails during the initial Neutron network creation step. Problem statement: The failure occurs at this point in the stack.sh script, for devstack environment creation in build process. Below command execution is failing in stack.sh [openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc d777d04b-93ff-4e94-b6a9-cc8de3e23921] Error Message: This results in a 500 Internal Server Error, as seen in the logs: [HttpException: 500: Server Error for url: http://10.x.x.x/networking/v2.0/routers/20b264df-5f19-4599-a83d-2e76e7cbf5dc...] Analysis for the issue: Below error is coming in Neutron API logs: [ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on un-acquired lock] 1. Tried overriding the required-projects in Zuul to use stable/2025.1 branches (instead of master) to identify project(master branch) which is causing root cause. 2. Tacker team has created DNM patch (952892) to invetsigate this in detail. https://review.opendev.org/c/openstack/tacker/+/952892 1. In DNM patch Patchset 11 (on June 29), we have taken devstack stable/2025.1. In stable branch the devstack environment created sucessfully. 2. In DNM patch Patchset 25 (on July 8), with the same devstack override (stable/2025.1), the job now fails earlier during DevStack setup. The logs show that Neutron failed to start, causing the job to fail across all nodes. As per observations and anlysis seems the recent changes in the Neutron master branch between June 29 and July 8 might be causing this issue. We’d appreciate your help in reviewing the changes in Neutron's master branch during that window that could cause this behavior and suggesting any debugging approaches or patches we can test on tacker. Please let us know if you'd like additional logs or info from the Zuul job runs. Thanks in advance for your support! Thanks & Regards, Renu Rani (レヌ ラニ) Mobile: +91-9311888962 E-mail: renu.rani@india.nec.com<mailto:renu.rani@india.nec.com> The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Hi Renu, On 7/15/25 3:16 AM, Renu Rani wrote:
Hi Neutron Team,
We are currently facing an issue in Tacker Zuul patch jobs build (with master branch), where the DevStack setup fails during the initial Neutron network creation step.
*Problem statement:*
The failure occurs at this point in the stack.sh script, for devstack environment creation in build process.
Below command execution is failing in stack.sh
*[openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc d777d04b-93ff-4e94-b6a9-cc8de3e23921]*
I looked quickly at the log files and it looks like you are also using the networking-sfc code. We just fixed an issue with eventlet there which could cause a bad interaction with neutron: https://review.opendev.org/c/openstack/networking-sfc/+/954633 I'm not sure if it's related but that change just merged an hour ago, so it might be worth re-running things to check. This was the version of networking-sfc I see being built: Successfully installed networking-sfc-20.1.0.dev3 And it was pulling-in eventlet (which was just removed in that patch): Requirement already satisfied: eventlet>=0.27.0 in /opt/stack/data/venv/lib/python3.12/site-packages (from networking-sfc==20.1.0.dev3) (0.40.1) If that doesn't help please file a bug against neutron so someone can look further. Thanks, -Brian
**
*Error Message:*
/This results in a 500 Internal Server Error, as seen in the logs:/
/[HttpException: 500: Server Error for url: http://10.x.x.x/networking/v2.0/routers/20b264df-5f19-4599-a83d-2e76e7cbf5dc... <http://10.x.x.x/networking/v2.0/routers/20b264df-5f19-4599-a83d-2e76e7cbf5dc/add_router_interface>]/
*Analysis for the issue:*
Below error is coming in Neutron API logs:
*[ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on un-acquired lock]***
1. Tried overriding the required-projects in Zuul to use stable/2025.1 branches (instead of master) to identify project(master branch) which is causing root cause. 2. Tacker team has created DNM patch (952892) to invetsigate this in detail.
https://review.opendev.org/c/openstack/tacker/+/952892 <https://review.opendev.org/c/openstack/tacker/+/952892>
3. In DNM patch Patchset 11 (on June 29), we have taken devstack stable/2025.1. In stable branch the devstack environment created sucessfully. 4. In DNM patch Patchset 25 (on July 8), with the same devstack override (stable/2025.1), the job now fails earlier during DevStack setup.
The logs show that Neutron failed to start, causing the job to fail across all nodes.
As per observations and anlysis seems the recent changes in the Neutron master branch between June 29 and July 8 might be causing this issue.
We’d appreciate your help in reviewing the changes in Neutron's master branch during that window that could cause this behavior and suggesting any debugging approaches or patches we can test on tacker.
Please let us know if you'd like additional logs or info from the Zuul job runs.
Thanks in advance for your support!
Thanks & Regards,
Renu Rani(レヌラニ)
Mobile: +91-9311888962
E-mail: renu.rani@india.nec.com <mailto:renu.rani@india.nec.com>
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Looking at the build result[1] in the DNM patch mentioned, I found that pip failed to install neutron from repo due to the following error. [1] https://zuul.opendev.org/t/openstack/build/ef1dbefc5dab40c9a6e13cb4badaa3b2 AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'? I think this indicates the issue is caused by incompatibility between pip and setuptools which is caused by recent multiple updates in packaging. One thing I noticed is that the job still runs in Ubuntu Jammy, although Jammy was removed from supported operating system in this cycle and should have been replaced by Ubuntu Noble. Probably you have to first try migrating these jobs to noble and see if that can resolve the issue. On 7/16/25 1:20 AM, Brian Haley wrote:
Hi Renu,
On 7/15/25 3:16 AM, Renu Rani wrote:
Hi Neutron Team,
We are currently facing an issue in Tacker Zuul patch jobs build (with master branch), where the DevStack setup fails during the initial Neutron network creation step.
*Problem statement:*
The failure occurs at this point in the stack.sh script, for devstack environment creation in build process.
Below command execution is failing in stack.sh
*[openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc d777d04b-93ff-4e94-b6a9-cc8de3e23921]*
I looked quickly at the log files and it looks like you are also using the networking-sfc code. We just fixed an issue with eventlet there which could cause a bad interaction with neutron:
https://review.opendev.org/c/openstack/networking-sfc/+/954633
I'm not sure if it's related but that change just merged an hour ago, so it might be worth re-running things to check.
This was the version of networking-sfc I see being built:
Successfully installed networking-sfc-20.1.0.dev3
And it was pulling-in eventlet (which was just removed in that patch):
Requirement already satisfied: eventlet>=0.27.0 in /opt/stack/data/venv/lib/python3.12/site-packages (from networking-sfc==20.1.0.dev3) (0.40.1)
If that doesn't help please file a bug against neutron so someone can look further.
Thanks,
-Brian
**
*Error Message:*
/This results in a 500 Internal Server Error, as seen in the logs:/
/[HttpException: 500: Server Error for url: http://10.x.x.x/networking/v2.0/routers/20b264df-5f19-4599-a83d-2e76e7cbf5dc... <http://10.x.x.x/networking/v2.0/routers/20b264df-5f19-4599-a83d-2e76e7cbf5dc/add_router_interface>]/
*Analysis for the issue:*
Below error is coming in Neutron API logs:
*[ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on un-acquired lock]***
1. Tried overriding the required-projects in Zuul to use stable/2025.1 branches (instead of master) to identify project(master branch) which is causing root cause. 2. Tacker team has created DNM patch (952892) to invetsigate this in detail.
https://review.opendev.org/c/openstack/tacker/+/952892 <https://review.opendev.org/c/openstack/tacker/+/952892>
3. In DNM patch Patchset 11 (on June 29), we have taken devstack stable/2025.1. In stable branch the devstack environment created sucessfully. 4. In DNM patch Patchset 25 (on July 8), with the same devstack override (stable/2025.1), the job now fails earlier during DevStack setup.
The logs show that Neutron failed to start, causing the job to fail across all nodes.
As per observations and anlysis seems the recent changes in the Neutron master branch between June 29 and July 8 might be causing this issue.
We’d appreciate your help in reviewing the changes in Neutron's master branch during that window that could cause this behavior and suggesting any debugging approaches or patches we can test on tacker.
Please let us know if you'd like additional logs or info from the Zuul job runs.
Thanks in advance for your support!
Thanks & Regards,
Renu Rani(レヌラニ)
Mobile: +91-9311888962
E-mail: renu.rani@india.nec.com <mailto:renu.rani@india.nec.com>
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Dear Takashi San, Thanks a lot for your response and really appreciate sharing your inputs. Just to update there are two types of Tacker jobs is check pipeline: one is Kubernetes-based, and the other is OpenStack-based. [1] Kubernetes-based Job: tacker-ft-v2-k8s It appears that you reviewed the logs for this job, which is currently failing due to the following error: "AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'?" We understand that this error is not related to Neutron changes, but rather due to an incompatibility between pip and setuptools, likely caused by recent packaging updates. We're actively investigating the root cause of this issue. [2] OpenStack-based Job: tacker-ft-v1-compliance-sol Our original request was regarding this job. It fails due to an "Internal Server Error" during the execution of the following command: "openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a8" Zuul Log (controller node): https://zuul.opendev.org/t/openstack/build/970baccf521846ec83441c1f18898199 To debug further, we tried checking out the stable/2025.1 branch of DevStack instead of master. However, the controller node now fails with this error: "[ERROR] /opt/stack/devstack/functions-common:2366 Neutron did not start" Zuul Log (controller node): https://zuul.opendev.org/t/openstack/build/09d39537443640c2a87673590f577829 Identified the specific commits that introduced these issues: DevStack Commit (leads to ��Internal Server Error��) �C May 20, 2025 [Commit sha: 988ce885d366dd0a7f8a012f9e52ee8f3403ffee] [https://github.com/openstack/devstack/commit/988ce885d366dd0a7f8a012f9e52ee8...] Neutron Commit (leads to ��Neutron did not start��) �C July 1, 2025 [Commit SHA: 1971c4a261ca54f077e2dec8b59774602bbbe3f5]] [https://github.com/openstack/neutron/commit/1971c4a261ca54f077e2dec8b5977460...]] Regards Renu -----Original Message----- From: Takashi Kajinami <kajinamit@oss.nttdata.com> Sent: Wednesday, July 16, 2025 12:42 PM To: Brian Haley <haleyb.dev@gmail.com>; Renu Rani <renu.rani@india.nec.com>; OpenStack Discuss <openstack-discuss@lists.openstack.org> Cc: Shivam Shukla <shivam.shukla3@india.nec.com>; Pankaj Kumar Pandey <pankaj.pandey@india.nec.com> Subject: Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches �C Suspected Regression in Neutron Master Branch [You don't often get email from kajinamit@oss.nttdata.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] External Message: Please be cautious when opening links or attachments in email Looking at the build result[1] in the DNM patch mentioned, I found that pip failed to install neutron from repo due to the following error. [1] https://zuul.opendev.org/t/openstack/build/ef1dbefc5dab40c9a6e13cb4badaa3b2 AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'? I think this indicates the issue is caused by incompatibility between pip and setuptools which is caused by recent multiple updates in packaging. One thing I noticed is that the job still runs in Ubuntu Jammy, although Jammy was removed from supported operating system in this cycle and should have been replaced by Ubuntu Noble. Probably you have to first try migrating these jobs to noble and see if that can resolve the issue. On 7/16/25 1:20 AM, Brian Haley wrote:
Hi Renu,
On 7/15/25 3:16 AM, Renu Rani wrote:
Hi Neutron Team,
We are currently facing an issue in Tacker Zuul patch jobs build (with master branch), where the DevStack setup fails during the initial Neutron network creation step.
*Problem statement:*
The failure occurs at this point in the stack.sh script, for devstack environment creation in build process.
Below command execution is failing in stack.sh
*[openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc d777d04b-93ff-4e94-b6a9-cc8de3e23921]*
I looked quickly at the log files and it looks like you are also using the networking-sfc code. We just fixed an issue with eventlet there which could cause a bad interaction with neutron:
https://revi/ ew.opendev.org%2Fc%2Fopenstack%2Fnetworking-sfc%2F%2B%2F954633&data=05 %7C02%7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7 Ccc4532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168427284%7CUnkno wn%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXa W4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=olPBCnubJho UYWt%2FCi6WoUJA2dzpiHKljXCxWs4lmG8%3D&reserved=0
I'm not sure if it's related but that change just merged an hour ago, so it might be worth re-running things to check.
This was the version of networking-sfc I see being built:
Successfully installed networking-sfc-20.1.0.dev3
And it was pulling-in eventlet (which was just removed in that patch):
Requirement already satisfied: eventlet>=0.27.0 in /opt/stack/data/venv/lib/python3.12/site-packages (from networking-sfc==20.1.0.dev3) (0.40.1)
If that doesn't help please file a bug against neutron so someone can look further.
Thanks,
-Brian
**
*Error Message:*
/This results in a 500 Internal Server Error, as seen in the logs:/
/[HttpException: 500: Server Error for url: http://10.x/ .x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7cb f5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com% 7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110c %7C0%7C0%7C638882467168459279%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGk iOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoy fQ%3D%3D%7C0%7C%7C%7C&sdata=qn%2FaQZX5%2FaGkPyInL8RKDpr1WhliJE%2BpJ90 HANXME%2FQ%3D&reserved=0 <http://0.0.0.10/. x.x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7c bf5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com %7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110 c%7C0%7C0%7C638882467168484477%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcG kiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjo yfQ%3D%3D%7C0%7C%7C%7C&sdata=HhIBD9MAnWFoMCBtXX5FY3k2tfvlKiBmMKDiTFuy iNQ%3D&reserved=0>]/
*Analysis for the issue:*
Below error is coming in Neutron API logs:
*[ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on un-acquired lock]***
1. Tried overriding the required-projects in Zuul to use stable/2025.1 branches (instead of master) to identify project(master branch) which is causing root cause. 2. Tacker team has created DNM patch (952892) to invetsigate this in detail.
https://rev/ iew.opendev.org%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02% 7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc45 32b813fc4545981297eef82e110c%7C0%7C0%7C638882467168506073%7CUnknown%7 CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4z MiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=anYUb6qkQlI%2 Fe1FSOcEtkdsr0iuv3OnNS96Dh2Zl4eY%3D&reserved=0 <https://re/ view.opendev.org%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02 %7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4 532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168521702%7CUnknown% 7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4 zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=0J1axj9cioqQ BuKReXSeCkA8rWpR2kPJ1YWmjQhG4DU%3D&reserved=0>
3. In DNM patch Patchset 11 (on June 29), we have taken devstack stable/2025.1. In stable branch the devstack environment created sucessfully. 4. In DNM patch Patchset 25 (on July 8), with the same devstack override (stable/2025.1), the job now fails earlier during DevStack setup.
The logs show that Neutron failed to start, causing the job to fail across all nodes.
As per observations and anlysis seems the recent changes in the Neutron master branch between June 29 and July 8 might be causing this issue.
We��d appreciate your help in reviewing the changes in Neutron's master branch during that window that could cause this behavior and suggesting any debugging approaches or patches we can test on tacker.
Please let us know if you'd like additional logs or info from the Zuul job runs.
Thanks in advance for your support!
Thanks & Regards,
Renu Rani(��̥��)
Mobile: +91-9311888962
E-mail: renu.rani@india.nec.com <mailto:renu.rani@india.nec.com>
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
participants (3)
-
Brian Haley
-
Renu Rani
-
Takashi Kajinami