[Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches �C Suspected Regression in Neutron Master Branch
Hi Neutron Team, We are currently facing an issue in Tacker Zuul patch jobs build (with master branch), where the DevStack setup fails during the initial Neutron network creation step. Problem statement: The failure occurs at this point in the stack.sh script, for devstack environment creation in build process. Below command execution is failing in stack.sh [openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc d777d04b-93ff-4e94-b6a9-cc8de3e23921] Error Message: This results in a 500 Internal Server Error, as seen in the logs: [HttpException: 500: Server Error for url: http://10.x.x.x/networking/v2.0/routers/20b264df-5f19-4599-a83d-2e76e7cbf5dc...] Analysis for the issue: Below error is coming in Neutron API logs: [ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on un-acquired lock] 1. Tried overriding the required-projects in Zuul to use stable/2025.1 branches (instead of master) to identify project(master branch) which is causing root cause. 2. Tacker team has created DNM patch (952892) to invetsigate this in detail. https://review.opendev.org/c/openstack/tacker/+/952892 1. In DNM patch Patchset 11 (on June 29), we have taken devstack stable/2025.1. In stable branch the devstack environment created sucessfully. 2. In DNM patch Patchset 25 (on July 8), with the same devstack override (stable/2025.1), the job now fails earlier during DevStack setup. The logs show that Neutron failed to start, causing the job to fail across all nodes. As per observations and anlysis seems the recent changes in the Neutron master branch between June 29 and July 8 might be causing this issue. We’d appreciate your help in reviewing the changes in Neutron's master branch during that window that could cause this behavior and suggesting any debugging approaches or patches we can test on tacker. Please let us know if you'd like additional logs or info from the Zuul job runs. Thanks in advance for your support! Thanks & Regards, Renu Rani (レヌ ラニ) Mobile: +91-9311888962 E-mail: renu.rani@india.nec.com<mailto:renu.rani@india.nec.com> The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Hi Neutron Team, We are currently facing an issue in Tacker Zuul patch jobs build (with master branch), where the DevStack setup fails during the initial Neutron network creation step. Problem statement: The failure occurs at this point in the stack.sh script, for devstack environment creation in build process. Below command execution is failing in stack.sh [openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc d777d04b-93ff-4e94-b6a9-cc8de3e23921] Error Message: This results in a 500 Internal Server Error, as seen in the logs: [HttpException: 500: Server Error for url: http://10.x.x.x/networking/v2.0/routers/20b264df-5f19-4599-a83d-2e76e7cbf5dc...] Analysis for the issue: Below error is coming in Neutron API logs: [ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on un-acquired lock] 1. Tried overriding the required-projects in Zuul to use stable/2025.1 branches (instead of master) to identify project(master branch) which is causing root cause. 2. Tacker team has created DNM patch (952892) to invetsigate this in detail. https://review.opendev.org/c/openstack/tacker/+/952892 1. In DNM patch Patchset 11 (on June 29), we have taken devstack stable/2025.1. In stable branch the devstack environment created sucessfully. 2. In DNM patch Patchset 25 (on July 8), with the same devstack override (stable/2025.1), the job now fails earlier during DevStack setup. The logs show that Neutron failed to start, causing the job to fail across all nodes. As per observations and anlysis seems the recent changes in the Neutron master branch between June 29 and July 8 might be causing this issue. We’d appreciate your help in reviewing the changes in Neutron's master branch during that window that could cause this behavior and suggesting any debugging approaches or patches we can test on tacker. Please let us know if you'd like additional logs or info from the Zuul job runs. Thanks in advance for your support! Thanks & Regards, Renu Rani (レヌ ラニ) Mobile: +91-9311888962 E-mail: renu.rani@india.nec.com<mailto:renu.rani@india.nec.com> The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Hi Renu, On 7/15/25 3:16 AM, Renu Rani wrote:
Hi Neutron Team,
We are currently facing an issue in Tacker Zuul patch jobs build (with master branch), where the DevStack setup fails during the initial Neutron network creation step.
*Problem statement:*
The failure occurs at this point in the stack.sh script, for devstack environment creation in build process.
Below command execution is failing in stack.sh
*[openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc d777d04b-93ff-4e94-b6a9-cc8de3e23921]*
I looked quickly at the log files and it looks like you are also using the networking-sfc code. We just fixed an issue with eventlet there which could cause a bad interaction with neutron: https://review.opendev.org/c/openstack/networking-sfc/+/954633 I'm not sure if it's related but that change just merged an hour ago, so it might be worth re-running things to check. This was the version of networking-sfc I see being built: Successfully installed networking-sfc-20.1.0.dev3 And it was pulling-in eventlet (which was just removed in that patch): Requirement already satisfied: eventlet>=0.27.0 in /opt/stack/data/venv/lib/python3.12/site-packages (from networking-sfc==20.1.0.dev3) (0.40.1) If that doesn't help please file a bug against neutron so someone can look further. Thanks, -Brian
**
*Error Message:*
/This results in a 500 Internal Server Error, as seen in the logs:/
/[HttpException: 500: Server Error for url: http://10.x.x.x/networking/v2.0/routers/20b264df-5f19-4599-a83d-2e76e7cbf5dc... <http://10.x.x.x/networking/v2.0/routers/20b264df-5f19-4599-a83d-2e76e7cbf5dc/add_router_interface>]/
*Analysis for the issue:*
Below error is coming in Neutron API logs:
*[ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on un-acquired lock]***
1. Tried overriding the required-projects in Zuul to use stable/2025.1 branches (instead of master) to identify project(master branch) which is causing root cause. 2. Tacker team has created DNM patch (952892) to invetsigate this in detail.
https://review.opendev.org/c/openstack/tacker/+/952892 <https://review.opendev.org/c/openstack/tacker/+/952892>
3. In DNM patch Patchset 11 (on June 29), we have taken devstack stable/2025.1. In stable branch the devstack environment created sucessfully. 4. In DNM patch Patchset 25 (on July 8), with the same devstack override (stable/2025.1), the job now fails earlier during DevStack setup.
The logs show that Neutron failed to start, causing the job to fail across all nodes.
As per observations and anlysis seems the recent changes in the Neutron master branch between June 29 and July 8 might be causing this issue.
We’d appreciate your help in reviewing the changes in Neutron's master branch during that window that could cause this behavior and suggesting any debugging approaches or patches we can test on tacker.
Please let us know if you'd like additional logs or info from the Zuul job runs.
Thanks in advance for your support!
Thanks & Regards,
Renu Rani(レヌラニ)
Mobile: +91-9311888962
E-mail: renu.rani@india.nec.com <mailto:renu.rani@india.nec.com>
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Looking at the build result[1] in the DNM patch mentioned, I found that pip failed to install neutron from repo due to the following error. [1] https://zuul.opendev.org/t/openstack/build/ef1dbefc5dab40c9a6e13cb4badaa3b2 AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'? I think this indicates the issue is caused by incompatibility between pip and setuptools which is caused by recent multiple updates in packaging. One thing I noticed is that the job still runs in Ubuntu Jammy, although Jammy was removed from supported operating system in this cycle and should have been replaced by Ubuntu Noble. Probably you have to first try migrating these jobs to noble and see if that can resolve the issue. On 7/16/25 1:20 AM, Brian Haley wrote:
Hi Renu,
On 7/15/25 3:16 AM, Renu Rani wrote:
Hi Neutron Team,
We are currently facing an issue in Tacker Zuul patch jobs build (with master branch), where the DevStack setup fails during the initial Neutron network creation step.
*Problem statement:*
The failure occurs at this point in the stack.sh script, for devstack environment creation in build process.
Below command execution is failing in stack.sh
*[openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc d777d04b-93ff-4e94-b6a9-cc8de3e23921]*
I looked quickly at the log files and it looks like you are also using the networking-sfc code. We just fixed an issue with eventlet there which could cause a bad interaction with neutron:
https://review.opendev.org/c/openstack/networking-sfc/+/954633
I'm not sure if it's related but that change just merged an hour ago, so it might be worth re-running things to check.
This was the version of networking-sfc I see being built:
Successfully installed networking-sfc-20.1.0.dev3
And it was pulling-in eventlet (which was just removed in that patch):
Requirement already satisfied: eventlet>=0.27.0 in /opt/stack/data/venv/lib/python3.12/site-packages (from networking-sfc==20.1.0.dev3) (0.40.1)
If that doesn't help please file a bug against neutron so someone can look further.
Thanks,
-Brian
**
*Error Message:*
/This results in a 500 Internal Server Error, as seen in the logs:/
/[HttpException: 500: Server Error for url: http://10.x.x.x/networking/v2.0/routers/20b264df-5f19-4599-a83d-2e76e7cbf5dc... <http://10.x.x.x/networking/v2.0/routers/20b264df-5f19-4599-a83d-2e76e7cbf5dc/add_router_interface>]/
*Analysis for the issue:*
Below error is coming in Neutron API logs:
*[ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on un-acquired lock]***
1. Tried overriding the required-projects in Zuul to use stable/2025.1 branches (instead of master) to identify project(master branch) which is causing root cause. 2. Tacker team has created DNM patch (952892) to invetsigate this in detail.
https://review.opendev.org/c/openstack/tacker/+/952892 <https://review.opendev.org/c/openstack/tacker/+/952892>
3. In DNM patch Patchset 11 (on June 29), we have taken devstack stable/2025.1. In stable branch the devstack environment created sucessfully. 4. In DNM patch Patchset 25 (on July 8), with the same devstack override (stable/2025.1), the job now fails earlier during DevStack setup.
The logs show that Neutron failed to start, causing the job to fail across all nodes.
As per observations and anlysis seems the recent changes in the Neutron master branch between June 29 and July 8 might be causing this issue.
We’d appreciate your help in reviewing the changes in Neutron's master branch during that window that could cause this behavior and suggesting any debugging approaches or patches we can test on tacker.
Please let us know if you'd like additional logs or info from the Zuul job runs.
Thanks in advance for your support!
Thanks & Regards,
Renu Rani(レヌラニ)
Mobile: +91-9311888962
E-mail: renu.rani@india.nec.com <mailto:renu.rani@india.nec.com>
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Dear Takashi San, Thanks a lot for your response and really appreciate sharing your inputs. Just to update there are two types of Tacker jobs is check pipeline: one is Kubernetes-based, and the other is OpenStack-based. [1] Kubernetes-based Job: tacker-ft-v2-k8s It appears that you reviewed the logs for this job, which is currently failing due to the following error: "AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'?" We understand that this error is not related to Neutron changes, but rather due to an incompatibility between pip and setuptools, likely caused by recent packaging updates. We're actively investigating the root cause of this issue. [2] OpenStack-based Job: tacker-ft-v1-compliance-sol Our original request was regarding this job. It fails due to an "Internal Server Error" during the execution of the following command: "openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a8" Zuul Log (controller node): https://zuul.opendev.org/t/openstack/build/970baccf521846ec83441c1f18898199 To debug further, we tried checking out the stable/2025.1 branch of DevStack instead of master. However, the controller node now fails with this error: "[ERROR] /opt/stack/devstack/functions-common:2366 Neutron did not start" Zuul Log (controller node): https://zuul.opendev.org/t/openstack/build/09d39537443640c2a87673590f577829 Identified the specific commits that introduced these issues: DevStack Commit (leads to ��Internal Server Error��) �C May 20, 2025 [Commit sha: 988ce885d366dd0a7f8a012f9e52ee8f3403ffee] [https://github.com/openstack/devstack/commit/988ce885d366dd0a7f8a012f9e52ee8...] Neutron Commit (leads to ��Neutron did not start��) �C July 1, 2025 [Commit SHA: 1971c4a261ca54f077e2dec8b59774602bbbe3f5]] [https://github.com/openstack/neutron/commit/1971c4a261ca54f077e2dec8b5977460...]] Regards Renu -----Original Message----- From: Takashi Kajinami <kajinamit@oss.nttdata.com> Sent: Wednesday, July 16, 2025 12:42 PM To: Brian Haley <haleyb.dev@gmail.com>; Renu Rani <renu.rani@india.nec.com>; OpenStack Discuss <openstack-discuss@lists.openstack.org> Cc: Shivam Shukla <shivam.shukla3@india.nec.com>; Pankaj Kumar Pandey <pankaj.pandey@india.nec.com> Subject: Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches �C Suspected Regression in Neutron Master Branch [You don't often get email from kajinamit@oss.nttdata.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] External Message: Please be cautious when opening links or attachments in email Looking at the build result[1] in the DNM patch mentioned, I found that pip failed to install neutron from repo due to the following error. [1] https://zuul.opendev.org/t/openstack/build/ef1dbefc5dab40c9a6e13cb4badaa3b2 AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'? I think this indicates the issue is caused by incompatibility between pip and setuptools which is caused by recent multiple updates in packaging. One thing I noticed is that the job still runs in Ubuntu Jammy, although Jammy was removed from supported operating system in this cycle and should have been replaced by Ubuntu Noble. Probably you have to first try migrating these jobs to noble and see if that can resolve the issue. On 7/16/25 1:20 AM, Brian Haley wrote:
Hi Renu,
On 7/15/25 3:16 AM, Renu Rani wrote:
Hi Neutron Team,
We are currently facing an issue in Tacker Zuul patch jobs build (with master branch), where the DevStack setup fails during the initial Neutron network creation step.
*Problem statement:*
The failure occurs at this point in the stack.sh script, for devstack environment creation in build process.
Below command execution is failing in stack.sh
*[openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc d777d04b-93ff-4e94-b6a9-cc8de3e23921]*
I looked quickly at the log files and it looks like you are also using the networking-sfc code. We just fixed an issue with eventlet there which could cause a bad interaction with neutron:
https://revi/ ew.opendev.org%2Fc%2Fopenstack%2Fnetworking-sfc%2F%2B%2F954633&data=05 %7C02%7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7 Ccc4532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168427284%7CUnkno wn%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXa W4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=olPBCnubJho UYWt%2FCi6WoUJA2dzpiHKljXCxWs4lmG8%3D&reserved=0
I'm not sure if it's related but that change just merged an hour ago, so it might be worth re-running things to check.
This was the version of networking-sfc I see being built:
Successfully installed networking-sfc-20.1.0.dev3
And it was pulling-in eventlet (which was just removed in that patch):
Requirement already satisfied: eventlet>=0.27.0 in /opt/stack/data/venv/lib/python3.12/site-packages (from networking-sfc==20.1.0.dev3) (0.40.1)
If that doesn't help please file a bug against neutron so someone can look further.
Thanks,
-Brian
**
*Error Message:*
/This results in a 500 Internal Server Error, as seen in the logs:/
/[HttpException: 500: Server Error for url: http://10.x/ .x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7cb f5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com% 7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110c %7C0%7C0%7C638882467168459279%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGk iOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoy fQ%3D%3D%7C0%7C%7C%7C&sdata=qn%2FaQZX5%2FaGkPyInL8RKDpr1WhliJE%2BpJ90 HANXME%2FQ%3D&reserved=0 <http://0.0.0.10/. x.x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7c bf5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com %7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110 c%7C0%7C0%7C638882467168484477%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcG kiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjo yfQ%3D%3D%7C0%7C%7C%7C&sdata=HhIBD9MAnWFoMCBtXX5FY3k2tfvlKiBmMKDiTFuy iNQ%3D&reserved=0>]/
*Analysis for the issue:*
Below error is coming in Neutron API logs:
*[ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on un-acquired lock]***
1. Tried overriding the required-projects in Zuul to use stable/2025.1 branches (instead of master) to identify project(master branch) which is causing root cause. 2. Tacker team has created DNM patch (952892) to invetsigate this in detail.
https://rev/ iew.opendev.org%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02% 7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc45 32b813fc4545981297eef82e110c%7C0%7C0%7C638882467168506073%7CUnknown%7 CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4z MiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=anYUb6qkQlI%2 Fe1FSOcEtkdsr0iuv3OnNS96Dh2Zl4eY%3D&reserved=0 <https://re/ view.opendev.org%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02 %7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4 532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168521702%7CUnknown% 7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4 zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=0J1axj9cioqQ BuKReXSeCkA8rWpR2kPJ1YWmjQhG4DU%3D&reserved=0>
3. In DNM patch Patchset 11 (on June 29), we have taken devstack stable/2025.1. In stable branch the devstack environment created sucessfully. 4. In DNM patch Patchset 25 (on July 8), with the same devstack override (stable/2025.1), the job now fails earlier during DevStack setup.
The logs show that Neutron failed to start, causing the job to fail across all nodes.
As per observations and anlysis seems the recent changes in the Neutron master branch between June 29 and July 8 might be causing this issue.
We��d appreciate your help in reviewing the changes in Neutron's master branch during that window that could cause this behavior and suggesting any debugging approaches or patches we can test on tacker.
Please let us know if you'd like additional logs or info from the Zuul job runs.
Thanks in advance for your support!
Thanks & Regards,
Renu Rani(��̥��)
Mobile: +91-9311888962
E-mail: renu.rani@india.nec.com <mailto:renu.rani@india.nec.com>
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Dear Takashi San, Thanks for your support. Please fine below further investigation updates from Tacker: 1. Kubernetes-Based Job – tacker-ft-v2-k8s Error: "AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'?" Root Cause: The issue is caused by the default setuptools version (59.6.0) included in the base OS image (Ubuntu Jammy). Since Global_Venv=False is set in local.conf, DevStack does not use a virtual environment, leading to the use of the outdated setuptools version. The required attribute is only available from setuptools version 64 onward. Fix: Set Global_Venv=True in local.conf. This enables the DevStack virtual environment where setuptools version 80 is installed, resolving the issue by using the correct setuptools version. 2.OpenStack-Based Job – tacker-ft-v1-compliance-sol Error#1: Internal Server Error during execution of the command: "openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a8" Root Cause: Below devstack commit started causing this issue. DevStack Commit (leads to “Internal Server Error”) – May 20, 2025 --> Seems Devstack code update related to neutron. [Commit sha: 988ce885d366dd0a7f8a012f9e52ee8f3403ffee] Error#2: [ERROR] Neutron did not start Root Cause: Below Neutron commit started causing this issue: Neutron Commit (leads to “Neutron did not start”) – July 1, 2025 [Commit SHA: 1971c4a261ca54f077e2dec8b59774602bbbe3f5]] Regards Renu -----Original Message----- From: Renu Rani <renu.rani@india.nec.com> Sent: Thursday, July 17, 2025 2:56 PM To: Takashi Kajinami <kajinamit@oss.nttdata.com>; Brian Haley <haleyb.dev@gmail.com>; OpenStack Discuss <openstack-discuss@lists.openstack.org> Cc: Shivam Shukla <shivam.shukla3@india.nec.com>; Pankaj Kumar Pandey <pankaj.pandey@india.nec.com> Subject: RE: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch Dear Takashi San, Thanks a lot for your response and really appreciate sharing your inputs. Just to update there are two types of Tacker jobs is check pipeline: one is Kubernetes-based, and the other is OpenStack-based. [1] Kubernetes-based Job: tacker-ft-v2-k8s It appears that you reviewed the logs for this job, which is currently failing due to the following error: "AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'?" We understand that this error is not related to Neutron changes, but rather due to an incompatibility between pip and setuptools, likely caused by recent packaging updates. We're actively investigating the root cause of this issue. [2] OpenStack-based Job: tacker-ft-v1-compliance-sol Our original request was regarding this job. It fails due to an "Internal Server Error" during the execution of the following command: "openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a8" Zuul Log (controller node): https://zuul.opendev.org/t/openstack/build/970baccf521846ec83441c1f18898199 To debug further, we tried checking out the stable/2025.1 branch of DevStack instead of master. However, the controller node now fails with this error: "[ERROR] /opt/stack/devstack/functions-common:2366 Neutron did not start" Zuul Log (controller node): https://zuul.opendev.org/t/openstack/build/09d39537443640c2a87673590f577829 Identified the specific commits that introduced these issues: DevStack Commit (leads to “Internal Server Error”) – May 20, 2025 [Commit sha: 988ce885d366dd0a7f8a012f9e52ee8f3403ffee] [https://github.com/openstack/devstack/commit/988ce885d366dd0a7f8a012f9e52ee8...] Neutron Commit (leads to “Neutron did not start”) – July 1, 2025 [Commit SHA: 1971c4a261ca54f077e2dec8b59774602bbbe3f5]] [https://github.com/openstack/neutron/commit/1971c4a261ca54f077e2dec8b5977460...]] Regards Renu -----Original Message----- From: Takashi Kajinami <kajinamit@oss.nttdata.com> Sent: Wednesday, July 16, 2025 12:42 PM To: Brian Haley <haleyb.dev@gmail.com>; Renu Rani <renu.rani@india.nec.com>; OpenStack Discuss <openstack-discuss@lists.openstack.org> Cc: Shivam Shukla <shivam.shukla3@india.nec.com>; Pankaj Kumar Pandey <pankaj.pandey@india.nec.com> Subject: Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch [You don't often get email from kajinamit@oss.nttdata.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] External Message: Please be cautious when opening links or attachments in email Looking at the build result[1] in the DNM patch mentioned, I found that pip failed to install neutron from repo due to the following error. [1] https://zuul.opendev.org/t/openstack/build/ef1dbefc5dab40c9a6e13cb4badaa3b2 AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'? I think this indicates the issue is caused by incompatibility between pip and setuptools which is caused by recent multiple updates in packaging. One thing I noticed is that the job still runs in Ubuntu Jammy, although Jammy was removed from supported operating system in this cycle and should have been replaced by Ubuntu Noble. Probably you have to first try migrating these jobs to noble and see if that can resolve the issue. On 7/16/25 1:20 AM, Brian Haley wrote:
Hi Renu,
On 7/15/25 3:16 AM, Renu Rani wrote:
Hi Neutron Team,
We are currently facing an issue in Tacker Zuul patch jobs build (with master branch), where the DevStack setup fails during the initial Neutron network creation step.
*Problem statement:*
The failure occurs at this point in the stack.sh script, for devstack environment creation in build process.
Below command execution is failing in stack.sh
*[openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc d777d04b-93ff-4e94-b6a9-cc8de3e23921]*
I looked quickly at the log files and it looks like you are also using the networking-sfc code. We just fixed an issue with eventlet there which could cause a bad interaction with neutron:
https://revi/ ew.opendev.org%2Fc%2Fopenstack%2Fnetworking-sfc%2F%2B%2F954633&data=05 %7C02%7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7 Ccc4532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168427284%7CUnkno wn%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXa W4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=olPBCnubJho UYWt%2FCi6WoUJA2dzpiHKljXCxWs4lmG8%3D&reserved=0
I'm not sure if it's related but that change just merged an hour ago, so it might be worth re-running things to check.
This was the version of networking-sfc I see being built:
Successfully installed networking-sfc-20.1.0.dev3
And it was pulling-in eventlet (which was just removed in that patch):
Requirement already satisfied: eventlet>=0.27.0 in /opt/stack/data/venv/lib/python3.12/site-packages (from networking-sfc==20.1.0.dev3) (0.40.1)
If that doesn't help please file a bug against neutron so someone can look further.
Thanks,
-Brian
**
*Error Message:*
/This results in a 500 Internal Server Error, as seen in the logs:/
/[HttpException: 500: Server Error for url: http://10.x/ .x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7cb f5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com% 7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110c %7C0%7C0%7C638882467168459279%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGk iOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoy fQ%3D%3D%7C0%7C%7C%7C&sdata=qn%2FaQZX5%2FaGkPyInL8RKDpr1WhliJE%2BpJ90 HANXME%2FQ%3D&reserved=0 <http://0.0.0.10/. x.x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7c bf5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com %7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110 c%7C0%7C0%7C638882467168484477%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcG kiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjo yfQ%3D%3D%7C0%7C%7C%7C&sdata=HhIBD9MAnWFoMCBtXX5FY3k2tfvlKiBmMKDiTFuy iNQ%3D&reserved=0>]/
*Analysis for the issue:*
Below error is coming in Neutron API logs:
*[ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on un-acquired lock]***
1. Tried overriding the required-projects in Zuul to use stable/2025.1 branches (instead of master) to identify project(master branch) which is causing root cause. 2. Tacker team has created DNM patch (952892) to invetsigate this in detail.
https://rev/ iew.opendev.org%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02% 7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc45 32b813fc4545981297eef82e110c%7C0%7C0%7C638882467168506073%7CUnknown%7 CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4z MiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=anYUb6qkQlI%2 Fe1FSOcEtkdsr0iuv3OnNS96Dh2Zl4eY%3D&reserved=0 <https://re/ view.opendev.org%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02 %7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4 532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168521702%7CUnknown% 7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4 zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=0J1axj9cioqQ BuKReXSeCkA8rWpR2kPJ1YWmjQhG4DU%3D&reserved=0>
3. In DNM patch Patchset 11 (on June 29), we have taken devstack stable/2025.1. In stable branch the devstack environment created sucessfully. 4. In DNM patch Patchset 25 (on July 8), with the same devstack override (stable/2025.1), the job now fails earlier during DevStack setup.
The logs show that Neutron failed to start, causing the job to fail across all nodes.
As per observations and anlysis seems the recent changes in the Neutron master branch between June 29 and July 8 might be causing this issue.
We’d appreciate your help in reviewing the changes in Neutron's master branch during that window that could cause this behavior and suggesting any debugging approaches or patches we can test on tacker.
Please let us know if you'd like additional logs or info from the Zuul job runs.
Thanks in advance for your support!
Thanks & Regards,
Renu Rani(レヌラニ)
Mobile: +91-9311888962
E-mail: renu.rani@india.nec.com <mailto:renu.rani@india.nec.com>
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Hi, On Mon, Jul 21, 2025 at 8:22 PM Renu Rani <renu.rani@india.nec.com> wrote:
Dear Takashi San,
Thanks for your support.
Please fine below further investigation updates from Tacker:
1. Kubernetes-Based Job – tacker-ft-v2-k8s Error: "AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'?" Root Cause: The issue is caused by the default setuptools version (59.6.0) included in the base OS image (Ubuntu Jammy). Since Global_Venv=False is set in local.conf, DevStack does not use a virtual environment, leading to the use of the outdated setuptools version. The required attribute is only available from setuptools version 64 onward. Fix: Set Global_Venv=True in local.conf. This enables the DevStack virtual environment where setuptools version 80 is installed, resolving the issue by using the correct setuptools version.
2.OpenStack-Based Job – tacker-ft-v1-compliance-sol Error#1: Internal Server Error during execution of the command: "openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a8" Root Cause: Below devstack commit started causing this issue. DevStack Commit (leads to “Internal Server Error”) – May 20, 2025 --> Seems Devstack code update related to neutron. [Commit sha: 988ce885d366dd0a7f8a012f9e52ee8f3403ffee]
Error#2: [ERROR] Neutron did not start Root Cause: Below Neutron commit started causing this issue: Neutron Commit (leads to “Neutron did not start”) – July 1, 2025 [Commit SHA: 1971c4a261ca54f077e2dec8b59774602bbbe3f5]]
For this I suggested some changes in the test patch https://review.opendev.org/c/openstack/tacker/+/952892, hopefully that will clear it.
Regards
Renu
-----Original Message----- From: Renu Rani <renu.rani@india.nec.com> Sent: Thursday, July 17, 2025 2:56 PM To: Takashi Kajinami <kajinamit@oss.nttdata.com>; Brian Haley < haleyb.dev@gmail.com>; OpenStack Discuss < openstack-discuss@lists.openstack.org> Cc: Shivam Shukla <shivam.shukla3@india.nec.com>; Pankaj Kumar Pandey < pankaj.pandey@india.nec.com> Subject: RE: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch
Dear Takashi San,
Thanks a lot for your response and really appreciate sharing your inputs.
Just to update there are two types of Tacker jobs is check pipeline: one is Kubernetes-based, and the other is OpenStack-based.
[1] Kubernetes-based Job: tacker-ft-v2-k8s It appears that you reviewed the logs for this job, which is currently failing due to the following error: "AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'?"
We understand that this error is not related to Neutron changes, but rather due to an incompatibility between pip and setuptools, likely caused by recent packaging updates. We're actively investigating the root cause of this issue.
[2] OpenStack-based Job: tacker-ft-v1-compliance-sol Our original request was regarding this job. It fails due to an "Internal Server Error" during the execution of the following command: "openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a8" Zuul Log (controller node): https://zuul.opendev.org/t/openstack/build/970baccf521846ec83441c1f18898199
To debug further, we tried checking out the stable/2025.1 branch of DevStack instead of master. However, the controller node now fails with this error: "[ERROR] /opt/stack/devstack/functions-common:2366 Neutron did not start" Zuul Log (controller node): https://zuul.opendev.org/t/openstack/build/09d39537443640c2a87673590f577829
Identified the specific commits that introduced these issues:
DevStack Commit (leads to “Internal Server Error”) – May 20, 2025 [Commit sha: 988ce885d366dd0a7f8a012f9e52ee8f3403ffee] [ https://github.com/openstack/devstack/commit/988ce885d366dd0a7f8a012f9e52ee8... ]
Neutron Commit (leads to “Neutron did not start”) – July 1, 2025 [Commit SHA: 1971c4a261ca54f077e2dec8b59774602bbbe3f5]] [ https://github.com/openstack/neutron/commit/1971c4a261ca54f077e2dec8b5977460... ]]
Regards Renu
-----Original Message----- From: Takashi Kajinami <kajinamit@oss.nttdata.com> Sent: Wednesday, July 16, 2025 12:42 PM To: Brian Haley <haleyb.dev@gmail.com>; Renu Rani <renu.rani@india.nec.com>; OpenStack Discuss <openstack-discuss@lists.openstack.org> Cc: Shivam Shukla <shivam.shukla3@india.nec.com>; Pankaj Kumar Pandey < pankaj.pandey@india.nec.com> Subject: Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch
[You don't often get email from kajinamit@oss.nttdata.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
External Message: Please be cautious when opening links or attachments in email
Looking at the build result[1] in the DNM patch mentioned, I found that pip failed to install neutron from repo due to the following error. [1] https://zuul.opendev.org/t/openstack/build/ef1dbefc5dab40c9a6e13cb4badaa3b2
AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'?
I think this indicates the issue is caused by incompatibility between pip and setuptools which is caused by recent multiple updates in packaging.
One thing I noticed is that the job still runs in Ubuntu Jammy, although Jammy was removed from supported operating system in this cycle and should have been replaced by Ubuntu Noble. Probably you have to first try migrating these jobs to noble and see if that can resolve the issue.
Hi Renu,
On 7/15/25 3:16 AM, Renu Rani wrote:
Hi Neutron Team,
We are currently facing an issue in Tacker Zuul patch jobs build (with master branch), where the DevStack setup fails during the initial Neutron network creation step.
*Problem statement:*
The failure occurs at this point in the stack.sh script, for devstack environment creation in build process.
Below command execution is failing in stack.sh
*[openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc d777d04b-93ff-4e94-b6a9-cc8de3e23921]*
I looked quickly at the log files and it looks like you are also using
On 7/16/25 1:20 AM, Brian Haley wrote: the networking-sfc code. We just fixed an issue with eventlet there which could cause a bad interaction with neutron:
https://revi/ ew.opendev.org%2Fc%2Fopenstack%2Fnetworking-sfc%2F%2B%2F954633&data=05 %7C02%7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7 Ccc4532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168427284%7CUnkno wn%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXa W4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=olPBCnubJho UYWt%2FCi6WoUJA2dzpiHKljXCxWs4lmG8%3D&reserved=0
I'm not sure if it's related but that change just merged an hour ago, so
it might be worth re-running things to check.
This was the version of networking-sfc I see being built:
Successfully installed networking-sfc-20.1.0.dev3
And it was pulling-in eventlet (which was just removed in that patch):
Requirement already satisfied: eventlet>=0.27.0 in /opt/stack/data/venv/lib/python3.12/site-packages (from networking-sfc==20.1.0.dev3) (0.40.1)
If that doesn't help please file a bug against neutron so someone can
look further.
Thanks,
-Brian
**
*Error Message:*
/This results in a 500 Internal Server Error, as seen in the logs:/
/[HttpException: 500: Server Error for url: http://10.x/ .x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7cb f5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com% 7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110c %7C0%7C0%7C638882467168459279%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGk iOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoy fQ%3D%3D%7C0%7C%7C%7C&sdata=qn%2FaQZX5%2FaGkPyInL8RKDpr1WhliJE%2BpJ90 HANXME%2FQ%3D&reserved=0 <http://0.0.0.10/. x.x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7c bf5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com %7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110 c%7C0%7C0%7C638882467168484477%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcG kiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjo yfQ%3D%3D%7C0%7C%7C%7C&sdata=HhIBD9MAnWFoMCBtXX5FY3k2tfvlKiBmMKDiTFuy iNQ%3D&reserved=0>]/
*Analysis for the issue:*
Below error is coming in Neutron API logs:
*[ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on un-acquired lock]***
1. Tried overriding the required-projects in Zuul to use stable/2025.1 branches (instead of master) to identify project(master branch) which is causing root cause. 2. Tacker team has created DNM patch (952892) to invetsigate this in detail.
https://rev/ iew.opendev.org%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02% 7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc45 32b813fc4545981297eef82e110c%7C0%7C0%7C638882467168506073%7CUnknown%7 CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4z MiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=anYUb6qkQlI%2 Fe1FSOcEtkdsr0iuv3OnNS96Dh2Zl4eY%3D&reserved=0 <https://re/ view.opendev.org%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02 %7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4 532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168521702%7CUnknown% 7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4 zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=0J1axj9cioqQ BuKReXSeCkA8rWpR2kPJ1YWmjQhG4DU%3D&reserved=0>
3. In DNM patch Patchset 11 (on June 29), we have taken devstack stable/2025.1. In stable branch the devstack environment created sucessfully. 4. In DNM patch Patchset 25 (on July 8), with the same devstack override (stable/2025.1), the job now fails earlier during DevStack setup.
The logs show that Neutron failed to start, causing the job to fail
As per observations and anlysis seems the recent changes in the Neutron
master branch between June 29 and July 8 might be causing this issue.
We’d appreciate your help in reviewing the changes in Neutron's master
branch during that window that could cause this behavior and suggesting any debugging approaches or patches we can test on tacker.
Please let us know if you'd like additional logs or info from the Zuul
job runs.
Thanks in advance for your support!
Thanks & Regards,
Renu Rani(レヌラニ)
Mobile: +91-9311888962
E-mail: renu.rani@india.nec.com <mailto:renu.rani@india.nec.com>
The contents of this e-mail and any attachment(s) are confidential and
intended for the named recipient(s) only.
It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the
If you have received this email in error please delete it and notify
across all nodes. prior written consent of the author of this e-mail is strictly prohibited. the sender immediately.
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
-- Thanks and Regards Yatin Karel
Dear Yatin, Thanks for your input. Regards Renu From: Yatin Karel <ykarel@redhat.com> Sent: Tuesday, July 22, 2025 11:43 AM To: Renu Rani <renu.rani@india.nec.com> Cc: Takashi Kajinami <kajinamit@oss.nttdata.com>; Brian Haley <haleyb.dev@gmail.com>; OpenStack Discuss <openstack-discuss@lists.openstack.org>; Shivam Shukla <shivam.shukla3@india.nec.com>; Pankaj Kumar Pandey <pankaj.pandey@india.nec.com> Subject: Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch You don't often get email from ykarel@redhat.com<mailto:ykarel@redhat.com>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> External Message: Please be cautious when opening links or attachments in email Hi, On Mon, Jul 21, 2025 at 8:22 PM Renu Rani <renu.rani@india.nec.com<mailto:renu.rani@india.nec.com>> wrote: Dear Takashi San, Thanks for your support. Please fine below further investigation updates from Tacker: 1. Kubernetes-Based Job – tacker-ft-v2-k8s Error: "AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'?" Root Cause: The issue is caused by the default setuptools version (59.6.0) included in the base OS image (Ubuntu Jammy). Since Global_Venv=False is set in local.conf, DevStack does not use a virtual environment, leading to the use of the outdated setuptools version. The required attribute is only available from setuptools version 64 onward. Fix: Set Global_Venv=True in local.conf. This enables the DevStack virtual environment where setuptools version 80 is installed, resolving the issue by using the correct setuptools version. 2.OpenStack-Based Job – tacker-ft-v1-compliance-sol Error#1: Internal Server Error during execution of the command: "openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a8" Root Cause: Below devstack commit started causing this issue. DevStack Commit (leads to “Internal Server Error”) – May 20, 2025 --> Seems Devstack code update related to neutron. [Commit sha: 988ce885d366dd0a7f8a012f9e52ee8f3403ffee] Error#2: [ERROR] Neutron did not start Root Cause: Below Neutron commit started causing this issue: Neutron Commit (leads to “Neutron did not start”) – July 1, 2025 [Commit SHA: 1971c4a261ca54f077e2dec8b59774602bbbe3f5]] For this I suggested some changes in the test patch https://review.opendev.org/c/openstack/tacker/+/952892, hopefully that will clear it. Regards Renu -----Original Message----- From: Renu Rani <renu.rani@india.nec.com<mailto:renu.rani@india.nec.com>> Sent: Thursday, July 17, 2025 2:56 PM To: Takashi Kajinami <kajinamit@oss.nttdata.com<mailto:kajinamit@oss.nttdata.com>>; Brian Haley <haleyb.dev@gmail.com<mailto:haleyb.dev@gmail.com>>; OpenStack Discuss <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> Cc: Shivam Shukla <shivam.shukla3@india.nec.com<mailto:shivam.shukla3@india.nec.com>>; Pankaj Kumar Pandey <pankaj.pandey@india.nec.com<mailto:pankaj.pandey@india.nec.com>> Subject: RE: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch Dear Takashi San, Thanks a lot for your response and really appreciate sharing your inputs. Just to update there are two types of Tacker jobs is check pipeline: one is Kubernetes-based, and the other is OpenStack-based. [1] Kubernetes-based Job: tacker-ft-v2-k8s It appears that you reviewed the logs for this job, which is currently failing due to the following error: "AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'?" We understand that this error is not related to Neutron changes, but rather due to an incompatibility between pip and setuptools, likely caused by recent packaging updates. We're actively investigating the root cause of this issue. [2] OpenStack-based Job: tacker-ft-v1-compliance-sol Our original request was regarding this job. It fails due to an "Internal Server Error" during the execution of the following command: "openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a8" Zuul Log (controller node): https://zuul.opendev.org/t/openstack/build/970baccf521846ec83441c1f18898199 To debug further, we tried checking out the stable/2025.1 branch of DevStack instead of master. However, the controller node now fails with this error: "[ERROR] /opt/stack/devstack/functions-common:2366 Neutron did not start" Zuul Log (controller node): https://zuul.opendev.org/t/openstack/build/09d39537443640c2a87673590f577829 Identified the specific commits that introduced these issues: DevStack Commit (leads to “Internal Server Error”) – May 20, 2025 [Commit sha: 988ce885d366dd0a7f8a012f9e52ee8f3403ffee] [https://github.com/openstack/devstack/commit/988ce885d366dd0a7f8a012f9e52ee8...] Neutron Commit (leads to “Neutron did not start”) – July 1, 2025 [Commit SHA: 1971c4a261ca54f077e2dec8b59774602bbbe3f5]] [https://github.com/openstack/neutron/commit/1971c4a261ca54f077e2dec8b5977460...]] Regards Renu -----Original Message----- From: Takashi Kajinami <kajinamit@oss.nttdata.com<mailto:kajinamit@oss.nttdata.com>> Sent: Wednesday, July 16, 2025 12:42 PM To: Brian Haley <haleyb.dev@gmail.com<mailto:haleyb.dev@gmail.com>>; Renu Rani <renu.rani@india.nec.com<mailto:renu.rani@india.nec.com>>; OpenStack Discuss <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> Cc: Shivam Shukla <shivam.shukla3@india.nec.com<mailto:shivam.shukla3@india.nec.com>>; Pankaj Kumar Pandey <pankaj.pandey@india.nec.com<mailto:pankaj.pandey@india.nec.com>> Subject: Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch [You don't often get email from kajinamit@oss.nttdata.com<mailto:kajinamit@oss.nttdata.com>. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] External Message: Please be cautious when opening links or attachments in email Looking at the build result[1] in the DNM patch mentioned, I found that pip failed to install neutron from repo due to the following error. [1] https://zuul.opendev.org/t/openstack/build/ef1dbefc5dab40c9a6e13cb4badaa3b2 AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'? I think this indicates the issue is caused by incompatibility between pip and setuptools which is caused by recent multiple updates in packaging. One thing I noticed is that the job still runs in Ubuntu Jammy, although Jammy was removed from supported operating system in this cycle and should have been replaced by Ubuntu Noble. Probably you have to first try migrating these jobs to noble and see if that can resolve the issue. On 7/16/25 1:20 AM, Brian Haley wrote:
Hi Renu,
On 7/15/25 3:16 AM, Renu Rani wrote:
Hi Neutron Team,
We are currently facing an issue in Tacker Zuul patch jobs build (with master branch), where the DevStack setup fails during the initial Neutron network creation step.
*Problem statement:*
The failure occurs at this point in the stack.sh script, for devstack environment creation in build process.
Below command execution is failing in stack.sh
*[openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc d777d04b-93ff-4e94-b6a9-cc8de3e23921]*
I looked quickly at the log files and it looks like you are also using the networking-sfc code. We just fixed an issue with eventlet there which could cause a bad interaction with neutron:
https://revi/ ew.opendev.org<http://ew.opendev.org/>%2Fc%2Fopenstack%2Fnetworking-sfc%2F%2B%2F954633&data=05 %7C02%7Crenu.rani%40india.nec.com<http://40india.nec.com/>%7C5fbac6b252c5459b0e4608ddc438073c%7 Ccc4532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168427284%7CUnkno wn%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXa W4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=olPBCnubJho UYWt%2FCi6WoUJA2dzpiHKljXCxWs4lmG8%3D&reserved=0
I'm not sure if it's related but that change just merged an hour ago, so it might be worth re-running things to check.
This was the version of networking-sfc I see being built:
Successfully installed networking-sfc-20.1.0.dev3
And it was pulling-in eventlet (which was just removed in that patch):
Requirement already satisfied: eventlet>=0.27.0 in /opt/stack/data/venv/lib/python3.12/site-packages (from networking-sfc==20.1.0.dev3) (0.40.1)
If that doesn't help please file a bug against neutron so someone can look further.
Thanks,
-Brian
**
*Error Message:*
/This results in a 500 Internal Server Error, as seen in the logs:/
/[HttpException: 500: Server Error for url: http://10.x/ .x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7cb f5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com<http://40india.nec.com/>% 7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110c %7C0%7C0%7C638882467168459279%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGk iOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoy fQ%3D%3D%7C0%7C%7C%7C&sdata=qn%2FaQZX5%2FaGkPyInL8RKDpr1WhliJE%2BpJ90 HANXME%2FQ%3D&reserved=0 <http://0.0.0.10/. x.x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7c bf5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com<http://40india.nec.com/> %7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110 c%7C0%7C0%7C638882467168484477%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcG kiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjo yfQ%3D%3D%7C0%7C%7C%7C&sdata=HhIBD9MAnWFoMCBtXX5FY3k2tfvlKiBmMKDiTFuy iNQ%3D&reserved=0>]/
*Analysis for the issue:*
Below error is coming in Neutron API logs:
*[ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on un-acquired lock]***
1. Tried overriding the required-projects in Zuul to use stable/2025.1 branches (instead of master) to identify project(master branch) which is causing root cause. 2. Tacker team has created DNM patch (952892) to invetsigate this in detail.
https://rev/ iew.opendev.org<http://iew.opendev.org/>%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02% 7Crenu.rani%40india.nec.com<http://40india.nec.com/>%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc45 32b813fc4545981297eef82e110c%7C0%7C0%7C638882467168506073%7CUnknown%7 CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4z MiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=anYUb6qkQlI%2 Fe1FSOcEtkdsr0iuv3OnNS96Dh2Zl4eY%3D&reserved=0 <https://re/ view.opendev.org<http://view.opendev.org/>%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02 %7Crenu.rani%40india.nec.com<http://40india.nec.com/>%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4 532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168521702%7CUnknown% 7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4 zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=0J1axj9cioqQ BuKReXSeCkA8rWpR2kPJ1YWmjQhG4DU%3D&reserved=0>
3. In DNM patch Patchset 11 (on June 29), we have taken devstack stable/2025.1. In stable branch the devstack environment created sucessfully. 4. In DNM patch Patchset 25 (on July 8), with the same devstack override (stable/2025.1), the job now fails earlier during DevStack setup.
The logs show that Neutron failed to start, causing the job to fail across all nodes.
As per observations and anlysis seems the recent changes in the Neutron master branch between June 29 and July 8 might be causing this issue.
We’d appreciate your help in reviewing the changes in Neutron's master branch during that window that could cause this behavior and suggesting any debugging approaches or patches we can test on tacker.
Please let us know if you'd like additional logs or info from the Zuul job runs.
Thanks in advance for your support!
Thanks & Regards,
Renu Rani(レヌラニ)
Mobile: +91-9311888962
E-mail: renu.rani@india.nec.com<mailto:renu.rani@india.nec.com> <mailto:renu.rani@india.nec.com<mailto:renu.rani@india.nec.com>>
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. -- Thanks and Regards Yatin Karel The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Hi Yatin, Thank you for your input regarding the shared issues. JFYI... I tried your suggested recommendation i.e., remove "logger" from Q_ML2_PLUGIN_MECHANISM_DRIVERS in Tacker base job configuration. It resolves the issues ("Internal Server Error" and "Neutron did not start") related to Neutron. Please refer this build result: https://zuul.opendev.org/t/openstack/build/48d742463f02457c8a99dd0fed0db1bb Also, I would also appreciate it if you could share some details about the root cause of the issue and explain how removing the 'logger' resolves it." Thank you once again for your inputs. Regards, Shivam ________________________________ From: Yatin Karel <ykarel@redhat.com> Sent: Monday, July 21, 2025 11:13 PM To: Renu Rani <renu.rani@india.nec.com> Cc: Takashi Kajinami <kajinamit@oss.nttdata.com>; Brian Haley <haleyb.dev@gmail.com>; OpenStack Discuss <openstack-discuss@lists.openstack.org>; Shivam Shukla <shivam.shukla3@india.nec.com>; Pankaj Kumar Pandey <pankaj.pandey@india.nec.com> Subject: Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch You don't often get email from ykarel@redhat.com. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> External Message: Please be cautious when opening links or attachments in email Hi, On Mon, Jul 21, 2025 at 8:22 PM Renu Rani <renu.rani@india.nec.com<mailto:renu.rani@india.nec.com>> wrote: Dear Takashi San, Thanks for your support. Please fine below further investigation updates from Tacker: 1. Kubernetes-Based Job – tacker-ft-v2-k8s Error: "AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'?" Root Cause: The issue is caused by the default setuptools version (59.6.0) included in the base OS image (Ubuntu Jammy). Since Global_Venv=False is set in local.conf, DevStack does not use a virtual environment, leading to the use of the outdated setuptools version. The required attribute is only available from setuptools version 64 onward. Fix: Set Global_Venv=True in local.conf. This enables the DevStack virtual environment where setuptools version 80 is installed, resolving the issue by using the correct setuptools version. 2.OpenStack-Based Job – tacker-ft-v1-compliance-sol Error#1: Internal Server Error during execution of the command: "openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a8" Root Cause: Below devstack commit started causing this issue. DevStack Commit (leads to “Internal Server Error”) – May 20, 2025 --> Seems Devstack code update related to neutron. [Commit sha: 988ce885d366dd0a7f8a012f9e52ee8f3403ffee] Error#2: [ERROR] Neutron did not start Root Cause: Below Neutron commit started causing this issue: Neutron Commit (leads to “Neutron did not start”) – July 1, 2025 [Commit SHA: 1971c4a261ca54f077e2dec8b59774602bbbe3f5]] For this I suggested some changes in the test patch https://review.opendev.org/c/openstack/tacker/+/952892, hopefully that will clear it. Regards Renu -----Original Message----- From: Renu Rani <renu.rani@india.nec.com<mailto:renu.rani@india.nec.com>> Sent: Thursday, July 17, 2025 2:56 PM To: Takashi Kajinami <kajinamit@oss.nttdata.com<mailto:kajinamit@oss.nttdata.com>>; Brian Haley <haleyb.dev@gmail.com<mailto:haleyb.dev@gmail.com>>; OpenStack Discuss <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> Cc: Shivam Shukla <shivam.shukla3@india.nec.com<mailto:shivam.shukla3@india.nec.com>>; Pankaj Kumar Pandey <pankaj.pandey@india.nec.com<mailto:pankaj.pandey@india.nec.com>> Subject: RE: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch Dear Takashi San, Thanks a lot for your response and really appreciate sharing your inputs. Just to update there are two types of Tacker jobs is check pipeline: one is Kubernetes-based, and the other is OpenStack-based. [1] Kubernetes-based Job: tacker-ft-v2-k8s It appears that you reviewed the logs for this job, which is currently failing due to the following error: "AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'?" We understand that this error is not related to Neutron changes, but rather due to an incompatibility between pip and setuptools, likely caused by recent packaging updates. We're actively investigating the root cause of this issue. [2] OpenStack-based Job: tacker-ft-v1-compliance-sol Our original request was regarding this job. It fails due to an "Internal Server Error" during the execution of the following command: "openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a8" Zuul Log (controller node): https://zuul.opendev.org/t/openstack/build/970baccf521846ec83441c1f18898199 To debug further, we tried checking out the stable/2025.1 branch of DevStack instead of master. However, the controller node now fails with this error: "[ERROR] /opt/stack/devstack/functions-common:2366 Neutron did not start" Zuul Log (controller node): https://zuul.opendev.org/t/openstack/build/09d39537443640c2a87673590f577829 Identified the specific commits that introduced these issues: DevStack Commit (leads to “Internal Server Error”) – May 20, 2025 [Commit sha: 988ce885d366dd0a7f8a012f9e52ee8f3403ffee] [https://github.com/openstack/devstack/commit/988ce885d366dd0a7f8a012f9e52ee8...] Neutron Commit (leads to “Neutron did not start”) – July 1, 2025 [Commit SHA: 1971c4a261ca54f077e2dec8b59774602bbbe3f5]] [https://github.com/openstack/neutron/commit/1971c4a261ca54f077e2dec8b5977460...]] Regards Renu -----Original Message----- From: Takashi Kajinami <kajinamit@oss.nttdata.com<mailto:kajinamit@oss.nttdata.com>> Sent: Wednesday, July 16, 2025 12:42 PM To: Brian Haley <haleyb.dev@gmail.com<mailto:haleyb.dev@gmail.com>>; Renu Rani <renu.rani@india.nec.com<mailto:renu.rani@india.nec.com>>; OpenStack Discuss <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> Cc: Shivam Shukla <shivam.shukla3@india.nec.com<mailto:shivam.shukla3@india.nec.com>>; Pankaj Kumar Pandey <pankaj.pandey@india.nec.com<mailto:pankaj.pandey@india.nec.com>> Subject: Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch [You don't often get email from kajinamit@oss.nttdata.com<mailto:kajinamit@oss.nttdata.com>. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] External Message: Please be cautious when opening links or attachments in email Looking at the build result[1] in the DNM patch mentioned, I found that pip failed to install neutron from repo due to the following error. [1] https://zuul.opendev.org/t/openstack/build/ef1dbefc5dab40c9a6e13cb4badaa3b2 AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'? I think this indicates the issue is caused by incompatibility between pip and setuptools which is caused by recent multiple updates in packaging. One thing I noticed is that the job still runs in Ubuntu Jammy, although Jammy was removed from supported operating system in this cycle and should have been replaced by Ubuntu Noble. Probably you have to first try migrating these jobs to noble and see if that can resolve the issue. On 7/16/25 1:20 AM, Brian Haley wrote:
Hi Renu,
On 7/15/25 3:16 AM, Renu Rani wrote:
Hi Neutron Team,
We are currently facing an issue in Tacker Zuul patch jobs build (with master branch), where the DevStack setup fails during the initial Neutron network creation step.
*Problem statement:*
The failure occurs at this point in the stack.sh script, for devstack environment creation in build process.
Below command execution is failing in stack.sh
*[openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc d777d04b-93ff-4e94-b6a9-cc8de3e23921]*
I looked quickly at the log files and it looks like you are also using the networking-sfc code. We just fixed an issue with eventlet there which could cause a bad interaction with neutron:
https://revi/ ew.opendev.org<http://ew.opendev.org/>%2Fc%2Fopenstack%2Fnetworking-sfc%2F%2B%2F954633&data=05 %7C02%7Crenu.rani%40india.nec.com<http://40india.nec.com/>%7C5fbac6b252c5459b0e4608ddc438073c%7 Ccc4532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168427284%7CUnkno wn%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXa W4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=olPBCnubJho UYWt%2FCi6WoUJA2dzpiHKljXCxWs4lmG8%3D&reserved=0
I'm not sure if it's related but that change just merged an hour ago, so it might be worth re-running things to check.
This was the version of networking-sfc I see being built:
Successfully installed networking-sfc-20.1.0.dev3
And it was pulling-in eventlet (which was just removed in that patch):
Requirement already satisfied: eventlet>=0.27.0 in /opt/stack/data/venv/lib/python3.12/site-packages (from networking-sfc==20.1.0.dev3) (0.40.1)
If that doesn't help please file a bug against neutron so someone can look further.
Thanks,
-Brian
**
*Error Message:*
/This results in a 500 Internal Server Error, as seen in the logs:/
/[HttpException: 500: Server Error for url: http://10.x/ .x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7cb f5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com<http://40india.nec.com/>% 7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110c %7C0%7C0%7C638882467168459279%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGk iOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoy fQ%3D%3D%7C0%7C%7C%7C&sdata=qn%2FaQZX5%2FaGkPyInL8RKDpr1WhliJE%2BpJ90 HANXME%2FQ%3D&reserved=0 <http://0.0.0.10/. x.x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7c bf5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com<http://40india.nec.com/> %7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110 c%7C0%7C0%7C638882467168484477%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcG kiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjo yfQ%3D%3D%7C0%7C%7C%7C&sdata=HhIBD9MAnWFoMCBtXX5FY3k2tfvlKiBmMKDiTFuy iNQ%3D&reserved=0>]/
*Analysis for the issue:*
Below error is coming in Neutron API logs:
*[ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on un-acquired lock]***
1. Tried overriding the required-projects in Zuul to use stable/2025.1 branches (instead of master) to identify project(master branch) which is causing root cause. 2. Tacker team has created DNM patch (952892) to invetsigate this in detail.
https://rev/ iew.opendev.org<http://iew.opendev.org/>%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02% 7Crenu.rani%40india.nec.com<http://40india.nec.com/>%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc45 32b813fc4545981297eef82e110c%7C0%7C0%7C638882467168506073%7CUnknown%7 CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4z MiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=anYUb6qkQlI%2 Fe1FSOcEtkdsr0iuv3OnNS96Dh2Zl4eY%3D&reserved=0 <https://re/ view.opendev.org<http://view.opendev.org/>%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02 %7Crenu.rani%40india.nec.com<http://40india.nec.com/>%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4 532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168521702%7CUnknown% 7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4 zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=0J1axj9cioqQ BuKReXSeCkA8rWpR2kPJ1YWmjQhG4DU%3D&reserved=0>
3. In DNM patch Patchset 11 (on June 29), we have taken devstack stable/2025.1. In stable branch the devstack environment created sucessfully. 4. In DNM patch Patchset 25 (on July 8), with the same devstack override (stable/2025.1), the job now fails earlier during DevStack setup.
The logs show that Neutron failed to start, causing the job to fail across all nodes.
As per observations and anlysis seems the recent changes in the Neutron master branch between June 29 and July 8 might be causing this issue.
We’d appreciate your help in reviewing the changes in Neutron's master branch during that window that could cause this behavior and suggesting any debugging approaches or patches we can test on tacker.
Please let us know if you'd like additional logs or info from the Zuul job runs.
Thanks in advance for your support!
Thanks & Regards,
Renu Rani(レヌラニ)
Mobile: +91-9311888962
E-mail: renu.rani@india.nec.com<mailto:renu.rani@india.nec.com> <mailto:renu.rani@india.nec.com<mailto:renu.rani@india.nec.com>>
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. -- Thanks and Regards Yatin Karel The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Hi Yatin, Hope you're doing well. First of all, thank you once again for your support and valuable input on the Neutron-related issue we were facing in the Tacker base job. Your suggestion to remove 'logger' from the Q_ML2_PLUGIN_MECHANISM_DRIVERS configuration resolved the "Internal Server Error" and Neutron startup problems, as confirmed in the following build: https://zuul.opendev.org/t/openstack/build/48d742463f02457c8a99dd0fed0db1bb Just as a gentle reminder — when you get a chance, it would be really helpful if you could share more details about the root cause of this issue. Specifically, we’d like to understand why the presence of 'logger' causes the failure, how removing it resolves the problem, and whether any functional or behavioral changes are expected as a result after its (logger) removal. This explanation will help us document the reasoning clearly in the patch and provide proper justification for the change. Looking forward to your insights. Thank You. Regards, Shivam ________________________________ From: Shivam Shukla <shivam.shukla3@india.nec.com> Sent: Sunday, July 27, 2025 10:35 PM To: Yatin Karel <ykarel@redhat.com>; Renu Rani <renu.rani@india.nec.com> Cc: Takashi Kajinami <kajinamit@oss.nttdata.com>; Brian Haley <haleyb.dev@gmail.com>; OpenStack Discuss <openstack-discuss@lists.openstack.org>; Pankaj Kumar Pandey <pankaj.pandey@india.nec.com> Subject: Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch Hi Yatin, Thank you for your input regarding the shared issues. JFYI... I tried your suggested recommendation i.e., remove "logger" from Q_ML2_PLUGIN_MECHANISM_DRIVERS in Tacker base job configuration. It resolves the issues ("Internal Server Error" and "Neutron did not start") related to Neutron. Please refer this build result: https://zuul.opendev.org/t/openstack/build/48d742463f02457c8a99dd0fed0db1bb Also, I would also appreciate it if you could share some details about the root cause of the issue and explain how removing the 'logger' resolves it." Thank you once again for your inputs. Regards, Shivam ________________________________ From: Yatin Karel <ykarel@redhat.com> Sent: Monday, July 21, 2025 11:13 PM To: Renu Rani <renu.rani@india.nec.com> Cc: Takashi Kajinami <kajinamit@oss.nttdata.com>; Brian Haley <haleyb.dev@gmail.com>; OpenStack Discuss <openstack-discuss@lists.openstack.org>; Shivam Shukla <shivam.shukla3@india.nec.com>; Pankaj Kumar Pandey <pankaj.pandey@india.nec.com> Subject: Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch You don't often get email from ykarel@redhat.com. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> External Message: Please be cautious when opening links or attachments in email Hi, On Mon, Jul 21, 2025 at 8:22 PM Renu Rani <renu.rani@india.nec.com<mailto:renu.rani@india.nec.com>> wrote: Dear Takashi San, Thanks for your support. Please fine below further investigation updates from Tacker: 1. Kubernetes-Based Job – tacker-ft-v2-k8s Error: "AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'?" Root Cause: The issue is caused by the default setuptools version (59.6.0) included in the base OS image (Ubuntu Jammy). Since Global_Venv=False is set in local.conf, DevStack does not use a virtual environment, leading to the use of the outdated setuptools version. The required attribute is only available from setuptools version 64 onward. Fix: Set Global_Venv=True in local.conf. This enables the DevStack virtual environment where setuptools version 80 is installed, resolving the issue by using the correct setuptools version. 2.OpenStack-Based Job – tacker-ft-v1-compliance-sol Error#1: Internal Server Error during execution of the command: "openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a8" Root Cause: Below devstack commit started causing this issue. DevStack Commit (leads to “Internal Server Error”) – May 20, 2025 --> Seems Devstack code update related to neutron. [Commit sha: 988ce885d366dd0a7f8a012f9e52ee8f3403ffee] Error#2: [ERROR] Neutron did not start Root Cause: Below Neutron commit started causing this issue: Neutron Commit (leads to “Neutron did not start”) – July 1, 2025 [Commit SHA: 1971c4a261ca54f077e2dec8b59774602bbbe3f5]] For this I suggested some changes in the test patch https://review.opendev.org/c/openstack/tacker/+/952892, hopefully that will clear it. Regards Renu -----Original Message----- From: Renu Rani <renu.rani@india.nec.com<mailto:renu.rani@india.nec.com>> Sent: Thursday, July 17, 2025 2:56 PM To: Takashi Kajinami <kajinamit@oss.nttdata.com<mailto:kajinamit@oss.nttdata.com>>; Brian Haley <haleyb.dev@gmail.com<mailto:haleyb.dev@gmail.com>>; OpenStack Discuss <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> Cc: Shivam Shukla <shivam.shukla3@india.nec.com<mailto:shivam.shukla3@india.nec.com>>; Pankaj Kumar Pandey <pankaj.pandey@india.nec.com<mailto:pankaj.pandey@india.nec.com>> Subject: RE: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch Dear Takashi San, Thanks a lot for your response and really appreciate sharing your inputs. Just to update there are two types of Tacker jobs is check pipeline: one is Kubernetes-based, and the other is OpenStack-based. [1] Kubernetes-based Job: tacker-ft-v2-k8s It appears that you reviewed the logs for this job, which is currently failing due to the following error: "AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'?" We understand that this error is not related to Neutron changes, but rather due to an incompatibility between pip and setuptools, likely caused by recent packaging updates. We're actively investigating the root cause of this issue. [2] OpenStack-based Job: tacker-ft-v1-compliance-sol Our original request was regarding this job. It fails due to an "Internal Server Error" during the execution of the following command: "openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a8" Zuul Log (controller node): https://zuul.opendev.org/t/openstack/build/970baccf521846ec83441c1f18898199 To debug further, we tried checking out the stable/2025.1 branch of DevStack instead of master. However, the controller node now fails with this error: "[ERROR] /opt/stack/devstack/functions-common:2366 Neutron did not start" Zuul Log (controller node): https://zuul.opendev.org/t/openstack/build/09d39537443640c2a87673590f577829 Identified the specific commits that introduced these issues: DevStack Commit (leads to “Internal Server Error”) – May 20, 2025 [Commit sha: 988ce885d366dd0a7f8a012f9e52ee8f3403ffee] [https://github.com/openstack/devstack/commit/988ce885d366dd0a7f8a012f9e52ee8...] Neutron Commit (leads to “Neutron did not start”) – July 1, 2025 [Commit SHA: 1971c4a261ca54f077e2dec8b59774602bbbe3f5]] [https://github.com/openstack/neutron/commit/1971c4a261ca54f077e2dec8b5977460...]] Regards Renu -----Original Message----- From: Takashi Kajinami <kajinamit@oss.nttdata.com<mailto:kajinamit@oss.nttdata.com>> Sent: Wednesday, July 16, 2025 12:42 PM To: Brian Haley <haleyb.dev@gmail.com<mailto:haleyb.dev@gmail.com>>; Renu Rani <renu.rani@india.nec.com<mailto:renu.rani@india.nec.com>>; OpenStack Discuss <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> Cc: Shivam Shukla <shivam.shukla3@india.nec.com<mailto:shivam.shukla3@india.nec.com>>; Pankaj Kumar Pandey <pankaj.pandey@india.nec.com<mailto:pankaj.pandey@india.nec.com>> Subject: Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch [You don't often get email from kajinamit@oss.nttdata.com<mailto:kajinamit@oss.nttdata.com>. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] External Message: Please be cautious when opening links or attachments in email Looking at the build result[1] in the DNM patch mentioned, I found that pip failed to install neutron from repo due to the following error. [1] https://zuul.opendev.org/t/openstack/build/ef1dbefc5dab40c9a6e13cb4badaa3b2 AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'? I think this indicates the issue is caused by incompatibility between pip and setuptools which is caused by recent multiple updates in packaging. One thing I noticed is that the job still runs in Ubuntu Jammy, although Jammy was removed from supported operating system in this cycle and should have been replaced by Ubuntu Noble. Probably you have to first try migrating these jobs to noble and see if that can resolve the issue. On 7/16/25 1:20 AM, Brian Haley wrote:
Hi Renu,
On 7/15/25 3:16 AM, Renu Rani wrote:
Hi Neutron Team,
We are currently facing an issue in Tacker Zuul patch jobs build (with master branch), where the DevStack setup fails during the initial Neutron network creation step.
*Problem statement:*
The failure occurs at this point in the stack.sh script, for devstack environment creation in build process.
Below command execution is failing in stack.sh
*[openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc d777d04b-93ff-4e94-b6a9-cc8de3e23921]*
I looked quickly at the log files and it looks like you are also using the networking-sfc code. We just fixed an issue with eventlet there which could cause a bad interaction with neutron:
https://revi/ ew.opendev.org<http://ew.opendev.org/>%2Fc%2Fopenstack%2Fnetworking-sfc%2F%2B%2F954633&data=05 %7C02%7Crenu.rani%40india.nec.com<http://40india.nec.com/>%7C5fbac6b252c5459b0e4608ddc438073c%7 Ccc4532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168427284%7CUnkno wn%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXa W4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=olPBCnubJho UYWt%2FCi6WoUJA2dzpiHKljXCxWs4lmG8%3D&reserved=0
I'm not sure if it's related but that change just merged an hour ago, so it might be worth re-running things to check.
This was the version of networking-sfc I see being built:
Successfully installed networking-sfc-20.1.0.dev3
And it was pulling-in eventlet (which was just removed in that patch):
Requirement already satisfied: eventlet>=0.27.0 in /opt/stack/data/venv/lib/python3.12/site-packages (from networking-sfc==20.1.0.dev3) (0.40.1)
If that doesn't help please file a bug against neutron so someone can look further.
Thanks,
-Brian
**
*Error Message:*
/This results in a 500 Internal Server Error, as seen in the logs:/
/[HttpException: 500: Server Error for url: http://10.x/ .x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7cb f5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com<http://40india.nec.com/>% 7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110c %7C0%7C0%7C638882467168459279%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGk iOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoy fQ%3D%3D%7C0%7C%7C%7C&sdata=qn%2FaQZX5%2FaGkPyInL8RKDpr1WhliJE%2BpJ90 HANXME%2FQ%3D&reserved=0 <http://0.0.0.10/. x.x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7c bf5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com<http://40india.nec.com/> %7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110 c%7C0%7C0%7C638882467168484477%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcG kiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjo yfQ%3D%3D%7C0%7C%7C%7C&sdata=HhIBD9MAnWFoMCBtXX5FY3k2tfvlKiBmMKDiTFuy iNQ%3D&reserved=0>]/
*Analysis for the issue:*
Below error is coming in Neutron API logs:
*[ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on un-acquired lock]***
1. Tried overriding the required-projects in Zuul to use stable/2025.1 branches (instead of master) to identify project(master branch) which is causing root cause. 2. Tacker team has created DNM patch (952892) to invetsigate this in detail.
https://rev/ iew.opendev.org<http://iew.opendev.org/>%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02% 7Crenu.rani%40india.nec.com<http://40india.nec.com/>%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc45 32b813fc4545981297eef82e110c%7C0%7C0%7C638882467168506073%7CUnknown%7 CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4z MiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=anYUb6qkQlI%2 Fe1FSOcEtkdsr0iuv3OnNS96Dh2Zl4eY%3D&reserved=0 <https://re/ view.opendev.org<http://view.opendev.org/>%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02 %7Crenu.rani%40india.nec.com<http://40india.nec.com/>%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4 532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168521702%7CUnknown% 7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4 zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=0J1axj9cioqQ BuKReXSeCkA8rWpR2kPJ1YWmjQhG4DU%3D&reserved=0>
3. In DNM patch Patchset 11 (on June 29), we have taken devstack stable/2025.1. In stable branch the devstack environment created sucessfully. 4. In DNM patch Patchset 25 (on July 8), with the same devstack override (stable/2025.1), the job now fails earlier during DevStack setup.
The logs show that Neutron failed to start, causing the job to fail across all nodes.
As per observations and anlysis seems the recent changes in the Neutron master branch between June 29 and July 8 might be causing this issue.
We’d appreciate your help in reviewing the changes in Neutron's master branch during that window that could cause this behavior and suggesting any debugging approaches or patches we can test on tacker.
Please let us know if you'd like additional logs or info from the Zuul job runs.
Thanks in advance for your support!
Thanks & Regards,
Renu Rani(レヌラニ)
Mobile: +91-9311888962
E-mail: renu.rani@india.nec.com<mailto:renu.rani@india.nec.com> <mailto:renu.rani@india.nec.com<mailto:renu.rani@india.nec.com>>
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. -- Thanks and Regards Yatin Karel The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Hi Yatin,
Hope you're doing well.
First of all, thank you once again for your support and valuable input on the Neutron-related issue we were facing in the Tacker base job. Your suggestion to remove 'logger' from the Q_ML2_PLUGIN_MECHANISM_DRIVERS configuration resolved the "Internal Server Error" and Neutron startup problems, as confirmed in the following build: https://zuul.opendev.org/t/openstack/build/48d742463f02457c8a99dd0fed0db1bb
Just as a gentle reminder — when you get a chance, it would be really helpful if you could share more details about the root cause of this issue. Specifically, we’d like to understand why the presence of 'logger' causes the failure, how removing it resolves the problem, and whether any functional or behavioral changes are expected as a result after its (logger) removal.
It's mainly related to eventlet removal[1], there you can see a couple of
Hi Renu and Shivam, On Thu, Jul 31, 2025 at 9:20 AM Shivam Shukla <shivam.shukla3@india.nec.com> wrote: patches related to dropping "logger" mechanism drivers. [1] https://review.opendev.org/q/topic:%22eventlet-removal%22+project:openstack/...
This explanation will help us document the reasoning clearly in the patch and provide proper justification for the change.
Looking forward to your insights.
Thank You.
Regards, Shivam ------------------------------ *From:* Shivam Shukla <shivam.shukla3@india.nec.com> *Sent:* Sunday, July 27, 2025 10:35 PM *To:* Yatin Karel <ykarel@redhat.com>; Renu Rani <renu.rani@india.nec.com> *Cc:* Takashi Kajinami <kajinamit@oss.nttdata.com>; Brian Haley < haleyb.dev@gmail.com>; OpenStack Discuss < openstack-discuss@lists.openstack.org>; Pankaj Kumar Pandey < pankaj.pandey@india.nec.com> *Subject:* Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch
Hi Yatin, Thank you for your input regarding the shared issues.
JFYI... I tried your suggested recommendation i.e., remove "logger" from Q_ML2_PLUGIN_MECHANISM_DRIVERS in Tacker base job configuration. It resolves the issues ("Internal Server Error" and "Neutron did not start") related to Neutron.
Please refer this build result: https://zuul.opendev.org/t/openstack/build/48d742463f02457c8a99dd0fed0db1bb
Also, I would also appreciate it if you could share some details about the root cause of the issue and explain how removing the 'logger' resolves it."
Thank you once again for your inputs.
Regards, Shivam ------------------------------ *From:* Yatin Karel <ykarel@redhat.com> *Sent:* Monday, July 21, 2025 11:13 PM *To:* Renu Rani <renu.rani@india.nec.com> *Cc:* Takashi Kajinami <kajinamit@oss.nttdata.com>; Brian Haley < haleyb.dev@gmail.com>; OpenStack Discuss < openstack-discuss@lists.openstack.org>; Shivam Shukla < shivam.shukla3@india.nec.com>; Pankaj Kumar Pandey < pankaj.pandey@india.nec.com> *Subject:* Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch
You don't often get email from ykarel@redhat.com. Learn why this is important <https://aka.ms/LearnAboutSenderIdentification> *External Message:* Please be cautious when opening links or attachments in email
Hi,
On Mon, Jul 21, 2025 at 8:22 PM Renu Rani <renu.rani@india.nec.com> wrote:
Dear Takashi San,
Thanks for your support.
Please fine below further investigation updates from Tacker:
1. Kubernetes-Based Job – tacker-ft-v2-k8s Error: "AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'?" Root Cause: The issue is caused by the default setuptools version (59.6.0) included in the base OS image (Ubuntu Jammy). Since Global_Venv=False is set in local.conf, DevStack does not use a virtual environment, leading to the use of the outdated setuptools version. The required attribute is only available from setuptools version 64 onward. Fix: Set Global_Venv=True in local.conf. This enables the DevStack virtual environment where setuptools version 80 is installed, resolving the issue by using the correct setuptools version.
2.OpenStack-Based Job – tacker-ft-v1-compliance-sol Error#1: Internal Server Error during execution of the command: "openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a8" Root Cause: Below devstack commit started causing this issue. DevStack Commit (leads to “Internal Server Error”) – May 20, 2025 --> Seems Devstack code update related to neutron. [Commit sha: 988ce885d366dd0a7f8a012f9e52ee8f3403ffee]
Error#2: [ERROR] Neutron did not start Root Cause: Below Neutron commit started causing this issue: Neutron Commit (leads to “Neutron did not start”) – July 1, 2025 [Commit SHA: 1971c4a261ca54f077e2dec8b59774602bbbe3f5]]
For this I suggested some changes in the test patch https://review.opendev.org/c/openstack/tacker/+/952892, hopefully that will clear it.
Regards Renu
-----Original Message----- From: Renu Rani <renu.rani@india.nec.com> Sent: Thursday, July 17, 2025 2:56 PM To: Takashi Kajinami <kajinamit@oss.nttdata.com>; Brian Haley < haleyb.dev@gmail.com>; OpenStack Discuss < openstack-discuss@lists.openstack.org> Cc: Shivam Shukla <shivam.shukla3@india.nec.com>; Pankaj Kumar Pandey < pankaj.pandey@india.nec.com> Subject: RE: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch
Dear Takashi San,
Thanks a lot for your response and really appreciate sharing your inputs.
Just to update there are two types of Tacker jobs is check pipeline: one is Kubernetes-based, and the other is OpenStack-based.
[1] Kubernetes-based Job: tacker-ft-v2-k8s It appears that you reviewed the logs for this job, which is currently failing due to the following error: "AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'?"
We understand that this error is not related to Neutron changes, but rather due to an incompatibility between pip and setuptools, likely caused by recent packaging updates. We're actively investigating the root cause of this issue.
[2] OpenStack-based Job: tacker-ft-v1-compliance-sol Our original request was regarding this job. It fails due to an "Internal Server Error" during the execution of the following command: "openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a8" Zuul Log (controller node): https://zuul.opendev.org/t/openstack/build/970baccf521846ec83441c1f18898199
To debug further, we tried checking out the stable/2025.1 branch of DevStack instead of master. However, the controller node now fails with this error: "[ERROR] /opt/stack/devstack/functions-common:2366 Neutron did not start" Zuul Log (controller node): https://zuul.opendev.org/t/openstack/build/09d39537443640c2a87673590f577829
Identified the specific commits that introduced these issues:
DevStack Commit (leads to “Internal Server Error”) – May 20, 2025 [Commit sha: 988ce885d366dd0a7f8a012f9e52ee8f3403ffee] [ https://github.com/openstack/devstack/commit/988ce885d366dd0a7f8a012f9e52ee8... ]
Neutron Commit (leads to “Neutron did not start”) – July 1, 2025 [Commit SHA: 1971c4a261ca54f077e2dec8b59774602bbbe3f5]] [ https://github.com/openstack/neutron/commit/1971c4a261ca54f077e2dec8b5977460... ]]
Regards Renu
-----Original Message----- From: Takashi Kajinami <kajinamit@oss.nttdata.com> Sent: Wednesday, July 16, 2025 12:42 PM To: Brian Haley <haleyb.dev@gmail.com>; Renu Rani <renu.rani@india.nec.com>; OpenStack Discuss <openstack-discuss@lists.openstack.org> Cc: Shivam Shukla <shivam.shukla3@india.nec.com>; Pankaj Kumar Pandey < pankaj.pandey@india.nec.com> Subject: Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch
[You don't often get email from kajinamit@oss.nttdata.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
External Message: Please be cautious when opening links or attachments in email
Looking at the build result[1] in the DNM patch mentioned, I found that pip failed to install neutron from repo due to the following error. [1] https://zuul.opendev.org/t/openstack/build/ef1dbefc5dab40c9a6e13cb4badaa3b2
AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'?
I think this indicates the issue is caused by incompatibility between pip and setuptools which is caused by recent multiple updates in packaging.
One thing I noticed is that the job still runs in Ubuntu Jammy, although Jammy was removed from supported operating system in this cycle and should have been replaced by Ubuntu Noble. Probably you have to first try migrating these jobs to noble and see if that can resolve the issue.
Hi Renu,
On 7/15/25 3:16 AM, Renu Rani wrote:
Hi Neutron Team,
We are currently facing an issue in Tacker Zuul patch jobs build (with master branch), where the DevStack setup fails during the initial Neutron network creation step.
*Problem statement:*
The failure occurs at this point in the stack.sh script, for devstack environment creation in build process.
Below command execution is failing in stack.sh
*[openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc d777d04b-93ff-4e94-b6a9-cc8de3e23921]*
I looked quickly at the log files and it looks like you are also using
On 7/16/25 1:20 AM, Brian Haley wrote: the networking-sfc code. We just fixed an issue with eventlet there which could cause a bad interaction with neutron:
https://revi/ ew.opendev.org%2Fc%2Fopenstack%2Fnetworking-sfc%2F%2B%2F954633&data=05 %7C02%7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7 Ccc4532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168427284%7CUnkno wn%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXa W4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=olPBCnubJho UYWt%2FCi6WoUJA2dzpiHKljXCxWs4lmG8%3D&reserved=0
I'm not sure if it's related but that change just merged an hour ago, so
it might be worth re-running things to check.
This was the version of networking-sfc I see being built:
Successfully installed networking-sfc-20.1.0.dev3
And it was pulling-in eventlet (which was just removed in that patch):
Requirement already satisfied: eventlet>=0.27.0 in /opt/stack/data/venv/lib/python3.12/site-packages (from networking-sfc==20.1.0.dev3) (0.40.1)
If that doesn't help please file a bug against neutron so someone can
look further.
Thanks,
-Brian
**
*Error Message:*
/This results in a 500 Internal Server Error, as seen in the logs:/
/[HttpException: 500: Server Error for url: http://10.x/ .x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7cb f5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com% 7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110c %7C0%7C0%7C638882467168459279%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGk iOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoy fQ%3D%3D%7C0%7C%7C%7C&sdata=qn%2FaQZX5%2FaGkPyInL8RKDpr1WhliJE%2BpJ90 HANXME%2FQ%3D&reserved=0 <http://0.0.0.10/. x.x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7c bf5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com %7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110 c%7C0%7C0%7C638882467168484477%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcG kiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjo yfQ%3D%3D%7C0%7C%7C%7C&sdata=HhIBD9MAnWFoMCBtXX5FY3k2tfvlKiBmMKDiTFuy iNQ%3D&reserved=0>]/
*Analysis for the issue:*
Below error is coming in Neutron API logs:
*[ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on un-acquired lock]***
1. Tried overriding the required-projects in Zuul to use stable/2025.1 branches (instead of master) to identify project(master branch) which is causing root cause. 2. Tacker team has created DNM patch (952892) to invetsigate this in detail.
https://rev/ iew.opendev.org%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02% 7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc45 32b813fc4545981297eef82e110c%7C0%7C0%7C638882467168506073%7CUnknown%7 CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4z MiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=anYUb6qkQlI%2 Fe1FSOcEtkdsr0iuv3OnNS96Dh2Zl4eY%3D&reserved=0 <https://re/ view.opendev.org%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02 %7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4 532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168521702%7CUnknown% 7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4 zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=0J1axj9cioqQ BuKReXSeCkA8rWpR2kPJ1YWmjQhG4DU%3D&reserved=0>
3. In DNM patch Patchset 11 (on June 29), we have taken devstack stable/2025.1. In stable branch the devstack environment created sucessfully. 4. In DNM patch Patchset 25 (on July 8), with the same devstack override (stable/2025.1), the job now fails earlier during DevStack setup.
The logs show that Neutron failed to start, causing the job to fail
As per observations and anlysis seems the recent changes in the Neutron
master branch between June 29 and July 8 might be causing this issue.
We’d appreciate your help in reviewing the changes in Neutron's master
branch during that window that could cause this behavior and suggesting any debugging approaches or patches we can test on tacker.
Please let us know if you'd like additional logs or info from the Zuul
job runs.
Thanks in advance for your support!
Thanks & Regards,
Renu Rani(レヌラニ)
Mobile: +91-9311888962
E-mail: renu.rani@india.nec.com <mailto:renu.rani@india.nec.com>
The contents of this e-mail and any attachment(s) are confidential and
intended for the named recipient(s) only.
It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the
If you have received this email in error please delete it and notify
across all nodes. prior written consent of the author of this e-mail is strictly prohibited. the sender immediately.
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
-- Thanks and Regards Yatin Karel
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
-- Thanks and Regards Yatin Karel
participants (5)
-
Brian Haley
-
Renu Rani
-
Shivam Shukla
-
Takashi Kajinami
-
Yatin Karel