This explanation will help us document the reasoning clearly in the patch and provide proper justification for the change.
Looking forward to your insights.
Thank You.
Regards,
Shivam
Hi Yatin,
Thank you for your input regarding the shared issues.
JFYI... I tried your suggested recommendation i.e., remove "logger"
from Q_ML2_PLUGIN_MECHANISM_DRIVERS in Tacker base job configuration. It resolves the issues ("Internal Server Error" and "Neutron did not start") related to Neutron.
Please refer this build result:
Also, I would also appreciate it if you could share some details about the root cause of the issue and explain how removing the 'logger' resolves it."
Thank you once again for your inputs.
Regards,
Shivam
|
External Message:
Please be cautious when opening links or attachments in email
|
Hi,
Dear Takashi San,
Thanks for your support.
Please fine below further investigation updates from Tacker:
1. Kubernetes-Based Job – tacker-ft-v2-k8s
Error: "AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'?"
Root Cause:
The issue is caused by the default setuptools version (59.6.0) included in the base OS image (Ubuntu Jammy). Since Global_Venv=False is set in local.conf, DevStack does
not use a virtual environment, leading to the use of the outdated setuptools version. The required attribute is only available from setuptools version 64 onward.
Fix:
Set Global_Venv=True in local.conf. This enables the DevStack virtual environment where setuptools version 80 is installed, resolving the issue by using the correct setuptools version.
2.OpenStack-Based Job – tacker-ft-v1-compliance-sol
Error#1:
Internal Server Error during execution of the command:
"openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a8"
Root Cause:
Below devstack commit started causing this issue.
DevStack Commit (leads to “Internal Server Error”) – May 20, 2025 --> Seems Devstack code update related to neutron.
[Commit sha: 988ce885d366dd0a7f8a012f9e52ee8f3403ffee]
Error#2:
[ERROR] Neutron did not start
Root Cause:
Below Neutron commit started causing this issue:
Neutron Commit (leads to “Neutron did not start”) – July 1, 2025
[Commit SHA: 1971c4a261ca54f077e2dec8b59774602bbbe3f5]]
Regards
Renu
-----Original Message-----
From: Renu Rani <renu.rani@india.nec.com>
Sent: Thursday, July 17, 2025 2:56 PM
To: Takashi Kajinami <kajinamit@oss.nttdata.com>; Brian Haley <haleyb.dev@gmail.com>; OpenStack Discuss <openstack-discuss@lists.openstack.org>
Cc: Shivam Shukla <shivam.shukla3@india.nec.com>; Pankaj Kumar Pandey <pankaj.pandey@india.nec.com>
Subject: RE: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch
Dear Takashi San,
Thanks a lot for your response and really appreciate sharing your inputs.
Just to update there are two types of Tacker jobs is check pipeline: one is Kubernetes-based, and the other is OpenStack-based.
[1] Kubernetes-based Job: tacker-ft-v2-k8s
It appears that you reviewed the logs for this job, which is currently failing due to the following error:
"AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'?"
We understand that this error is not related to Neutron changes, but rather due to an incompatibility between pip and setuptools, likely caused by recent packaging updates. We're actively investigating the root cause of this issue.
[2] OpenStack-based Job: tacker-ft-v1-compliance-sol
Our original request was regarding this job. It fails due to an "Internal Server Error" during the execution of the following command:
"openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a8"
Zuul Log (controller node):
https://zuul.opendev.org/t/openstack/build/970baccf521846ec83441c1f18898199
To debug further, we tried checking out the stable/2025.1 branch of DevStack instead of master. However, the controller node now fails with this error:
"[ERROR] /opt/stack/devstack/functions-common:2366 Neutron did not start"
Zuul Log (controller node):
https://zuul.opendev.org/t/openstack/build/09d39537443640c2a87673590f577829
Identified the specific commits that introduced these issues:
DevStack Commit (leads to “Internal Server Error”) – May 20, 2025 [Commit sha: 988ce885d366dd0a7f8a012f9e52ee8f3403ffee]
[https://github.com/openstack/devstack/commit/988ce885d366dd0a7f8a012f9e52ee8f3403ffee]
Neutron Commit (leads to “Neutron did not start”) – July 1, 2025 [Commit SHA: 1971c4a261ca54f077e2dec8b59774602bbbe3f5]]
[https://github.com/openstack/neutron/commit/1971c4a261ca54f077e2dec8b59774602bbbe3f5]]
Regards
Renu
-----Original Message-----
From: Takashi Kajinami <kajinamit@oss.nttdata.com>
Sent: Wednesday, July 16, 2025 12:42 PM
To: Brian Haley <haleyb.dev@gmail.com>; Renu Rani <renu.rani@india.nec.com>; OpenStack Discuss <openstack-discuss@lists.openstack.org>
Cc: Shivam Shukla <shivam.shukla3@india.nec.com>; Pankaj Kumar Pandey <pankaj.pandey@india.nec.com>
Subject: Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch
[You don't often get email from
kajinamit@oss.nttdata.com. Learn why this is important at
https://aka.ms/LearnAboutSenderIdentification ]
External Message: Please be cautious when opening links or attachments in email
Looking at the build result[1] in the DNM patch mentioned, I found that pip failed to install neutron from repo due to the following error.
[1]
https://zuul.opendev.org/t/openstack/build/ef1dbefc5dab40c9a6e13cb4badaa3b2
AttributeError: module 'setuptools.build_meta' has no attribute 'get_requires_for_build_editable'. Did you mean: 'get_requires_for_build_sdist'?
I think this indicates the issue is caused by incompatibility between pip and setuptools which is caused by recent multiple updates in packaging.
One thing I noticed is that the job still runs in Ubuntu Jammy, although Jammy was removed from supported operating system in this cycle and should have been replaced by Ubuntu Noble. Probably you have to first try migrating these jobs to noble and see if that
can resolve the issue.
On 7/16/25 1:20 AM, Brian Haley wrote:
> Hi Renu,
>
> On 7/15/25 3:16 AM, Renu Rani wrote:
>> Hi Neutron Team,
>>
>> We are currently facing an issue in Tacker Zuul patch jobs build (with master branch), where the DevStack setup fails during the initial Neutron network creation step.
>>
>> *Problem statement:*
>>
>> The failure occurs at this point in the stack.sh script, for devstack environment creation in build process.
>>
>> Below command execution is failing in stack.sh
>>
>> *[openstack --os-cloud devstack-admin-demo --os-region RegionOne
>> router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc
>> d777d04b-93ff-4e94-b6a9-cc8de3e23921]*
>
> I looked quickly at the log files and it looks like you are also using the networking-sfc code. We just fixed an issue with eventlet there which could cause a bad interaction with neutron:
>
> https://revi/
>
ew.opendev.org%2Fc%2Fopenstack%2Fnetworking-sfc%2F%2B%2F954633&data=05
> %7C02%7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7
> Ccc4532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168427284%7CUnkno
> wn%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXa
> W4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=olPBCnubJho
> UYWt%2FCi6WoUJA2dzpiHKljXCxWs4lmG8%3D&reserved=0
>
> I'm not sure if it's related but that change just merged an hour ago, so it might be worth re-running things to check.
>
> This was the version of networking-sfc I see being built:
>
> Successfully installed networking-sfc-20.1.0.dev3
>
> And it was pulling-in eventlet (which was just removed in that patch):
>
> Requirement already satisfied: eventlet>=0.27.0 in
> /opt/stack/data/venv/lib/python3.12/site-packages (from
> networking-sfc==20.1.0.dev3) (0.40.1)
>
> If that doesn't help please file a bug against neutron so someone can look further.
>
> Thanks,
>
> -Brian
>
>
>>
>> **
>>
>> *Error Message:*
>>
>> /This results in a 500 Internal Server Error, as seen in the logs:/
>>
>> /[HttpException: 500: Server Error for url:
>>
http://10.x/
>> .x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7cb
>> f5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com%
>> 7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110c
>> %7C0%7C0%7C638882467168459279%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGk
>> iOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoy
>> fQ%3D%3D%7C0%7C%7C%7C&sdata=qn%2FaQZX5%2FaGkPyInL8RKDpr1WhliJE%2BpJ90
>> HANXME%2FQ%3D&reserved=0
>> <http://0.0.0.10/.
>> x.x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7c
>> bf5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com
>> %7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110
>> c%7C0%7C0%7C638882467168484477%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcG
>> kiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjo
>> yfQ%3D%3D%7C0%7C%7C%7C&sdata=HhIBD9MAnWFoMCBtXX5FY3k2tfvlKiBmMKDiTFuy
>> iNQ%3D&reserved=0>]/
>>
>> *Analysis for the issue:*
>>
>> Below error is coming in Neutron API logs:
>>
>> *[ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on
>> un-acquired lock]***
>>
>> 1. Tried overriding the required-projects in Zuul to use stable/2025.1
>> branches (instead of master) to identify project(master branch)
>> which is causing root cause.
>> 2. Tacker team has created DNM patch (952892) to invetsigate this in
>> detail.
>>
>> https://rev/
>>
iew.opendev.org%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02%
>> 7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc45
>> 32b813fc4545981297eef82e110c%7C0%7C0%7C638882467168506073%7CUnknown%7
>> CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4z
>> MiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=anYUb6qkQlI%2
>> Fe1FSOcEtkdsr0iuv3OnNS96Dh2Zl4eY%3D&reserved=0
>> <https://re/
>>
view.opendev.org%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02
>> %7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4
>> 532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168521702%7CUnknown%
>> 7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4
>> zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=0J1axj9cioqQ
>> BuKReXSeCkA8rWpR2kPJ1YWmjQhG4DU%3D&reserved=0>
>>
>> 3. In DNM patch Patchset 11 (on June 29), we have taken devstack
>> stable/2025.1. In stable branch the devstack environment created
>> sucessfully.
>> 4. In DNM patch Patchset 25 (on July 8), with the same devstack
>> override (stable/2025.1), the job now fails earlier during DevStack
>> setup.
>>
>> The logs show that Neutron failed to start, causing the job to fail across all nodes.
>>
>> As per observations and anlysis seems the recent changes in the Neutron master branch between June 29 and July 8 might be causing this issue.
>>
>> We’d appreciate your help in reviewing the changes in Neutron's master branch during that window that could cause this behavior and suggesting any debugging approaches or patches we can test on tacker.
>>
>> Please let us know if you'd like additional logs or info from the Zuul job runs.
>>
>> Thanks in advance for your support!
>>
>> Thanks & Regards,
>>
>> Renu Rani(レヌラニ)
>>
>> Mobile: +91-9311888962
>>
>> E-mail: renu.rani@india.nec.com <mailto:renu.rani@india.nec.com>
>>
>> The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
>> It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates.
>> Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates.
>> Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited.
>> If you have received this email in error please delete it and notify the sender immediately.
>>
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates.
Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates.
Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited.
If you have received this email in error please delete it and notify the sender immediately.
--
Thanks and Regards
Yatin Karel
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates.
Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates.
Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited.
If you have received this email in error please delete it and notify the sender immediately.