openstack-discuss search results for query "#eventlet-removal"
openstack-discuss@lists.openstack.org- 186 messages
Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch
by Yatin Karel
Hi Renu and Shivam,
On Thu, Jul 31, 2025 at 9:20 AM Shivam Shukla <shivam.shukla3(a)india.nec.com>
wrote:
> Hi Yatin,
>
> Hope you're doing well.
>
> First of all, thank you once again for your support and valuable input on
> the Neutron-related issue we were facing in the Tacker base job. Your
> suggestion to remove 'logger' from the Q_ML2_PLUGIN_MECHANISM_DRIVERS configuration
> resolved the "Internal Server Error" and Neutron startup problems, as
> confirmed in the following build:
> https://zuul.opendev.org/t/openstack/build/48d742463f02457c8a99dd0fed0db1bb
>
> Just as a gentle reminder — when you get a chance, it would be really
> helpful if you could share more details about the root cause of this issue.
> Specifically, we’d like to understand why the presence of 'logger' causes
> the failure, how removing it resolves the problem, and whether any
> functional or behavioral changes are expected as a result after its
> (logger) removal.
>
> It's mainly related to eventlet removal[1], there you can see a couple of
patches related to dropping "logger" mechanism drivers.
[1]
https://review.opendev.org/q/topic:%22eventlet-removal%22+project:openstack…
> This explanation will help us document the reasoning clearly in the patch
> and provide proper justification for the change.
>
> Looking forward to your insights.
>
> Thank You.
>
> Regards,
> Shivam
> ------------------------------
> *From:* Shivam Shukla <shivam.shukla3(a)india.nec.com>
> *Sent:* Sunday, July 27, 2025 10:35 PM
> *To:* Yatin Karel <ykarel(a)redhat.com>; Renu Rani <renu.rani(a)india.nec.com>
> *Cc:* Takashi Kajinami <kajinamit(a)oss.nttdata.com>; Brian Haley <
> haleyb.dev(a)gmail.com>; OpenStack Discuss <
> openstack-discuss(a)lists.openstack.org>; Pankaj Kumar Pandey <
> pankaj.pandey(a)india.nec.com>
> *Subject:* Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron
> in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch
>
> Hi Yatin,
> Thank you for your input regarding the shared issues.
>
> JFYI... I tried your suggested recommendation i.e., remove "logger" from
> Q_ML2_PLUGIN_MECHANISM_DRIVERS in Tacker base job configuration. It
> resolves the issues ("Internal Server Error" and "Neutron did not start")
> related to Neutron.
>
> Please refer this build result:
> https://zuul.opendev.org/t/openstack/build/48d742463f02457c8a99dd0fed0db1bb
>
> Also, I would also appreciate it if you could share some details about the
> root cause of the issue and explain how removing the 'logger' resolves it."
>
> Thank you once again for your inputs.
>
> Regards,
> Shivam
> ------------------------------
> *From:* Yatin Karel <ykarel(a)redhat.com>
> *Sent:* Monday, July 21, 2025 11:13 PM
> *To:* Renu Rani <renu.rani(a)india.nec.com>
> *Cc:* Takashi Kajinami <kajinamit(a)oss.nttdata.com>; Brian Haley <
> haleyb.dev(a)gmail.com>; OpenStack Discuss <
> openstack-discuss(a)lists.openstack.org>; Shivam Shukla <
> shivam.shukla3(a)india.nec.com>; Pankaj Kumar Pandey <
> pankaj.pandey(a)india.nec.com>
> *Subject:* Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron
> in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch
>
> You don't often get email from ykarel(a)redhat.com. Learn why this is
> important <https://aka.ms/LearnAboutSenderIdentification>
> *External Message:* Please be cautious when opening links or attachments
> in email
>
> Hi,
>
> On Mon, Jul 21, 2025 at 8:22 PM Renu Rani <renu.rani(a)india.nec.com> wrote:
>
> Dear Takashi San,
>
> Thanks for your support.
>
> Please fine below further investigation updates from Tacker:
>
> 1. Kubernetes-Based Job – tacker-ft-v2-k8s
> Error: "AttributeError: module 'setuptools.build_meta' has no attribute
> 'get_requires_for_build_editable'. Did you mean:
> 'get_requires_for_build_sdist'?"
> Root Cause:
> The issue is caused by the default setuptools version
> (59.6.0) included in the base OS image (Ubuntu Jammy). Since
> Global_Venv=False is set in local.conf, DevStack does
> not use a virtual environment, leading to the use of the
> outdated setuptools version. The required attribute is only available from
> setuptools version 64 onward.
> Fix:
> Set Global_Venv=True in local.conf. This enables the
> DevStack virtual environment where setuptools version 80 is installed,
> resolving the issue by using the correct setuptools version.
>
>
> 2.OpenStack-Based Job – tacker-ft-v1-compliance-sol
> Error#1:
> Internal Server Error during execution of the command:
> "openstack --os-cloud devstack-admin-demo --os-region RegionOne
> router add subnet 20b264df-5f19-4599-a8"
> Root Cause:
> Below devstack commit started causing this issue.
> DevStack Commit (leads to “Internal Server Error”) – May 20, 2025
> --> Seems Devstack code update related to neutron.
> [Commit sha: 988ce885d366dd0a7f8a012f9e52ee8f3403ffee]
>
> Error#2:
> [ERROR] Neutron did not start
> Root Cause:
> Below Neutron commit started causing this issue:
> Neutron Commit (leads to “Neutron did not start”) – July 1, 2025
> [Commit SHA: 1971c4a261ca54f077e2dec8b59774602bbbe3f5]]
>
> For this I suggested some changes in the test patch
> https://review.opendev.org/c/openstack/tacker/+/952892, hopefully that
> will clear it.
>
> Regards
> Renu
>
> -----Original Message-----
> From: Renu Rani <renu.rani(a)india.nec.com>
> Sent: Thursday, July 17, 2025 2:56 PM
> To: Takashi Kajinami <kajinamit(a)oss.nttdata.com>; Brian Haley <
> haleyb.dev(a)gmail.com>; OpenStack Discuss <
> openstack-discuss(a)lists.openstack.org>
> Cc: Shivam Shukla <shivam.shukla3(a)india.nec.com>; Pankaj Kumar Pandey <
> pankaj.pandey(a)india.nec.com>
> Subject: RE: [Neutron][Tacker][Issue Investigation] Issue with Neutron in
> Tacker Zuul Patches – Suspected Regression in Neutron Master Branch
>
> Dear Takashi San,
>
> Thanks a lot for your response and really appreciate sharing your inputs.
>
>
> Just to update there are two types of Tacker jobs is check pipeline: one
> is Kubernetes-based, and the other is OpenStack-based.
>
> [1] Kubernetes-based Job: tacker-ft-v2-k8s
> It appears that you reviewed the logs for this job, which is currently
> failing due to the following error:
> "AttributeError: module 'setuptools.build_meta' has no attribute
> 'get_requires_for_build_editable'. Did you mean:
> 'get_requires_for_build_sdist'?"
>
> We understand that this error is not related to Neutron changes, but
> rather due to an incompatibility between pip and setuptools, likely caused
> by recent packaging updates. We're actively investigating the root cause of
> this issue.
>
> [2] OpenStack-based Job: tacker-ft-v1-compliance-sol
> Our original request was regarding this job. It fails due to an
> "Internal Server Error" during the execution of the following command:
> "openstack --os-cloud devstack-admin-demo --os-region RegionOne router
> add subnet 20b264df-5f19-4599-a8"
> Zuul Log (controller node):
> https://zuul.opendev.org/t/openstack/build/970baccf521846ec83441c1f18898199
>
>
> To debug further, we tried checking out the stable/2025.1 branch of
> DevStack instead of master. However, the controller node now fails with
> this error:
> "[ERROR] /opt/stack/devstack/functions-common:2366 Neutron did not start"
> Zuul Log (controller node):
> https://zuul.opendev.org/t/openstack/build/09d39537443640c2a87673590f577829
>
>
> Identified the specific commits that introduced these issues:
>
> DevStack Commit (leads to “Internal Server Error”) – May 20, 2025 [Commit
> sha: 988ce885d366dd0a7f8a012f9e52ee8f3403ffee]
> [
> https://github.com/openstack/devstack/commit/988ce885d366dd0a7f8a012f9e52ee…
> ]
>
> Neutron Commit (leads to “Neutron did not start”) – July 1, 2025 [Commit
> SHA: 1971c4a261ca54f077e2dec8b59774602bbbe3f5]]
> [
> https://github.com/openstack/neutron/commit/1971c4a261ca54f077e2dec8b597746…
> ]]
>
>
> Regards
> Renu
>
> -----Original Message-----
> From: Takashi Kajinami <kajinamit(a)oss.nttdata.com>
> Sent: Wednesday, July 16, 2025 12:42 PM
> To: Brian Haley <haleyb.dev(a)gmail.com>; Renu Rani <renu.rani(a)india.nec.com>;
> OpenStack Discuss <openstack-discuss(a)lists.openstack.org>
> Cc: Shivam Shukla <shivam.shukla3(a)india.nec.com>; Pankaj Kumar Pandey <
> pankaj.pandey(a)india.nec.com>
> Subject: Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron in
> Tacker Zuul Patches – Suspected Regression in Neutron Master Branch
>
> [You don't often get email from kajinamit(a)oss.nttdata.com. Learn why this
> is important at https://aka.ms/LearnAboutSenderIdentification ]
>
> External Message: Please be cautious when opening links or attachments in
> email
>
>
> Looking at the build result[1] in the DNM patch mentioned, I found that
> pip failed to install neutron from repo due to the following error.
> [1]
> https://zuul.opendev.org/t/openstack/build/ef1dbefc5dab40c9a6e13cb4badaa3b2
>
> AttributeError: module 'setuptools.build_meta' has no attribute
> 'get_requires_for_build_editable'. Did you mean:
> 'get_requires_for_build_sdist'?
>
> I think this indicates the issue is caused by incompatibility between pip
> and setuptools which is caused by recent multiple updates in packaging.
>
> One thing I noticed is that the job still runs in Ubuntu Jammy, although
> Jammy was removed from supported operating system in this cycle and should
> have been replaced by Ubuntu Noble. Probably you have to first try
> migrating these jobs to noble and see if that can resolve the issue.
>
> On 7/16/25 1:20 AM, Brian Haley wrote:
> > Hi Renu,
> >
> > On 7/15/25 3:16 AM, Renu Rani wrote:
> >> Hi Neutron Team,
> >>
> >> We are currently facing an issue in Tacker Zuul patch jobs build (with
> master branch), where the DevStack setup fails during the initial Neutron
> network creation step.
> >>
> >> *Problem statement:*
> >>
> >> The failure occurs at this point in the stack.sh script, for devstack
> environment creation in build process.
> >>
> >> Below command execution is failing in stack.sh
> >>
> >> *[openstack --os-cloud devstack-admin-demo --os-region RegionOne
> >> router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc
> >> d777d04b-93ff-4e94-b6a9-cc8de3e23921]*
> >
> > I looked quickly at the log files and it looks like you are also using
> the networking-sfc code. We just fixed an issue with eventlet there which
> could cause a bad interaction with neutron:
> >
> > https://revi/
> > ew.opendev.org%2Fc%2Fopenstack%2Fnetworking-sfc%2F%2B%2F954633&data=05
> > %7C02%7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7
> > Ccc4532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168427284%7CUnkno
> > wn%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXa
> > W4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=olPBCnubJho
> > UYWt%2FCi6WoUJA2dzpiHKljXCxWs4lmG8%3D&reserved=0
> >
> > I'm not sure if it's related but that change just merged an hour ago, so
> it might be worth re-running things to check.
> >
> > This was the version of networking-sfc I see being built:
> >
> > Successfully installed networking-sfc-20.1.0.dev3
> >
> > And it was pulling-in eventlet (which was just removed in that patch):
> >
> > Requirement already satisfied: eventlet>=0.27.0 in
> > /opt/stack/data/venv/lib/python3.12/site-packages (from
> > networking-sfc==20.1.0.dev3) (0.40.1)
> >
> > If that doesn't help please file a bug against neutron so someone can
> look further.
> >
> > Thanks,
> >
> > -Brian
> >
> >
> >>
> >> **
> >>
> >> *Error Message:*
> >>
> >> /This results in a 500 Internal Server Error, as seen in the logs:/
> >>
> >> /[HttpException: 500: Server Error for url:
> >> http://10.x/
> >> .x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7cb
> >> f5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com%
> >> 7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110c
> >> %7C0%7C0%7C638882467168459279%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGk
> >> iOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoy
> >> fQ%3D%3D%7C0%7C%7C%7C&sdata=qn%2FaQZX5%2FaGkPyInL8RKDpr1WhliJE%2BpJ90
> >> HANXME%2FQ%3D&reserved=0
> >> <http://0.0.0.10/.
> >> x.x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7c
> >> bf5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com
> >> %7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110
> >> c%7C0%7C0%7C638882467168484477%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcG
> >> kiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjo
> >> yfQ%3D%3D%7C0%7C%7C%7C&sdata=HhIBD9MAnWFoMCBtXX5FY3k2tfvlKiBmMKDiTFuy
> >> iNQ%3D&reserved=0>]/
> >>
> >> *Analysis for the issue:*
> >>
> >> Below error is coming in Neutron API logs:
> >>
> >> *[ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on
> >> un-acquired lock]***
> >>
> >> 1. Tried overriding the required-projects in Zuul to use stable/2025.1
> >> branches (instead of master) to identify project(master branch)
> >> which is causing root cause.
> >> 2. Tacker team has created DNM patch (952892) to invetsigate this in
> >> detail.
> >>
> >> https://rev/
> >> iew.opendev.org%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02%
> >> 7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc45
> >> 32b813fc4545981297eef82e110c%7C0%7C0%7C638882467168506073%7CUnknown%7
> >> CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4z
> >> MiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=anYUb6qkQlI%2
> >> Fe1FSOcEtkdsr0iuv3OnNS96Dh2Zl4eY%3D&reserved=0
> >> <https://re/
> >> view.opendev.org%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02
> >> %7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4
> >> 532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168521702%7CUnknown%
> >> 7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4
> >> zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=0J1axj9cioqQ
> >> BuKReXSeCkA8rWpR2kPJ1YWmjQhG4DU%3D&reserved=0>
> >>
> >> 3. In DNM patch Patchset 11 (on June 29), we have taken devstack
> >> stable/2025.1. In stable branch the devstack environment created
> >> sucessfully.
> >> 4. In DNM patch Patchset 25 (on July 8), with the same devstack
> >> override (stable/2025.1), the job now fails earlier during DevStack
> >> setup.
> >>
> >> The logs show that Neutron failed to start, causing the job to fail
> across all nodes.
> >>
> >> As per observations and anlysis seems the recent changes in the Neutron
> master branch between June 29 and July 8 might be causing this issue.
> >>
> >> We’d appreciate your help in reviewing the changes in Neutron's master
> branch during that window that could cause this behavior and suggesting any
> debugging approaches or patches we can test on tacker.
> >>
> >> Please let us know if you'd like additional logs or info from the Zuul
> job runs.
> >>
> >> Thanks in advance for your support!
> >>
> >> Thanks & Regards,
> >>
> >> Renu Rani(レヌラニ)
> >>
> >> Mobile: +91-9311888962
> >>
> >> E-mail: renu.rani(a)india.nec.com <mailto:renu.rani@india.nec.com>
> >>
> >> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> >> It shall not attach any liability on the originator or NEC Corporation
> India Private Limited or its affiliates.
> >> Any views or opinions presented in this email are solely those of the
> author and may not necessarily reflect the opinions of NEC Corporation
> India Private Limited or its affiliates.
> >> Any form of reproduction, dissemination, copying, disclosure,
> modification, distribution and / or publication of this message without the
> prior written consent of the author of this e-mail is strictly prohibited.
> >> If you have received this email in error please delete it and notify
> the sender immediately.
> >>
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> It shall not attach any liability on the originator or NEC Corporation
> India Private Limited or its affiliates.
> Any views or opinions presented in this email are solely those of the
> author and may not necessarily reflect the opinions of NEC Corporation
> India Private Limited or its affiliates.
> Any form of reproduction, dissemination, copying, disclosure,
> modification, distribution and / or publication of this message without the
> prior written consent of the author of this e-mail is strictly prohibited.
> If you have received this email in error please delete it and notify the
> sender immediately.
>
>
>
> --
> Thanks and Regards
> Yatin Karel
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> It shall not attach any liability on the originator or NEC Corporation
> India Private Limited or its affiliates.
> Any views or opinions presented in this email are solely those of the
> author and may not necessarily reflect the opinions of NEC Corporation
> India Private Limited or its affiliates.
> Any form of reproduction, dissemination, copying, disclosure,
> modification, distribution and / or publication of this message without the
> prior written consent of the author of this e-mail is strictly prohibited.
> If you have received this email in error please delete it and notify the
> sender immediately.
>
--
Thanks and Regards
Yatin Karel
5 months
[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-19)
by Goutham Pacha Ravi
Hello Stackers,
We're now past milestone-1 of the 2025.1 ("Epoxy") release cycle [1].
As a contributor/maintainer, you have probably made plans to adjust
your goals and project deliverables in light of the likely drop in
capacity for the year-end holiday season that's ahead. In the past
week, the OpenStack Technical Committee didn't commit any new
governance changes; however, we're actively working on our action
items from the PTG [2].
=== Weekly Meeting ===
The TC's weekly meeting last week was well attended [3]. Clark Boylan
(clarkb) noted that contributors were reaching out to OpenDev
Infrastructure Maintainers via IRC and the Zuul email list seeking
help to set up and configure "third-party" CI systems to meet the
requirements set by several projects (Nova, Neutron, Cinder, Manila,
...). Contributors often lack clear documentation and support for
setting up these systems, leading to misplaced queries on Zuul or
OpenDev channels. We agreed that there is a lack of centralized,
up-to-date documentation regarding CI systems put together to test
things outside of the purview of the community's Zuul CI. We have lost
mindshare regarding third-party CI setup and maintenance, and this
hurts the participation of vendor driver maintainers. The TC would
love to see interested parties come together to re-form the
"Third-Party CI SIG." If we're unable to meaningfully help
contributors with this, we must evaluate whether third-party CI should
remain a mandatory requirement for service projects. It was
highlighted that Nova maintains a dedicated weekly topic to discuss
specific third-party CI issues, a model other projects might emulate.
We also discussed kicking off the election planning for the 2025.2
cycle. There will be 5 seats available on the OpenStack TC, and every
OpenStack project team will elect a PTL or liaison [4]. Election
officials will convene over the next few weeks to propose election
dates and other details. There's still time to volunteer to be an
election official. Please contact me off-list if you're interested!
The next meeting of the OpenStack Technical Committee is today
(11/18/2024) at 1800 UTC. This meeting will be held via IRC on OFTC's
#openstack-tc channel. Please find the meeting agenda on the meeting
wiki [5]. I hope you're able to attend. Please remember, you can
always propose new topics for this meeting, and it's a great way to
synchronously engage with the TC every week.
=== Governance Proposals ===
==== Open for Review ====
rework the initial goal proposal as suggested by people |
https://review.opendev.org/c/openstack/governance/+/931254
Propose to select the eventlet-removal community goal |
https://review.opendev.org/c/openstack/governance/+/931254
Add Cinder Huawei charm |
https://review.opendev.org/c/openstack/governance/+/867588
Resolve to adhere to non-biased language |
https://review.opendev.org/c/openstack/governance/+/934907
==== Merged ====
Add ubuntu noble migration goal tracking etherpad |
https://review.opendev.org/c/openstack/governance/+/935461
Thank you very much for reading!
On behalf of the OpenStack TC,
Goutham Pacha Ravi (gouthamr)
OpenStack TC Chair
[1] 2025.1 "Epoxy" Release Schedule:
https://releases.openstack.org/epoxy/schedule.html
[2] OpenStack TC's Epoxy PTG Summary:
https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack…
[3] TC Meeting IRC Log 2024-11-12:
https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-12-18.00.log.html
[4] OpenStack Election website: https://governance.openstack.org/election/
[5] TC Meeting Agenda, 2024-11-19:
https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee
1 year, 1 month
[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-17)
by Goutham Pacha Ravi
Hello Stackers,
This week marks the 2024.2 ("Dalmatian") release trailing deadline
[1]. All projects following the cycle-trailing release model [2][3]
must release their 2024.2 ("Dalmatian") deliverables by December 5,
2024.
No new governance changes were merged in the past week; goal champions
and volunteers continued their efforts on the cross-community goals
that have been committed to [4]. As Ghanshyam Mann (gmann) highlighted
in this mailing list [5], nine project teams need to address CI jobs
within their repositories due to the switch of base DevStack (and some
tox) jobs to Ubuntu 24.04 LTS. Teams can work around the problem
temporarily by pinning the nodeset used to run their failing jobs to
Ubuntu 22.04 LTS while they triage and resolve these issues.
The Ironic and Neutron teams recently dropped functional testing with
the PostgreSQL database. As highlighted at the PTG [6], there is very
little motivation within OpenStack's contributor community to support
PostgreSQL as a database alternative to MySQL and its derivatives with
Oslo.db. The TC has clarified this in a past resolution as well [7].
We are currently identifying places within the documentation where
PostgreSQL references need to be removed and aim to address these
promptly. If you rely on PostgreSQL, please voice your concerns and
consider dedicating time and resources to reviving testing with this
database.
=== Weekly Meeting ===
TC members met last Tuesday (2024-11-26) on OFTC's #openstack-tc
channel [8]. A significant portion of the discussion focused on the
state of CI. Members of the Testing and Collaboration Tools SIG
highlighted that much of the gate flakiness is due to jobs
encountering the new rate limits on DockerHub [9]. Alternatives that
may be more suitable for CI usage within open-source communities were
discussed. As a result, we encourage project teams to explore these
alternatives, as many open-source communities have already done.
The OpenStack TC's next weekly meeting is today (2024-12-03). It will
be held simultaneously via video (Zoom) and IRC (OFTC's #openstack-tc
channel) at 18:00 UTC. The agenda for this meeting, along with
participation links, can be found on the wiki [10]. I hope to see you
there!
=== Governance Proposals ===
==== Open for Review ====
- Retire Murano/Senlin/Sahara OpenStack-Ansible roles:
https://review.opendev.org/c/openstack/governance/+/935677
- Add ansible-role-httpd repo to OSA-owned projects:
https://review.opendev.org/c/openstack/governance/+/935694
- Rework the initial goal proposal as suggested by the community:
https://review.opendev.org/c/openstack/governance/+/931254
- Resolve to adhere to non-biased language:
https://review.opendev.org/c/openstack/governance/+/934907
- Propose to select the eventlet-removal community goal:
https://review.opendev.org/c/openstack/governance/+/934936
Thank you for reading!
On behalf of the OpenStack TC,
Goutham Pacha Ravi (gouthamr)
OpenStack TC Chair
[1] 2025.1 "Epoxy" Release Schedule:
https://releases.openstack.org/epoxy/schedule.html
[2] Cycle-trailing release model:
https://releases.openstack.org/reference/release_models.html#cycle-trailing
[3] Cycle-trailing projects:
https://codesearch.opendev.org/?q=type%3A%20trailing&i=nope&literal=nope&fi…
[4] Proposed and selected community goals:
https://governance.openstack.org/tc/goals/#community-goals
[5] Migrate to noble communication:
https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack…
[6] OpenStack TC PTG Notes (around line 159):
https://etherpad.opendev.org/p/r.0f25532f564fb786a89cfe1de39b004b
[7] State of community support for databases:
https://governance.openstack.org/tc/resolutions/20170613-postgresql-status.…
[8] TC Meeting IRC Log 2024-11-26:
https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-26-18.01.log.html
[9] DockerHub rate limit changes:
https://www.docker.com/blog/november-2024-updated-plans-announcement/
[10] TC Meeting Agenda, 2024-12-03:
https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee
1 year
[ironic][ptg] 2025.2 (Flamingo) PTG Summary
by Julia Kreger
Greetings everyone!
Last week Ironic had a wildly successful PTG! We averaged around
fifteen attendees most days, we drew upwards of 22 attendees for most
of Wednesday while we discussed a number of related topics around
networking.
Overall, networking aspects seem to show the most current interest.
This is the result in a change in market ecosystem where vendors are
less focused on SDN integrations, and also deliberate limitations
inside of ironic in order to limit scope creep and still attempt to
meet the majority of infrastructure operator requirements when the
original networking multi-tenancy effort was executed back in
2015-2016. Other major topics were Eventlet, some operator feedback on
quarks of features, which highlighted some possible bugs and areas
for improvement. We also got into some areas which have been a bit
contentious in the past, but we reached some reasonable compromises
which allowed us to find better paths forward.
Monday:
We largely discussed where we were at and where we were going. To
highlight, ironic-lib has now been retired. Redfish based Graphical
Console support merged and we identified some more work was likely
needed. Bootable Container support *and* support for artifacts from an
OCI container registry support was also viewed as completed during the
cycle which ultimately improves options to infrastructure operators
who are operating in mixed environments or with mixed requirements
outside of the classical "everything is a VM in OpenStack" context.
We've improved the linting across the majority of project
repositories. Out of band inspection rules likely need more work
around ensuring data structures are what we expect, but otherwise we
are re-affirming our deprecation of ironic-inspector as a standalone
project. Work to support Kea as a DHCP backend has paused for now. The
plan exists, but realistically the contributor working in that area
has a different focus at the moment which is okay! Container based IPA
steps also didn't make it into the Epoxy cycle release, but are almost
done already in Flamingo. Schema validation and OpenAPI work is also
still underway and we broadly expect this to merge into Flamingo.
In-band disk encryption as part of the ``direct`` deploy interface
also didn't make any forward progress due to shifting contributor
priorities. As a note, the bootable container work did extend this as
a possible option, however it remains unclear if that will solve the
overall need for the contributor who did propose adding support for
encrypted volumes to the ``direct`` interface. Efforts to find an
alternative to TinyCore usage in CI also stalled out. Turns out is
less of a trivial problem to make a super-low memory ramdisk, and
other fixes to the build jobs improved general reliability in the
meantime so it is less pressing. We also didn't make any progress on
Project Mercury as it was likely framed to try and keep too many
options open for operators while we also lost contributor velocity due
to some CVE work this past cycle.
Having discussed what we achieved, this allowed us to focus on key
aspects contributors know they needed to focus on this next cycle.
Networking was the biggest topic in this area. Promptly followed by
eventlet removal. There was also a consensus that we needed to ensure
we're working together a bit better, but also don't explicitly block
any one aspect, as there is power in user/operator choice as well.
Some possible interest was also expressed around extending network
device firmware upgrades, further delineating/improving metrics,
possibly supporting more of a "push" firmware update model which more
vendors are adopting.
We promptly shifted to CI, and discussed challenges and path forward.
Generally, there is some interest in trying to make some of our job
executions a bit more selective, and also to dial back our reliance
upon integration scenario jobs. Overall, it is a broad area of work
which will require some further discussion to try and frame an ideal
future state. In this we also discussed Centos Stream 10, python
versions, and ultimately possible paths forward to at least
highlighting other options which may enable maintenance activities to
ensure these cases work.
Afterwards, we shifted to CI knowledge sharing later in the day
outside of the PTG schedule, to help spread overall context among
newer members of the team.
Tuesday!
We started Tuesday with the topic of supporting a use case desired by
some operators wishing to "allocate" baremetal nodes into a state
which, to Ironic signifies the node has been deployed. The use case
behind that is a bit odder, more for research and academic cases where
other tools may be desired to meet very specific requirements. The
discussion yielded an operator in the scientific space which could
benefit, so some ideas and a possible path was identified.
Then we got into the world of eventlet and a mini-retrospective of the
challenges and identification of future steps. What we thought would
be simple, turned out to be the hardest problem. Ironic Python Agent,
originally identified as a good candidate for an early eventlet
migration, has turned out to be extremely difficult due to it's need
to spawn it's own WSGI server late in the process. Upon further
reflection; Ironic itself has this issue as well, because we actually
have two places a WSGI server might be spun up: ironic-api (obviously)
and the conductor when using json-rpc. It was clear that solving that
technical problem was the first step in our migration. We found some
examples of gunicorn used in this way; that may be what we look to for
an initial prototype.
We then spoke about node / hardware monitoring patterns that operators
are using and interested in. Redfish based hardware supports sending a
notification to a web service on a threshold violation so we discussed
adding a listener into Ironic that could receive these and integrate
this into the current hardware event notifications. We agreed that a
spec would be necessary and we would craft one and proceed from there.
Further improvements around making it easier for operators to separate
monitoring/events of the services from the hardware itself was
discussed but oslo_messaging does not allow for this type of split
routing today. We spoke about looking further into
ironic-prometheus-exporter to see how it could be configured to
potentially support this separation.
To wrap up Tuesday, we drifted into discussion of the serial console
support which Ironic has today. In essence, there is quite a bit of
room for possible improvement there and some operators who are not
using Nova wouldn't mind this area to see attention moving forward.
The broad idea is we might explore creation of a new interface which
enables SSH connections to be proxied through to the IPMI
Serial-over-LAN capabilities in a BMC. This distinctly would require
IPMI, since there really is not a great answer to serial
console/interface access with Redfish. Ultimately this may also mean
we extend redfish to support an IPMI interface as well, which seems
weird, but discussion was in broad agreement that this space has
unique challenges where this makes sense. Some initial ideas were
entered into an spec document for discussion and refinement as some of
the interested parties are also on the perimeter of the Ironic
project.
Wednesday!
Wednesday was our deep dive into networking!
Our very first topic was focused on improving the capabilities around
the intentional limitations which were encoded in Ironic's early
multi-tenant networking work. The early work was mainly focused on
ensuring tenant level isolation on to separate L2 domains, such as
VLANs. In large part, this boils down to the mapping, scheduling, and
ultimately the dynamic assembly of a port grouping (bonded
interfaces). We came into this session with a specification document
from a discussion which occurred a few weeks prior and there was
general consensus in this option and discussion highlighted a few
other intertwined aspects which need to be accounted for.
We then dove into ACL support for Networking Generic Switch. Overall,
consensus resulted. This discussion also highlighted a number of
aspects we also likely need to keep in mind, and potentially may start
best be described as overall needs for additional documentation. There
are also some further discussions which may need to be revisited
networking wise. Ultimately, this is very much an "integrated" use
case which was only appealing to about half of the attendees, which is
typical given the variety of use cases Ironic solves and is able to
support.
Then we reached the topic of Standalone Switch management. This
quickly became a retrospective into why we didn't make progress with
Mercury, and then drove into what are the real minimal requirements
which would need to be taken into account. Some of the discussion
revolved around questions which were semi-answered in the mercury plan
itself, to try and provide enough overlap to generally be usable in
more than one use model/case while also enabling or preventing
fragmentation. This discussion also crossed over into support of DPUs
and how to support them, in large part because the overall model is
very different. Some discussion also sort of shifted into maybe
merging and tools together.
Ultimately this topic requires much more discussion, and to make solid
progress on networking aspects we need to enable ourselves to move
together in two directions at once, while not blocking each other.
With this in mind, we're forming up into two sub-teams which will
focus on each area with their own advocates/champions. The goal being
to report back to the community each week in a quick update style of
engagement.
Thursday!
Thursday marked our final day of PTG for Ironic. We started by
wrapping up some of the networking discussions, in large part the
focus on how to make progress. We used this time to identify our high
level organizational plans, determine broad interest levels and
ultimately outreach. One key aspect which was highlighted, is the
initial primary focus for some will likely be making progress on
eventlet before shifting gears into networking.
We then shifted gears to discussing interest and requirements of
mapping storage volumes to overall hosts via DPU devices, which
somewhat crossed over into the networking topics. The broad idea was
presented with a potential need to frame compound drivers around
facilitating broadly different configuration actions. For example,
directly via SSH to invoke commands, or for example update a CRD in a
distinct OpenShift cluster. Overall, the discussion yielded that maybe
invoking a CRD update was not worthwhile given the flux those areas
can experience along with competing requirements. Ultimately this is
an area in early discussion and would require investment in time and
hardware to move forward, but there were really no objections to doing
so if we could model and extend what already exists in a useful and
applicable way.
We then drifted back into networking with our next topic, bridging
distinctly different types of fabric together. This was raised much
more as a question to see if there was an interest/need and/or
requirement. In essence, this is similar to the original l2gw project.
A huge highlight of embracing or extending into a more ideal model is
the reality that if this existed, it might not be necessary for VXLAN
to be considered on the physical side.
Discussions then shifted to improving servicing interactions.
Servicing functionality was added to Ironic during one of the recent
past development cycles to enable firmware upgrades to take place on
deployed nodes. In discussion, it was highlighted that in the current
model this could take five or more reboots to facilitate the
deployment as the current mechanism is largely modeled on the use of
agent heartbeats. The discussion shifted to how we could avoid this.
And the obvious answer was "add a periodic". Then concerns over adding
more periodic queries and jobs shifted the discussion into how we can
address that challenge. Ultimately, this may end up with us in a
better place for tasks which need to follow-up on state changes in
hardware and revisit the state before proceeding with the next step to
execute.
As the final topic, we dove into deploy steps. This really boiled down
to "how do I know what will occur" when I ask Ironic to do something
along with "how do I know what occurred". Consensus reached that
largely the key aspect to highlight back to the user is "what was
done" or "what do we expect to be done" in terms of steps. An idea was
raised to record this into node history, which is a historical record
list of actions/errors to a node which has existed in Ironic for a
number of cycles now, but is most useful if you're trying to figure
out prior events without going to logs. Consensus was that doing it
this way would be a "quick win".
At some point, Ironic contributors also sat down and reviewed jammed
outside of the PTG schedule to review/discuss and merge existing items
sitting awaiting reviews.
Overall, the week was extremely productive! Thanks to everyone who
took part. Additional thanks to everyone who collaborated on this
summary, and everyone who also helped push our project priorities
published for this work cycle[3].
Please remember if you have action items to follow-up. And onward to Flamingo!
- The Ironic Team
[0]: https://etherpad.opendev.org/p/ironic-ptg-april-2025
[1]: https://review.opendev.org/c/openstack/ironic-specs/+/945642
[2]: https://review.opendev.org/c/openstack/ironic-specs/+/946723
[3]: https://specs.openstack.org/openstack/ironic-specs/priorities/2025-2-workit…
8 months, 2 weeks
[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-21)
by Goutham Pacha Ravi
Hello Stackers,
Thanks to all moderators and volunteers for compiling virtual PTG
notes and sharing summaries through this mailing list. A lot of
exciting work seems to be tracked through the action items from these
summaries. I hope you were able to review the TC's PTG summary [1].
Next week is milestone-1 of the 2025.1 "Epoxy" release cycle [2]. This
is typically a bug-targeting milestone for project teams. The
following is a summary of this past week's happenings within the
OpenStack Technical Committee.
=== Weekly Meeting ===
During the past week, the OpenStack Technical Committee met over IRC
[3]. We followed up on action items from weeks preceding the Project
Teams Gathering. We finalized PTL and DPL appointments for the
"mistral" and "watcher" project teams. We spent some time reviewing
action items related to translating documentation and user strings
within OpenStack services. Automated jobs posting translation updates
from Zanata are broken due to incompatibility with newer Python
versions. Migration to Weblate remains a priority and the i18n team
may require more support to complete this migration. We're
collaborating over IRC to identify the needs and next steps.
The TC also discussed progress on the community goal to migrate
integration testing to Python 3.12 and to the latest LTS version of
the operating system used in DevStack test jobs. Like me, you may have
wondered why we’re combining these efforts. The reason is that it
allows us to have a simpler test matrix despite the number of Python
versions and OS distributions that OpenStack software is tested with.
While we maintain unit test jobs for each supported Python version, it
is impractical to duplicate each integration job for every OS
distribution available. Therefore, we choose the latest LTS (e.g.,
Ubuntu 24.04 LTS / "Noble Numbat"), which includes the upper bound of
the supported Python versions (Python 3.12). Additionally, each
project will have at least one test job for the minimum supported
Python version (Python 3.9), so you'll see a test job for the previous
LTS release of the OS that will begin in this release and continue
running as it transitions to a "stable" release branch.
The next meeting of the OpenStack Technical Committee is on 2024-11-05
(today) at 1800 UTC. This is our monthly video meeting too! This
meeting will occur simultaneously on Video (Zoom) and IRC (OFTC's
#openstack-tc channel). Allison Price (aprice) and Jimmy McArthur
(jimmymcarthur) from the OpenStack VMware Migration Working Group [4]
will be joining us to discuss the working group's efforts. Meeting
information, including other agenda items, is on the meeting wiki [5].
I hope you'll be able to join us. Please remember that you can always
propose items to the TC's weekly meeting agenda.
=== Governance Proposals ===
==== Merged ====
Add Watcher DPL for Epoxy |
https://review.opendev.org/c/openstack/governance/+/933018
Adding Axel Vanzaghi as PTL for Mistral |
https://review.opendev.org/c/openstack/governance/+/927962
==== Open for Review ====
Propose to select the eventlet-removal community goal |
https://review.opendev.org/c/openstack/governance/+/931254
=== How to Contact the TC ===
You can reach the TC in several ways:
1. **Email:** Send an email with the tag [tc] on this mailing list.
2. **IRC:** Use the 'tc-members' keyword on the #openstack-tc IRC
channel on OFTC.
3. **Weekly Meeting:** Join us every Tuesday at 1800 UTC [5].
Thank you very much for reading!
On behalf of the OpenStack TC,
Goutham Pacha Ravi (gouthamr)
OpenStack TC Chair
[1] 2025.1 "Epoxy" TC PTG Summary:
https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack…
[2] 2025.1 "Epoxy" Release Schedule:
https://releases.openstack.org/epoxy/schedule.html
[3] TC Meeting IRC Log 2024-10-29:
https://meetings.opendev.org/meetings/tc/2024/tc.2024-10-29-18.00.log.html
[4] OpenStack VMware Migration Group:
https://www.openstack.org/vmware-migration-to-openstack/
[5] TC Meeting Agenda, 2024-11-05:
https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting
1 year, 1 month
[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-16)
by Goutham Pacha Ravi
Hello Stackers,
We're 16 weeks away from the 2025.1 ("Epoxy") release date
(2024-04-02) [1]. OpenStack project teams are planning bug squashes
and review jams in the next couple of weeks before contribution
activity slows due to end-of-the-year holidays around the world. The
OpenStack Technical Committee did not merge any new governance changes
this week; however, a few proposals are iterating with community and
TC review.
=== Weekly Meeting ===
The OpenStack Technical Committee held its monthly video conference
last week, which was simultaneously recorded via text on IRC [2][3].
The discussion tracked several TC initiatives for this release cycle
[4]. Most notably, the recent migration of base integration jobs to
Ubuntu 24.04 LTS and Python 3.12 has caused CI jobs to break for
several code repos [5]. I’d like to reiterate that project teams must
resolve these issues or temporarily pin the nodeset of their test jobs
to "ubuntu-jammy" to buy some time for fixes.
We then checked on the status of "unmaintained" branches. The release
team wanted the TC to solidify processes for two specific issues:
nominating "unmaintained" liaisons and establishing guidelines for
retiring these branches. The lack of such processes has resulted in
the proliferation of "unmaintained" code branches across the community
and failure to implement the TC resolution that created these branches
[6].
The oldest unmaintained branch is "unmaintained/victoria", and it’s
unclear if anyone benefits from keeping it around. Discussion on
cleanup has been delayed due to uncertainty about what to do [7].
Meanwhile, branches like Yoga may still have high usage and active
contributors, emphasizing the need for a clear process to decide their
fate. The technical committee may consider advising the release team
to eliminate branches with low usage, but this doesn’t solve the
problem of finding volunteers to maintain high-usage branches. The
idea is that projects can opt to maintain more branches if there is
sufficient user interest and support, but there is currently no
process for this, nor is it clear how to recruit volunteers. We agreed
to continue this discussion at the next meeting. However, in the
meantime, there were no objections to EOL-ing the Victoria, Wallaby,
and Xena branches.
Clark Boylan (clarkb) noted an issue with building CentOS Stream 10
images using Disk Image Builder, as they cannot boot on several clouds
due to the x86_64 v3 CPU instruction set requirement. Only a few
provider clouds, including OpenMetal and Rackspace Flex, meet the CPU
requirements to boot CentOS 10. This limitation may affect options for
regular test nodes, and project teams are advised to take notice.
Until all Zuul/Nodepool cloud providers meet the CPU requirements,
teams may need to adjust their testing strategies.
Jeremy Stanley (fungi) reminded us that nominations for individual
member representatives for the OpenInfra Foundation Board of Directors
are still open [8]. Ten nominations from members are required to make
it onto the ballot.
The OpenStack TC's next weekly meeting is today, 2024-12-10. This will
be an IRC meeting hosted on #openstack-tc at 1800 UTC. Please find the
agenda on the meeting wiki [9]. I look forward to seeing you there.
=== Governance Proposals ===
==== Open for Review ====
Add ansible-role-httpd repo to OSA-owned projects |
https://review.opendev.org/c/openstack/governance/+/935694
Retire Murano/Senlin/Sahara OpenStack-Ansible roles |
https://review.opendev.org/c/openstack/governance/+/935677
Rework the initial goal proposal as suggested by people |
https://review.opendev.org/c/openstack/governance/+/931254
Resolve to adhere to non-biased language |
https://review.opendev.org/c/openstack/governance/+/934907
Propose to select the eventlet-removal community goal |
https://review.opendev.org/c/openstack/governance/+/934936
Thank you very much for reading!
On behalf of the OpenStack TC,
Goutham Pacha Ravi (gouthamr)
OpenStack TC Chair
[1] 2025.1 "Epoxy" Release Schedule:
https://releases.openstack.org/epoxy/schedule.html
[2] TC Meeting Video Recording, 2024-12-03: https://youtu.be/KlYbFxjhACs
[3] TC Meeting IRC Log 2024-12-03:
https://meetings.opendev.org/meetings/tc/2024/tc.2024-12-03-18.00.log.html
[4] TC Tracker for 2025.1: https://etherpad.opendev.org/p/tc-2025.1-tracker
[5] Runtime update to Ubuntu 24.04/Python3.12:
https://etherpad.opendev.org/p/migrate-to-noble
[6] TC Resolution to create "unmaintained" code branches:
https://governance.openstack.org/tc/resolutions/20230724-unmaintained-branc…
[7] Transition "victoria" code branch to EOL:
https://review.opendev.org/c/openstack/releases/+/935373
[8] Nominations for OpenInfra Individual Board Member Elections:
https://lists.openinfra.dev/archives/list/foundation@lists.openinfra.dev/th…
[9] TC Meeting Agenda, 2024-12-10:
https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee
1 year