November 2024 - openstack-discuss

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-19)
by Goutham Pacha Ravi 18 Nov '24

18 Nov '24

Hello Stackers, We're now past milestone-1 of the 2025.1 ("Epoxy") release cycle [1]. As a contributor/maintainer, you have probably made plans to adjust your goals and project deliverables in light of the likely drop in capacity for the year-end holiday season that's ahead. In the past week, the OpenStack Technical Committee didn't commit any new governance changes; however, we're actively working on our action items from the PTG [2]. === Weekly Meeting === The TC's weekly meeting last week was well attended [3]. Clark Boylan (clarkb) noted that contributors were reaching out to OpenDev Infrastructure Maintainers via IRC and the Zuul email list seeking help to set up and configure "third-party" CI systems to meet the requirements set by several projects (Nova, Neutron, Cinder, Manila, ...). Contributors often lack clear documentation and support for setting up these systems, leading to misplaced queries on Zuul or OpenDev channels. We agreed that there is a lack of centralized, up-to-date documentation regarding CI systems put together to test things outside of the purview of the community's Zuul CI. We have lost mindshare regarding third-party CI setup and maintenance, and this hurts the participation of vendor driver maintainers. The TC would love to see interested parties come together to re-form the "Third-Party CI SIG." If we're unable to meaningfully help contributors with this, we must evaluate whether third-party CI should remain a mandatory requirement for service projects. It was highlighted that Nova maintains a dedicated weekly topic to discuss specific third-party CI issues, a model other projects might emulate. We also discussed kicking off the election planning for the 2025.2 cycle. There will be 5 seats available on the OpenStack TC, and every OpenStack project team will elect a PTL or liaison [4]. Election officials will convene over the next few weeks to propose election dates and other details. There's still time to volunteer to be an election official. Please contact me off-list if you're interested! The next meeting of the OpenStack Technical Committee is today (11/18/2024) at 1800 UTC. This meeting will be held via IRC on OFTC's #openstack-tc channel. Please find the meeting agenda on the meeting wiki [5]. I hope you're able to attend. Please remember, you can always propose new topics for this meeting, and it's a great way to synchronously engage with the TC every week. === Governance Proposals === ==== Open for Review ==== rework the initial goal proposal as suggested by people | https://review.opendev.org/c/openstack/governance/+/931254 Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/931254 Add Cinder Huawei charm | https://review.opendev.org/c/openstack/governance/+/867588 Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 ==== Merged ==== Add ubuntu noble migration goal tracking etherpad | https://review.opendev.org/c/openstack/governance/+/935461 Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] OpenStack TC's Epoxy PTG Summary: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [3] TC Meeting IRC Log 2024-11-12: https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-12-18.00.log.html [4] OpenStack Election website: https://governance.openstack.org/election/ [5] TC Meeting Agenda, 2024-11-19: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee

1 0

Re: [Devstack] Trying to setup development environment on VM
by Ishan Shanware (ishanwar) 18 Nov '24

18 Nov '24

Hey Martin, Artemio, Thanks for responding. The VM is sufficiently large for development with about 60 GB of root storage. I was able to fix the keystone failure, by changing the OVS_RUNDIR like this OVS_RUNDIR=$OVS_PREFIX/var/run/ovn. However the script hangs here, and doesn’t move ahead. +functions:wait_for_compute:505 echo 'Didn'\''t find service registered by hostname after 60 seconds' Didn't find service registered by hostname after 60 seconds +functions:wait_for_compute:506 openstack --os-cloud devstack-admin --os-region RegionOne compute service list I am trying to setup a development environment for the nova service. Thanks Ishan From: Martin Kopec <mkopec(a)redhat.com> Date: Tuesday, 19 November 2024 at 4:05 AM To: Artemio Mendoza <artemiom681(a)gmail.com> Cc: Ishan Shanware (ishanwar) <ishanwar(a)cisco.com>, openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: Re: [Devstack] Trying to setup development environment on VM Hi there, the VM should be big enough, the snip from the logs doesn't suggest any reason why keystone didn't start. There is one job that runs devstack on debian every day - https://zuul.openstack.org/builds?job_name=devstack-platform-debian-bookwor… You can try to compare your local.conf with the one used in the jobs. On Mon, 18 Nov 2024 at 21:08, Artemio Mendoza <artemiom681(a)gmail.com<mailto:artemiom681@gmail.com>> wrote: Hi Ishan, In my experience, I was able to set up devstack in the Ubuntu 22.04 (Jammy) VM. I never tried Debian. Hopefully, it can help you. Thank you, Artemio On Mon, Nov 18, 2024 at 12:46 PM Ishan Shanware (ishanwar) <ishanwar(a)cisco.com<mailto:ishanwar@cisco.com>> wrote: Hi Everyone, I am currently trying to setup in all-in-one development environment for OpenStack on a Debian VM. The VM has 16 GB RAM and 4 CPUs. However, I face an error while trying to run FORCE=yes ./stack.sh. I get an error that keystone does not start. This causes the entire process to fail. Requesting assistance in solving this issue. I am following the this guide: https://docs.openstack.org/devstack/latest/guides/single-machine.html PFA the error log and error message on the terminal. [cid:ii_19340db3c084cff311] Thanks in advance, Ishan -- Martin Kopec IRC: kopecmartin

1 0

[tc][ptg] 2025.1 "Epoxy" Technical Committee PTG Summary
by Goutham Pacha Ravi 18 Nov '24

18 Nov '24

Hello Stackers, The following is a summary of the Technical Committee's virtual PTG discussions during the last week. For longer notes / recordings, please consult the Etherpad [1] where we kept minutes, or recordings within a YouTube playlist [2]. === Reviving the Inclusive language discussion === The TC brainstormed with the Diversity and Inclusion Working Group [3] about the renaming of the "master" development branch in repositories. We evaluated whether a TC resolution is necessary to provide a clear, standardized approach. Much of OpenDev tooling allows projects the ability to customize their default branch names, with an understanding that this could sometimes be confusing for tools and end-users. Some open-source communities have attempted similar renaming efforts but faced pushback or incomplete adoption, as highlighted by Kubernetes, Ceph, and GNOME. Additionally, there remain concerns over the feasibility, workload, and support available in the OpenStack community to undertake a coordinated renaming effort. The discussion did not yield a consensus and we will not be pursuing an OpenStack wide effort to renaming all the development branches in OpenStack. This doesn't prevent a future change of course, however. If we do undertake this in the future, we will begin by assessing the community's bandwidth for the effort and we'll identify contributors who will lead and assist before proceeding. The conversation also considered the broader scope of inclusive language. The D&I WG does not strongly recommend renaming the "master" branch of git repositories as it is not linked to problematic contexts like slavery in this case. More concerning terminology, like "slave," should be prioritized: https://wiki.openstack.org/wiki/Diversity/Inclusivity Action Items: - Evaluate and Draft a TC Resolution (gouthamr): TC resolution will propose formalizing OpenStack's stance on inclusive language, which would include specific recommendations about branch naming. - Encourage a timeline to deal with non-Inclusive language in code and documentation (tc) === Community Leaders meeting === Subtopic 1: Translating OpenStack documentation and i18n SIG's challenges The OpenStack i18n team discussed challenges and potential improvements for translating OpenStack documentation, particularly due to a slow shift from Zanata to Weblate. The translation team currently has limited activity and faces infrastructure issues, such as Weblate integration problems after a recent cloud upgrade. Key goals are to engage Asian communities (e.g., Korea, Vietnam, Indonesia) to support documentation translations, focusing initially on a subset of projects like Nova, Cinder, Neutron, and Glance. To streamline the process, the i18n team proposed starting with machine translations, which would be refined by regional language moderators. They also discussed AI-generated translations and compliance with OpenInfra's AI policy, requiring clear indication of AI involvement in commits. The i18n team needs infrastructure support, including Zuul automation for translation updates, and guidance for project core teams on reviewing translations. The discussion continued in the i18n SIG meetings during the rest of the week: https://etherpad.opendev.org/p/oct2024-ptg-i18n Action Items - I18n SIG will work on weblate migration with priority, and that will need resolving infrastructure issues as well as figuring out Zuul integration. - Sylvain Bauza (bauzas) takes on the role of the i18n TC liaison, and will highlight the SIGs issues to the TC - Begin translation efforts for a subset of projects (starting with Nova) by translating content in the “docs” folders. - Apply machine translation as a first step, with periodic updates pushed to each project until Zuul automation is set up. - Clearly mark AI-assisted translations in commit messages and add a notice on documentation pages to indicate machine translation usage, in line with OpenInfra’s AI policy. - Define and document processes for core teams to review translated content, considering creating specialized review teams for translation directories. Subtopic 2: Vulnerability Management and your project We discussed the processes within the OpenStack Vulnerability Management Team (VMT) and the need for consistency/coherence in the face of evolving regulation across the world, besides our interest in maintaining good security hygiene. A key concern expressed was the need to keep core security contacts up to date, and the potential for automatic inclusion of all OpenStack projects under the VMT's purview. Adding all project teams and deliverables under the VMT will provide consistency in our security bug management process. It will also allow the VMT to formally help smaller project teams to maintain a good security hygiene. It will allow for cross-project bugs to be resolved quickly while still maintaining the tenets of embargoed disclosures. We also discussed the possibility of a Common Vulnerabilities and Exposures (CVE) authority within OpenStack. Jeremy Stanley (fungi) emphasised that as a community we must be prioritizing alignment and avoiding premature optimizations while regulations are still being drafted. There were suggestions that the VMT wanted the TC to act as a path of escalation to projects where the core security contacts were unresponsive. If we include all TC governed projects under the VMT, there were also concerns about the VMT's workload increasing. fungi clarified factors that drain the VMT's time (mainly chasing project security contacts to triage issues in a timely manner), and didn't think that adding all OpenStack projects would be a burden on the VMT. Having consistent processes may in fact reduce the VMT's workload. There is a strong desire however to grow the VMT with dedicated volunteers. So if you're reading this and are interested, please hop into OFTC's #openstack-security channel and join us. Action Items: - Project PTLs (or DPLs) must update their core-sec teams on Launchpad/Storyboard, removing inactive members where necessary (e.g., Nova to review its `nova-coresec` members) - PTLs of each project must confirm or designate a security liaison for each project, or confirm that the PTL will take on this role. - gouthamr will propose a TC resolution to include all governed projects automatically included in VMT oversight - the security-sig will continue engaging with the Open Regulatory Compliance Working Group to monitor and influence relevant regulatory developments (without immediate process changes on our end at this time) - We will also create a path of escalation for projects not actively following security processes, which may include notifying TC without detailed specifics or, as a last resort, considering project removal from OpenStack. - The VMT and OpenStack Security Team will evaluate the need and feasibility of a common CVE authority for OpenStack and assess community feedback on its potential value. Subtopic 3: Remove Postgres CI jobs from the projects We discussed whether to discontinue Postgres CI jobs across projects, with Ironic and Neutron teams already planning to do so. This stems from a lack of testing coverage and recurring errors in some projects, particularly Neutron. The group raised concerns about the need for a consistent, community-wide approach to database support, especially given the challenges of switching database backends. Current user survey data indicate that about 5% of deployments use Postgres, but maintaining its support is not feasible without dedicated resources. Action Items - Update OpenStack documentation to clearly outline the level of support and testing for each database backend, helping users understand support status before deploying. - Projects (like Ironic, Neutron) must drop their postgres CI jobs, earlier in this cycle so we can prevent unexpected CI failures in interconnected projects - Reviving postgres support will need volunteers. The TC last published a resolution in 2017 [4] explaining the state of support for postgresql. There must be a follow up to state that non-MySQL backends are not tested within the community. Subtopic 4: Migrate CI testing from Ubuntu 22.04 to Ubuntu 24.04 (this implies python3.12 support/usage) Ghanshyam Mann (gmann) is championing the goal to ensure that the advertised runtimes for 2025.1 are enforced in the community CI jobs. Base CI job changes are already underway Action Items: - gmann will share the goal progress with the community as this progresses - project teams must triage integration job failures with Python3.12/Ubuntu Noble and prioritize fixes. They may temporarily pin selective test jobs to python3.9/Ubuntu Jammy however, we expect these issues to be resolved sooner within this release cycle. === Eventlet Removal === This was a "kick-off" cross project discussion on the proposed goal to drop the usage of the "eventlet" library across OpenStack repositories [5]. The migration plan includes identifying suitable alternatives (e.g., native threads, asyncio, awaitlet) to meet different services' needs while addressing challenges around async compatibility with existing infrastructure (e.g., WSGI). Hervé Beraud (hberaud) led this discussion, and we discussed the pop-up team that has been formed to guide this transition. I'd refer you to the summary that hberaud shared on the openstack-discuss ML earlier for a more in-depth summary of further discussions [6] Action Items: - The TC and the community must review the proposal [7] for outcomes, timeline and select the "Eventlet Removal" goal as a cross-community goal - hberaud will set up working groups focused on specific migration needs: background tasks, networking/HTTP, and database interactions. Details are outlined on the Eventlet Removal Wiki [4] - Mike Bayer (zzzeek) and hberaud will work with project teams to assess where asyncio may be beneficial or problematic, and provide guidelines on selecting alternative solutions - We need volunteers from project teams to join the pop-up team and evaluate eventlet usages in the projects they maintain. Please head to OFTC's #openstack-eventlet-removal channel to participate in the discussion === Bridging the gap between community and contributing organizations === Ildiko Vancsa (ildikov) and Jeremy Stanley (fungi) brought the fourth of a series of community discussions to the TC and the community. We brainstormed ways to increase contributions from OpenInfra member organizations, improve contributor experience, and address blockers to community growth and engagement. The goals outlined are to bridge gaps between casual contributors and experienced community members, optimize review processes, and enhance visibility of contribution pathways and expectations. The challenges discussed included visibility of documentation, contributor engagement with maintainers, review delays, and difficulty navigating IRC. The brainstorming that followed proposed some ideas to try, such as contributor spotlights for maintainers (e.g., Superuser articles or video interviews). Please review the "bridging the gap" PTG etherpad for further discussions around the topic [8] Action Items: - Project PTLs must ensure we have clear documentation on patch review and escalation paths. This includes clearly highlighting the core maintainers, liaisons and PTL and ways to contact them. Teams must also list any code review rules that projects have evolved in the documentation. - Project maintainers must consider “review days” to prioritize and expedite small patches. - Develop a Matrix/Element connection guide for OpenStack rooms as an alternative to IRC promoting the guide across "so you want to contribute" pages. - Formalize guidelines around code review pitfalls === OceanBase Database === Members of the OceanBase project community had a cross-project discussion with OpenStack maintainers through this session. OceanBase was stated to be a compatible alternative to MySQL within OpenStack, without necessitating code changes in OpenStack services. The OceanBase team aims to support broader community involvement by contributing integration code for devstack and deployment tools like Kolla, OpenStack-Helm, and OpenStack-Ansible (OSA). The team learned about resource constraints in the community CI jobs, and we brainstormed approaches to support integration testing. Action Items: - OceanBase contributors will implement OceanBase support in devstack, potentially as a plugin if setup is complex. - The team will explore an OpenStack’s CI job by working with the OpenStack QA team, targeting tempest testing - The team will initiate feature requests within deployment projects (e.g., Kolla, OpenStack-Helm, OSA), and potentially collaborate with OpenStack's Large Scale SIG === Updating the OpenStack tenant policy on CI === As a community, we have been discouraging indiscriminate "recheck"s on failed CI jobs; however, our recent experiences landing interdependent CVE fixes within several repositories necessitated fighting repeated unrelated failures. The discussion was around some desire to avoid this in the future and several approaches were brainstormed. Clark Boylan (clarkb) helped the TC understand OpenStack's policy regarding "clean check" practices in the Zuul CI [9]. This policy is intended to reduce repeated gate failures and encourage debugging. It would be an anti-pattern to explore a way to bypass this. The idea of dropping this policy was discussed and the attendees overwhelmingly agreed that there were significant downsides to doing this, including the infeasibility of re-orienting each project team towards how to responsibly craft their CI to avoid pitfalls. There was a suggestion to allocate a limited “infra budget” per project, allowing them to assess re-check frequency and test requirements based on specific project stability and job complexity. There was a tangential discussion on why Zuul's philosophy of testing does not allow re-running single failed jobs, and several contributors shared their experience with CI elsewhere where this convenience came at the expense of introducing serious bugs. Currently, the community can reach out in OFTC's #opendev channel to get help merging a change despite Zuul's disagreements, de-queing a change, or re-enqueuing it on Zuul. These bypass mechanisms were deemed sufficient for the problem at hand. Action Items: - The need to recheck stems from unstable test jobs. Teams must try to move flaky tests to experimental or periodic test queues and attempt to isolate and fix issues with them. - If you spot common failure patterns (e.g., timeouts, mirror issues, OOM errors), please raise awareness to the project teams or to the Infra administrators via the openstack-discuss mailing list or the #opendev channel on OFTC - Zuul CI could use documentation that clarifies why re-running individual jobs in a failed buildset is not a supported action to manage contributor expectations. === Testing and shipping non-OSI compatible software within OpenStack binary artifacts === This discussion was an opportunity to clarify the community's stance on adding, testing, maintaining, supporting, documenting and shipping OpenStack software that pertains to components that are not licensed with an OSI-approved license [10]. A specific recent example was Masakari's Consul integration. Consul ships with a non-OSI-compliant BSL 1.1 license, and is as such incompatible with OpenStack's license policy. However, the general rule has been that the community would strive to support and test only open source solutions. While there may be integrations to proprietary or non-OSI licensed components, they will not be tested with Community Infrastructure, and in each case, a free and open source alternative must be available. In addition, if an OpenStack service supports a feature, there must be a OSI compatible implementation that the community will support and test with CI, and there can be any number of implementations for the feature concerning non-OSI components, but these must be tested by vendors or users of such integrations. Some projects (like Neutron and Cinder) have documented guidelines for this, other projects (like Nova and Manila) enforce this as an unwritten rule. This creates ambiguity around licensing and integration expectations. At the tail end, there was a suggestion to explore the feasibility of hosting an OpenStack container registry (e.g., registry.openstack.org) This could help us better manage binary artifacts and maintain container consistency across services, though permanent registry storage remains a concern. The discussion around this item was tabled. Action Items: - The TC will formally define and document a policy requiring open-source alternatives for non-OSI software integrations. We must ensure that it is applied consistently across all OpenStack projects. - In continuation of this topic, we will review and clarify how to handle existing integrations with dependencies that have changed licenses (e.g., Redis). Determine whether to seek alternatives, discontinue first-party CI testing, or require third-party CI for these integrations. That's a wrap! It was great fun seeing you all at the PTG. I look forward to working on these AIs with you! Thanks, On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] PTG Etherpad for TC discussions: https://etherpad.opendev.org/p/r.0f25532f564fb786a89cfe1de39b004b [2] Recordings of the TC PTG: https://www.youtube.com/playlist?list=PLhwOhbQKWT7XGjIwT0mtPpixpuY-tKYoh [3] PTG Etherpad of the D&I working group: https://etherpad.opendev.org/p/r.4f257833bbd7e0284c26f34e2bbc87c6 [4] TC Resolution on the state of testing of database systems: https://governance.openstack.org/tc/resolutions/20170613-postgresql-status.… [5] Eventlet removal wiki: https://wiki.openstack.org/wiki/Eventlet-removal [6] Eventlet removal PTG Summary: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [7] Eventlet removal Goal Proposal: https://review.opendev.org/c/openstack/governance/+/931254 [8] "Bridging the Gap" PTG Etherpad: https://etherpad.opendev.org/p/r.522712847f6a5a1d7bd2031566cde4e9 [9] "Clean Check" CI policy: https://docs.openstack.org/contributors/common/zuul-status.html#why-do-chan… [10] OSI Approved Licenses: https://opensource.org/licenses

2 1

[watcher] re-starting a weekly irc meeting
by Marios Andreou 18 Nov '24

18 Nov '24

Hello OpenStack o/, As discussed at PTG [1] and since there are now a few of us looking to contribute to watcher, it seems like a good time to re-start a regular irc meeting in #openstack-watcher for plans, reviews or any other relevant topics. It seems that Tuesday or Thursday work well for most of the current/new contributors. I started a doodle poll with these two options there https://doodle.com/meeting/participate/id/aMoRKp5d to see if there is any clear preference between the two. I provisionally propose 1200 UTC to (try) and accommodate different time zones (obviously there is no perfect solution for this). If you are interested but the time and/or days do not work for you please reply here so we can try and work it out (If there is enough interest we may consider something like alternate times but first lets see if there is a preference or problem with the current proposal), thanks! marios [1] https://etherpad.opendev.org/p/r.79644f8f3cd04686c4f45c10aa35d8ac#L1

2 1

[Glance] Survey: Property protection file
by Cyril Roelandt 18 Nov '24

18 Nov '24

Hello, Glance allows admins to define custom rules regarding CRUD (create/read/update/delete) operations on custom image properties[1]. This is done by setting the property_protection_file option in glance-api.conf to the path to an INI file defining such permissions. We believe this feature has not seen a lot of use, and are thinking about removing it. We would like to know what you think about this. We set up an online form[2] that you can fill to help us understand real-world use cases for this feature. 1) Are you using a property protection file? 2) Are you using roles, polices, or both for property protection? (Or "None" if you do not use this feature) 3) Can you give examples of config files you use? 4) Would it be an issue for you if Glance were to remove the feature? Only the role-base protection? Only the policy-based protection? On behalf of the Glance team, Cyril [1] https://docs.openstack.org/glance/latest/admin/property-protections.html [2] https://framaforms.org/survey-glance-property-protection-file-1731944193

1 0

[neutron][requirements] Lock failures within Neutron with Eventlet 0.37.0
by Herve Beraud 18 Nov '24

18 Nov '24

Hey Neutrinos, Mid-September we released Eventlet 0.37.0 [1]. Then Jens diligently proposed a patch [2] to bump our global requirements to this newest version of Eventlet. Unfortunately this requirement upgrade is stalled by the Neutron cross job [3]. There are plenty of tests failing with lock errors in threading context (`RuntimeError: cannot notify on un-acquired lock`, please see the logs [3]). A new patch [4] changed the way we manage RLocks and threading at runtime. This patch Upgrade RLocks as the last thing we do. Green existing locks after patching modules, since patching modules might involve imports that create new locks. Neutron tests might potentially need some adjustment to properly work with this new Eventlet version. The support of Python 3.13 is coming and it includes this behavioural change, so if we want the Python 3.13 support of Eventlet in OpenStack, we need first to adjust our OpenStack deliverables to prepare the ground. Can you please check this issue with your Neutron expert eyes. Feel free to also take a look at the changelog of the version 0.37.0 of Eventlet [5], but I think the other patches are irrelevant to this context and can be safely ignored as they are unrelated to locks or threading. To finish Masakari and Mistral also failed but the errors seem a bit different than the Neutron ones. In all the cases Masakri and Mistral cross jobs are non-voting jobs. Thanks for reading. [1] https://github.com/eventlet/eventlet/issues/979 [2] https://review.opendev.org/c/openstack/requirements/+/933257 [3] https://zuul.opendev.org/t/openstack/build/e399c76765f34bfda62e2125b80caf84… [4] https://github.com/eventlet/eventlet/commit/06ec82896ebb9a26edaf6e1ad4d6339… [5] https://github.com/eventlet/eventlet/issues/979 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

2 2

[neutron] Bug deputy - week of Nov 11th
by Sławek Kapłoński 18 Nov '24

18 Nov '24

Hi, I was Neutron bug deputy last week. Here's summary of what new bugs were opened during that period: ## Critical - https://bugs.launchpad.net/neutron/+bug/2088423 - [functional test] test_ovn_nb_sync* tests fails randomly with Internal Server Error - needs assignment, - https://bugs.launchpad.net/neutron/+bug/2088285 - [neutron-tempest-plugin] neutron-dynamic-routing using os-ken Quagga are failing - assignent to Lajos, fix already proposed, we have https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/935366 and https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/935368 ## High - https://bugs.launchpad.net/neutron/+bug/2088166 - os-ken ``DockerImage`` class failing during the image creation - assigned to Rodolfo and patch is proposed already, - https://bugs.launchpad.net/neutron/+bug/2088286 - external port created by neutron-vpnaas with ovn driver is not fully configured - needs assignment, this seems to be bug related to the `ovn-bgp-agent` and `vpnaas` ## Medium - https://bugs.launchpad.net/neutron/+bug/2088154 - ``ProcessManager.disable`` failing if the process doesn't exist - assigned to Rodolfo and patch is proposed already, ## Low - https://bugs.launchpad.net/neutron/+bug/2087909 - When network log exists, neutron-ovn-db-sync-util has a wrong log message - needs assignment, ## Whishlist - https://bugs.launchpad.net/neutron/+bug/2087863 - fullstack job doesn't collect keepalived and other backend logs - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087939 - [eventlet-deprecation] Remove the usage of eventlet in the OVS agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087941 - [eventlet-deprecation] Remove the usage of eventlet in the OVN metadata agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087942 - [eventlet-deprecation] Remove the usage of eventlet in the OVN agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087943 - [eventlet-deprecation] Remove the usage of eventlet in the L3 agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087944 - [eventlet-deprecation] Remove the usage of eventlet in the DHCP agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087946 - [eventlet-deprecation] Remove the usage of eventlet in the SR-IOV agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087947 - [eventlet-deprecation] Remove the usage of eventlet in the Linux Bridge agent - needs assignment, - https://bugs.launchpad.net/neutron/+bug/2087953 - [eventlet-deprecation] Remove the usage of eventlet in the Neutron API - needs assignment There is bunch of the "eventlet-deprecation" bugs reported this week. I marked all of them with the tag "eventlet-deprecation" so that all can be easily found: https://bugs.launchpad.net/neutron/+bugs?field.tag=eventlet-deprecation -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[neutron] external port created by neutron-vpnaas with ovn driver is not fully configured
by przemyslaw.kuczynski＠ochk.pl 18 Nov '24

18 Nov '24

Guys can You help with https://bugs.launchpad.net/neutron/+bug/2088286 Hello We are deploying OpenStack/Neturon with OVN, BGP (ovn-bgp-agent) and VPNaaS and we encountered a problem - because of incomplete configuration of VPN services external port bgp agent isn't announcing it's address. Summary: external port created by neutron-vpnaas with ovn plugin/driver is not fully/properly configured and because of this ovn-bgp-agent won't announce/publish/configure routing path for this address. $ /var/lib/kolla/venv/bin/neutron-ovn-vpn-agent --version neutron-ovn-vpn-agent 24.0.2.dev42 $ openstack network show public1 +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2024-09-06T15:48:18Z | | description | | | dns_domain | None | | id | ad8c81c1-08fd-4503-833f-912675d1c6d8 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | False | | is_vlan_transparent | None | | mtu | 1500 | | name | public1 | | port_security_enabled | True | | project_id | 0cfa2dc8d9024b7fa0462a9be5d8b832 | | provider:network_type | vlan | | provider:physical_network | physnet1 | | provider:segmentation_id | 2 | | qos_policy_id | None | | revision_number | 2 | | router:external | External | | segments | None | | shared | True | | status | ACTIVE | | subnets | 306ea02d-a5ec-4c1b-bd2f-bff1a88750d6 | | tags | | | tenant_id | 0cfa2dc8d9024b7fa0462a9be5d8b832 | | updated_at | 2024-09-06T15:48:19Z | +---------------------------+--------------------------------------+ $ openstack subnet show 306ea02d-a5ec-4c1b-bd2f-bff1a88750d6 +----------------------+--------------------------------------+ | Field | Value | +----------------------+--------------------------------------+ | allocation_pools | 203.0.113.226-203.0.113.254 | | cidr | 203.0.113.224/27 | | created_at | 2024-09-06T15:48:19Z | | description | | | dns_nameservers | | | dns_publish_fixed_ip | None | | enable_dhcp | False | | gateway_ip | 203.0.113.225 | | host_routes | | | id | 306ea02d-a5ec-4c1b-bd2f-bff1a88750d6 | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | name | public1 | | network_id | ad8c81c1-08fd-4503-833f-912675d1c6d8 | | project_id | 0cfa2dc8d9024b7fa0462a9be5d8b832 | | revision_number | 0 | | segment_id | None | | service_types | | | subnetpool_id | None | | tags | | | updated_at | 2024-09-06T15:48:19Z | +----------------------+--------------------------------------+ $ openstack port list --network ad8c81c1-08fd-4503-833f-912675d1c6d8 --long +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | Security Groups | Device Owner | Tags | +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ | 1dc2b248-1d21-44ff-923f-2405f9f28f4e | | fa:16:3e:3e:97:10 | ip_address='203.0.113.228', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | N/A | None | network:floatingip | | | 71a1d9eb-11bb-4278-979b-6a1d83e87ecc | | fa:16:3e:15:5e:b5 | ip_address='203.0.113.229', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | | 7398b975-552a-408f-8289-25e52d5cb8fc | | fa:16:3e:e7:dd:5d | ip_address='203.0.113.252', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | | ccbb42e0-3258-4ade-a5e6-c873ca0530b7 | | fa:16:3e:97:9d:d7 | | DOWN | None | network:distributed | | | e6a43606-2ea3-4967-9aa2-967c800cbdbe | | fa:16:3e:4e:e9:d8 | ip_address='203.0.113.243', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ $ openstack vpn service create --router router_vpn test_vpn_service +----------------+--------------------------------------+ | Field | Value | +----------------+--------------------------------------+ | Description | | | Flavor | None | | ID | 0d8436f0-6cc5-4af5-84f7-10b54ac99784 | | Name | test_vpn_service | | Project | 0cfa2dc8d9024b7fa0462a9be5d8b832 | | Router | 30e448d5-078e-40fe-9418-901f8195b6cb | | State | True | | Status | PENDING_CREATE | | Subnet | None | | external_v4_ip | 203.0.113.246 | | external_v6_ip | None | | project_id | 0cfa2dc8d9024b7fa0462a9be5d8b832 | +----------------+--------------------------------------+ $ openstack vpn ipsec site connection create --peer-id 1.2.3.4 --peer-address 1.2.3.4 --psk 1234 --vpnservice test_vpn_service --ikepolicy ikepolicy --ipsecpolicy ipsecpolicy test_vpn_tunnel --peer-endpoint-group west-peer-epg --local-endpoint-group local_network +--------------------------+----------------------------------------------------+ | Field | Value | +--------------------------+----------------------------------------------------+ | Authentication Algorithm | psk | | Description | | | ID | ca7f63dc-685f-4f9d-bc23-47c7b5b1c577 | | IKE Policy | 1457dc12-c1ec-4574-8985-9e93dcf06f56 | | IPSec Policy | c68d172f-220f-455c-b46c-b9ff8b9e46e4 | | Initiator | bi-directional | | Local Endpoint Group ID | 0f7f645c-8ae9-4dc8-a286-40afabbc2dd7 | | Local ID | | | MTU | 1500 | | Name | test_vpn_tunnel | | Peer Address | 1.2.3.4 | | Peer CIDRs | | | Peer Endpoint Group ID | 842cc6e8-d124-4258-b9d8-8b901d06cd97 | | Peer ID | 1.2.3.4 | | Pre-shared Key | 1234 | | Project | 0cfa2dc8d9024b7fa0462a9be5d8b832 | | Route Mode | static | | State | True | | Status | PENDING_CREATE | | VPN Service | 0d8436f0-6cc5-4af5-84f7-10b54ac99784 | | dpd | {'action': 'hold', 'interval': 30, 'timeout': 120} | | project_id | 0cfa2dc8d9024b7fa0462a9be5d8b832 | +--------------------------+----------------------------------------------------+ $ openstack port list --network ad8c81c1-08fd-4503-833f-912675d1c6d8 --long +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | Security Groups | Device Owner | Tags | +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ | 1dc2b248-1d21-44ff-923f-2405f9f28f4e | | fa:16:3e:3e:97:10 | ip_address='203.0.113.228', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | N/A | None | network:floatingip | | | 3d2dbb2a-b004-4955-a4ca-4d4a88d2f702 | vpn-gw-30e448d5-078e-40fe-9418-901f8195b6cb | fa:16:3e:e5:e0:69 | ip_address='203.0.113.246', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:vpn_router_gateway | | | 71a1d9eb-11bb-4278-979b-6a1d83e87ecc | | fa:16:3e:15:5e:b5 | ip_address='203.0.113.229', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | | 7398b975-552a-408f-8289-25e52d5cb8fc | | fa:16:3e:e7:dd:5d | ip_address='203.0.113.252', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | | ccbb42e0-3258-4ade-a5e6-c873ca0530b7 | | fa:16:3e:97:9d:d7 | | DOWN | None | network:distributed | | | e6a43606-2ea3-4967-9aa2-967c800cbdbe | | fa:16:3e:4e:e9:d8 | ip_address='203.0.113.243', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ routing table on gateway/network node: 203.0.113.228 dev vrf1d63891f-e7 scope link 203.0.113.243 dev vrf1d63891f-e7 scope link 203.0.113.252 dev vrf1d63891f-e7 scope link 203.0.113.229 dev vrf1d63891f-e7 scope link $ ovn-nbctl lsp-list `ovn-nbctl ls-list | grep ad8c81c1-08fd-4503-833f-912675d1c6d8 | cut -f1 -d\ ` | while read a b; do echo -n "$b "; ovn-nbctl lsp-get-addresses $a; done (3d2dbb2a-b004-4955-a4ca-4d4a88d2f702) unknown (71a1d9eb-11bb-4278-979b-6a1d83e87ecc) fa:16:3e:15:5e:b5 203.0.113.229/27 (7398b975-552a-408f-8289-25e52d5cb8fc) fa:16:3e:e7:dd:5d 203.0.113.252/27 (ccbb42e0-3258-4ade-a5e6-c873ca0530b7) fa:16:3e:97:9d:d7 (e6a43606-2ea3-4967-9aa2-967c800cbdbe) fa:16:3e:4e:e9:d8 203.0.113.243/27 (provnet-5aa931a9-ac56-4144-ab7d-c61819a46c2a) unknown 3d2dbb2a-b004-4955-a4ca-4d4a88d2f702 (vpn-gw-30e448d5-078e-40fe-9418-901f8195b6cb) stands out from other ports, has no addresses $ ovn-nbctl lsp-set-addresses 3d2dbb2a-b004-4955-a4ca-4d4a88d2f702 "fa:16:3e:e5:e0:69 203.0.113.246/27" $ ovn-nbctl lsp-list `ovn-nbctl ls-list | grep ad8c81c1-08fd-4503-833f-912675d1c6d8 | cut -f1 -d\ ` | while read a b; do echo -n "$b "; ovn-nbctl lsp-get-addresses $a; done (3d2dbb2a-b004-4955-a4ca-4d4a88d2f702) fa:16:3e:e5:e0:69 203.0.113.246/27 (71a1d9eb-11bb-4278-979b-6a1d83e87ecc) fa:16:3e:15:5e:b5 203.0.113.229/27 (7398b975-552a-408f-8289-25e52d5cb8fc) fa:16:3e:e7:dd:5d 203.0.113.252/27 (ccbb42e0-3258-4ade-a5e6-c873ca0530b7) fa:16:3e:97:9d:d7 (e6a43606-2ea3-4967-9aa2-967c800cbdbe) fa:16:3e:4e:e9:d8 203.0.113.243/27 (provnet-5aa931a9-ac56-4144-ab7d-c61819a46c2a) unknown routing table after: 203.0.113.228 dev vrf1d63891f-e7 scope link 203.0.113.243 dev vrf1d63891f-e7 scope link 203.0.113.246 dev vrf1d63891f-e7 scope link 203.0.113.252 dev vrf1d63891f-e7 scope link 203.0.113.229 dev vrf1d63891f-e7 scope link And now it's possible for tunnel to connect. I tried to identify code of ovn driver/plugin in neutron-vpnaas responsible for external port allocation, but I wasn't able to find it. --

1 0

[ironic] Having issue with deploying ISO image to node
by James Leong 17 Nov '24

17 Nov '24

Hi all, I am trying to play around with ironic, and I am having an issue with deploying an Ubuntu 20.04 ISO image onto my bare metal node. Since I am quite new to bare metal, I am not entirely sure what went wrong as I couldn't find any error messages shown on my log file (ironic-conductor, ironic-api, ironic_neutron_agent, and nova-compute-ironic). This might be my guess; the ironic inspection is hitting timeout, which is causing the deployment to timeout. However, I have no idea why the inspection is failing at timeout. Any help would be appreciated. Best, James

6 7

[openstack-helm] [Cinder] Cinder-Scheduler oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start
by daniel890723＠gmail.com 17 Nov '24

17 Nov '24

My Enviroment: | OS | Ubuntu 22.04 | | Kernel | 5.15.0-87-generic | | kubernetes | v1.30.3 | (tested with both v1.28.14 and v1.30.3) Hello I'm testing with Openstack-helm the installation works fine but when i trying to create volume from image and when creating Cinder-Scheduler shows error and the log output: Cinder-Scheduler log: 2024-11-18 01:13:21.651 1 ERROR cinder.scheduler.filter_scheduler [None req-d4455f4e-a374-4251-ab04-7c3523809b6f d7a3b766f6274ea6b83d6f4b28639b6e 5fc62074ca69400f8192bf00941616d5 - - - -] Error scheduling e838be3b-80a2-4c6c-98cc-53cc84527236 from last vol-service: cinder-volume-worker@rbd1#rbd1 : ['Traceback (most recent call last):\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task\n result = task.execute(**arguments)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/volume/flows/manager/create_volume.py", line 1241, in execute\n model_update = self._create_from_image(context,\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/utils.py", line 669, in _wrapper\n return r(f, *args, **kwargs)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/tenacity/__init__.py", line 379, in __call__\n do = self.iter(retry_state=retry_state)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/tenacity/__init__.py", line 314, in iter\n return fut.result()\n', ' File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result\n return self.__get_result()\n', ' File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result\n raise self._exception\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/tenacity/__init__.py", line 382, in __call__\n result = fn(*args, **kwargs)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/volume/flows/manager/create_volume.py", line 1115, in _create_from_image\n model_update, cloned = self._prepare_image_cache_entry(\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/decorator.py", line 232, in fun\n return caller(func, *(extras + args), **kw)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/coordination.py", line 239, in _synchronized\n return f(*a, **k)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/volume/flows/manager/create_volume.py", line 894, in _prepare_image_cache_entry\n model_update = self._create_from_image_cache_or_download(\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/volume/flows/manager/create_volume.py", line 978, in _create_from_image_cache_or_download\n with image_utils.TemporaryImages.fetch(\n', ' File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__\n return next(self.gen)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/image/image_utils.py", line 1375, in fetch\n fetch_verify_image(context, image_service, image_id, tmp)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/image/image_utils.py", line 861, in fetch_verify_image\n data = get_qemu_data(image_id, has_meta, format_raw,\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/image/image_utils.py", line 700, in get_qemu_data\n data = qemu_img_info(dest,\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/image/image_utils.py", line 169, in qemu_img_info\n format_name = cinder.privsep.format_inspector.get_format_if_safe(\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 269, in _wrap\n self.start()\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 283, in start\n channel = daemon.RootwrapClientChannel(context=self)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 374, in __init__\n super(RootwrapClientChannel, self).__init__(sock, context)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 187, in __init__\n self.exchange_ping()\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 201, in exchange_ping\n raise FailedToDropPrivileges(msg)\n', 'oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start\n'] 2024-11-18 01:13:21.652 1 INFO cinder.message.api [None req-d4455f4e-a374-4251-ab04-7c3523809b6f d7a3b766f6274ea6b83d6f4b28639b6e 5fc62074ca69400f8192bf00941616d5 - - - -] Creating message record for request_id = req-d4455f4e-a374-4251-ab04-7c3523809b6f 2024-11-18 01:13:21.667 1 ERROR cinder.scheduler.flows.create_volume [None req-d4455f4e-a374-4251-ab04-7c3523809b6f d7a3b766f6274ea6b83d6f4b28639b6e 5fc62074ca69400f8192bf00941616d5 - - - -] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. Exceeded max scheduling attempts 3 for resource e838be3b-80a2-4c6c-98cc-53cc84527236: cinder.exception.NoValidBackend: No valid backend was found. Exceeded max scheduling attempts 3 for resource e838be3b-80a2-4c6c-98cc-53cc84527236 I also tried several way to test by set SYS_ADMIN on Cinder_scheduler pod change the version to 2023.2 but non of the above method works Is this because the pod permission or there is something i need to set when deploy using openstack-helm ? P.S : i can create Empty Volume and mount to instance

1 0