November 2024 - openstack-discuss

[neutron][ptg] 2025.1 Epoxy PTG summary
by Brian Haley 08 Nov '24

08 Nov '24

Hi all, First, sorry for taking so long to get this out, was traveling the week after the PTG. Second, thanks for all that attended the meetings, we had a good turnout for Neutron, the Ironic cross-team and some eventlet discussions that were important to the team. For complete notes see the etherpad [0] but a summary is below. And please comment if you feel I missed anything. Thank you, -Brian ### Dalmatian retrospective ### Everyone felt we were able to improve our CI stability, despite a few spikes during the cycle. We were also able to improve the OVN parity gap with ML2/OVS, and were able to move OVN to supporting WSGI (thanks Rodolfo!) Lack of developers and reviewers continues to be an issue, and we have struggled to reduce our bug backlog significantly. ### Eventlet deprecation ### We have started to identify and document changes needed to deprecate eventlet [1]. We will start creating bugs to track all this work. ### Spring cleanup ### At the beginning of every cycle, we always try and cleanup the code base, for example, removing deprecated and dead code, doc updates, job consolidation. We have made progress on those already. We had a discussion on two items that have been marked experimental in the code - Linuxbridge support and IPv6 prefix delegation - neither of which is well maintained or tested in our CI. I will send a follow-up on next steps for these in the coming week. ### Code base modernization ### As python 3.9 is now the minimal version, we are going through the steps of running pyupgrade/autopep8 on both the neutron and neutron-lib trees to adopt any new conventions. In the process we've been fixing a number of pylint warnings that is still ongoing. The mypy job is now voting, and we have started adding type hints to the code (thanks Ihar!). ### Ironic cross-project ### The high-level discussion point was on Neutron better supporting baremetal (of course). There are some issues with trunk ports as well as external network Single Source of Truth (to help better integrate Neutron with DC networking). An RFE will be proposed for either creating a pre-commit hook to help validate a VNI is okay to use, or to be able to modify the VNI range via the API, as it is currently fixed in the conf file. The other topic was around implementing an additional DHCP backend, as dnsmasq has not been actively maintained, and there was at least one issue Ironic was seeing with it crashing. The consensus was to start supporting Kea, an RFE will be created. See etherpad for further information. ### ML2/OVS future ### As two of the major distros have moved to make OVN their default, there has been less of an emphasis on ML2/OVS development over the past few cycles, and as such, there are an increasing number of bugs in the backlog. There are still a large number of clouds running ML2/OVS as seen in the 2024 User Survey [2], so there is some risk there. As a team we wanted to make sure we communicated this concern, and also wanted to make it known that users doing new deployments, or migrating from older ones (eg Linuxbridge) strongly consider using OVN. We will attempt to give any help we can in this area as migration and deployment issues arise. ### Distributed metadata ### This has been ongoing work from Liu to better support distributed metadata in ML2/OVS, similar to what is done in OVN. Patches have been updated accordingly after discussion. ### Performance/scale testing ### Some recent bugs have shown there are some places where Neutron has DB scaling issues. Any we've found so far have been fixed, but there is a need to do some targeted testing to see if there are others. We also discussed changing our rally testing to try and better detect such issues. My take away was we should be vigilant when any DB change is proposed to make sure it doesn't regress things, and to start looking at places we don't use 'selectin' to see if a simple change would help. ### Gate stability ### Similar to the weekly Neutron CI meeting, we discusses issues in the gate, from things related to the WSGI switch, scenario and fullstack failures. Please see the etherpad for specific bug links. ### RFE discussions ### On-demand topic to discuss any in-progress or proposed RFE changes, please see etherpad for more information. [0] https://etherpad.opendev.org/p/oct2024-ptg-neutron [1] https://etherpad.opendev.org/p/neutron-eventlet-deprecation [2] https://www.openstack.org/analytics

1 0

[ironic/neutron] Using neutron to bind a specific port on an ironic node
by Michael Sherman 08 Nov '24

08 Nov '24

So, my question is: Is there a way for an end-user, launching an instance with nova, to specify which ironic port neutron should use? For context, a subset of our ironic nodes have relatively heterogenous network interfaces, but are configured as being attached to the same physical networks. Examples include nodes with both Broadcom and Mellanox 25g interfaces attached to the same switch, but also nodes with both 1g, 25g, and 100g NICs attached to different switches, but with the same vlans available. In our configuration we’re using Neutron with openvswitch, networking-baremetal, and networking-generic-switch. From initial observations, it seems like if we *don’t* specify, neutron allocates ports in the same order that they’re returned from “baremetal port list”. What we’d like to be able to do is let our users send some API call to bind neutron port X based on some property of the ironic port, ether UUID, tags for speed or vendor, or something else we’re able to define, but it’s not clear to me what information is even available to neutron to use in say, a binding profile? Any pointers would be appreciated. Thanks! -Mike Sherman

1 0

[release] Release countdown for week R-20
by Thierry Carrez 08 Nov '24

08 Nov '24

Development Focus ----------------- The Epoxy-1 milestone is next week, on November 14th! Project team plans for the 2025.1 Epoxy cycle should now be solidified. General Information ------------------- Libraries need to be released at least once per milestone period. Next week, the release team will propose releases for any library which had changes but has not been otherwise released since the 2024.2 Dalmatian release. PTL's or release liaisons, please watch for these and give a +1 to acknowledge them. If there is some reason to hold off on a release, let us know that as well, by posting a -1. If we do not hear anything at all by the end of the week, we will assume things are OK to proceed. NB: If one of your libraries is still releasing 0.x versions, start thinking about when it will be appropriate to do a 1.0 version. The version number does signal the state, real or perceived, of the library, so we strongly encourage going to a full major version once things are in a good and usable state. Upcoming Deadlines & Dates -------------------------- Epoxy-1 milestone: November 14th, 2024 2025.1 Epoxy final release: April 2nd, 2025 -- Thierry Carrez (ttx)

1 0

[kolla][OSA][release] 2024.2 Dalmatian cycle-trailing release deadline
by Elõd Illés 08 Nov '24

08 Nov '24

Hello teams with deliverables following the cycle-trailing release model! This is just a reminder to wrap up your trailing deliverables for 2024.2 Dalmatian. The release deadline is *December 5th, 2024.* [1]. If things are ready sooner than that though, all the better for our downstream consumers. Some of them have been delivered already, so let's list here the ones that still need a release before the deadline: kayobe openstack-ansible-roles openstack-ansible [1] https://releases.openstack.org/epoxy/schedule.html#e-cycle-trail Thanks in advance, Előd Illés irc: elodilles @ #openstack-release

1 0

[all] CFP for OpenInfra Days North America Open for One More week!
by Jeremy Stanley 07 Nov '24

07 Nov '24

For those of you who may not be subscribed to the OpenInfra Foundation mailing list[*], just a heads up that we have another week until the OpenInfra Days North America planning committee will begin reviewing talks, so the CFP deadline has been extended to November 15. If you want to present in Pasadena, California this March, get your proposal in ASAP![**] [*] https://lists.openinfra.dev/archives/list/foundation@lists.openinfra.dev/me… [**] https://forms.gle/81QV3SFV6ixSdsBq8 -- Jeremy Stanley

1 0

[neutron] Drivers meeting canceled Friday, November 8th
by Brian Haley 07 Nov '24

07 Nov '24

Hello Neutrinos: I'm going to cancel tomorrow's drivers meeting as there is no agenda and I will be offline. I will triage incoming ones next week so plan to meet next Friday. Have a nice weekend! -Brian

1 0

[Cinder][Ceph] Volumetria errada no cluster ceph
by lfsilva＠binario.cloud 07 Nov '24

07 Nov '24

Hello, I'm experiencing an issue with the volume usage of the Ceph cluster, which is currently used in OpenStack for volumes. I am working with Ceph version Octopus (15.2.17). When I run the ceph df command, I get this output: --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 120 TiB 33 TiB 87 TiB 87 TiB 72.72 TOTAL 120 TiB 33 TiB 87 TiB 87 TiB 72.72 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 32 69 MiB 101 137 MiB 0 11 TiB images 2 256 1.3 TiB 166.90k 2.5 TiB 10.72 11 TiB vms 3 64 574 KiB 21 2.5 MiB 0 11 TiB volumes 4 2048 41 TiB 3.94M 82 TiB 79.42 11 TiB backups 5 1024 407 GiB 111.85k 818 GiB 3.63 11 TiB And when I run the rbd du -p volumes command, I get this total: NAME PROVISIONED USED <TOTAL> 14 TiB 8.3 TiB This "volumes" pool is currently set to replica 2, and mirroring is not enabled. I have checked for any locked or deleted snapshots but found none. I also ran a ceph osd pool deep-scrub volumes, but it didn't resolve the issue. Has anyone encountered this problem before? Could someone provide assistance?

3 21

[kolla] Vacation - kevko
by Michal Arbet 07 Nov '24

07 Nov '24

Hello, I would like to inform you that I will be on vacation from November 7th, 2024, to November 23rd, 2024. Thank you, Michal Arbet Michal Arbet Openstack Engineer Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic +420 604 228 897 michal.arbet(a)ultimum.io *https://ultimum.io <https://ultimum.io/>* LinkedIn <https://www.linkedin.com/company/ultimum-technologies> | Twitter <https://twitter.com/ultimumtech> | Facebook <https://www.facebook.com/ultimumtechnologies/timeline>

1 0

Pointers to FOSDEM devrooms CfPs
by Felix Kronlage-Dammers 06 Nov '24

06 Nov '24

Hi, in the last couple weeks the Call for Participations for the various FOSDEM[1] devrooms have been published. In case not everyone knows FOSDEM: FOSDEM is _the_ yearly European free and open source software event where people from around the globe meet and discuss. Aside from the main track FOSDEM consists mainly of many devrooms which focus on certain topics. I wanted to quickly highlight a few of the relevant devrooms for the OpenStack audience and would like to encourage to submit talks :) An overview of the devrooms can be seen here: https://fosdem.org/2025/schedule/ The CfP for the room is linked from the details page of each devroom. Virtualization and Cloud Infrastructure: https://lists.fosdem.org/pipermail/fosdem/2024q4/003574.html Confidential Computing: https://falder.org/fosdem25-cfp Software Defined Storage: https://lists.fosdem.org/pipermail/fosdem/2024q4/003563.html Monitoring and Observability: https://docs.google.com/document/d/1KQmNqzN3cO29ZZUSpd83REvJY_QhbvxIYr4IRBr… The SCS projects organizes, together with the german ZenDiS and the french DINUM, a room with the topic ‘Government Collaboration’. As part of the room we would love to receive proposals that cover: - Standardization and certification of digital sovereign solutions and infrastructure - Automating deployment and operations of such solutions on sovereign infrastructure - Examples and success stories of digital sovereign cloud computing by the public sector The CfP for this room is here: https://lists.fosdem.org/pipermail/fosdem/2024q4/003608.html Am sure there are more relevant devrooms :) An easy overview is also the mailing list archive: https://lists.fosdem.org/pipermail/fosdem/2024q4/thread.html regards felix [1] <url: https://fosdem.org/2025/>

1 0

Question about Package Inclusion and User Creation in Kolla Base Image
by Amir Hossein Ahmadi 06 Nov '24

06 Nov '24

Hello OpenStack Community, I'm currently reading through the OpenStack Kolla source code, and I noticed a few design choices in the kolla/base image that I’d like to understand better: - The kolla/base image includes a number of packages (such as Apache) that don’t appear to be necessary for most containers. - Additionally, user and group accounts for all services are created in the base image, even though they aren’t used by all containers. Could someone explain the reasoning behind including these components in the base image? Is there any documentation available that provides insight into the design approach for Kolla images? Thank you in advance for your help and guidance. Best regards, Amir Hossein, Burna innovation Bridge

1 0

[publiccloud-sig] Reminder: next meeting November 6th (tomorrow) - 0800 UTC
by Felix Kronlage-Dammers 05 Nov '24

05 Nov '24

Hi everyone, This Wednesday, tomorrow, (November 6th) the next meeting of the publiccloud-sig is going to happen. We’re going to do a quick re-cap from PTG as well as the ALASCA Summit that happened last week. Furthermore I’d like to see wether we can spark some activity around the discoverability service that we’ve been thinking and discussing about in the past. We’ll be meeting at 0800 UTC here: https://meetpad.opendev.org/publiccloud-sig <https://meetpad.opendev.org/publiccloud-sig> See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG <https://wiki.openstack.org/wiki/PublicCloudSIG> See you tomorrow! felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

debugging eventlet services in vscode.
by Sean Mooney 05 Nov '24

05 Nov '24

hi folks i recently had to debug a problematic interaction between eventlet and a native thread in watcher. while i probably could have printed my way though that eventually i decided to try and get debugging working in vscode instead. in this case i happened to be looking at watcher but this should apply to any eventlet service installed with devstack. in my .vscode directoy i created a launch.json that looks like this. { // Use IntelliSense to learn about possible attributes. // Hover to view descriptions of existing attributes. // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 "version": "0.2.0", "configurations": [ { "name": "Python Debugger: watcher-applier", "type": "debugpy", "request": "launch", "program": "/opt/stack/data/venv/bin/watcher-applier", "console": "integratedTerminal", "args": [ "--config-file", "/etc/watcher/watcher.conf" ], "gevent": true, "env" : { "PYTEST_CURRENT_TEST": "true" } }, { "name": "Python Debugger: watcher-decision-engine", "type": "debugpy", "request": "launch", "program": "/opt/stack/data/venv/bin/watcher-decision-engine", "console": "integratedTerminal", "args": [ "--config-file", "/etc/watcher/watcher.conf" ], "gevent": true, "env" : { "PYTEST_CURRENT_TEST": "true" } } ] } there are a few odd things about this config. first the program i am launching is the console script run in the devstack@* systemd service file. you can find what that is using systemctl show devstack@<name> to make this work without conflict do `sudo systemctl stop devstack@<name>` first. after your done debugging you can just start the systemd service again. second when using vscode to view the source of a project select the devstack global virtual env as the project virtual env. third im telling vscode to enable gevent support. now this will raise a warning/error as gevent is not installed but you can ignore that. for those of you not familiar with gevent its a fork of eventlet that actually has much better integration with debuggers due to its wider usage outside of openstack. eventlet and gevent both use greenlet and the eventlet implementation is close enough that when you enable gevent debugging it mostly works. forth to fix the "mostly" part, specifically the fact that dajngo will explode on import, we set "PYTEST_CURRENT_TEST": "true" in the terminal env. PYTEST_CURRENT_TEST is an undocumented environment variable intended only for unit testing that is check by eventlet https://github.com/eventlet/eventlet/blob/8637820f468268ffb0b8504561ea4772d… when we set that to true we skip this `isinstance(o, rlock_type)` check https://github.com/eventlet/eventlet/blob/8637820f468268ffb0b8504561ea4772d… which crashes the debugpy debugger because it tries to load setting form the danjgo module which get loaded into the python interpreter as a side effect of debugpy. i have not tried this on other project yet but it enables me to single step debug, run to breakpoints and many of the normal debug action like using the debug console to inspect the current function scope. all of the above worked without any code change to watcher or running in a specal debug mode in the service code. we occasionally get questions on this list about how to debug openstack services and i usually respond with "its possible to do, is often fragile and need a lot of context to do right" in this case it seam to work reasonable ok in the limited usage i have made of it so far. given the current incantation is pretty simple and non invasive i decided to share this in case others find it useful. keep in mind however most service are distributed systems so if you leave it stopped on a break point for 5 mins while you inspect things, what ever you were originally doing will probably time out and it might fail in odd ways but for local development that may still be fine. please don't try this in production. i have not tried this with a wsgi app. that probably will work fine with the eventlet wsgi web-server via the console scripts i.e. nova-api binary but probably wont work for uwsig/Apache. regards sean

2 1

[cinder][PTG] 2025.1 Epoxy PTG summary
by Jon Bernard 05 Nov '24

05 Nov '24

Hello everyone, thanks for a very productive PTG for the Epoxy cycle. Below is a brief summary of the highlights. The raw etherpad notes are here [1] and both the summary and transcribed etherpad notes are captured in our wiki here [2]. # Retrospective We spent the first few minutes discussing the state the Dalmatian release, what went well, not so well, and how we can improve in the Epoxy release cycle. The most notable point raised is that we have two less cores this cycle. This adds additional strain to an already sizable review queue. We will be looking to cultivate and elevate any qualified contributors if the opportunity arises. # NetApp We covered 6 different topics related to NetApp's feature development plans for the Epoxy cycle in an effort to make sure we're all aligned. Below is a summary of the full set: * Certificate-based authentication Right now only basic auth is implemented, which requires a username and password. There is a strong desire to add support for certificate-based auth. Discussions covered possible approaches and a few helpful links to existing implementations that can be used as a reference. * HA Active-Active support for NVMe driver This will allow users to configure NVMe driver backends in cinder clustered environments. We covered the failover process and which methods needed to be implemented. The related use cases were discussed as well as testing scenarios. We do not have replication and failover coverage in our upstream gate, so special care is needed in testing and verification on the developer side at the moment. Mention of Rally for testing as a possibility. * SVM scoping support for NVMe Desire to add support to the NVMe driver to SVM scoping. At the moment the driver fails to initialize. Work is ongoing to add full support for this feature. Nothing concerning was raised, looking forward to this for Epoxy. * ASA r2 support ASA r2 systems use storage units and consistency groups to simplify storage management and data protection. The REST API is different, and some features are restricted. We'll see how this unfolds in this cycle, good to have this on the radar as we go forward. * Replication - Snapmirror Sync Support Synchronous mirroring achieves a recovery point objective (RPO) of zero lost data by having a copy of important data available if a disaster happens on one of the two storage arrays. The copy is identical to production data at every moment because each time a write is done to the primary volume, a write is done to the secondary volume. The host does not receive an acknowledgment that the write was successful until the secondary volume is successfully updated with the changes that were made on the primary volume. * Metrocluster Support Metro cluster is a clustering solution with synchronous replication to deliver continuous availability, immediately duplicating all of the mission-critical data on a transaction-by-transaction basis. MetroCluster configurations enhance the built-in high availability and non disruptive operations of NetApp hardware and ONTAP storage software, providing an additional layer of protection for the entire storage and host environment. A few changes beyond driver-specific updates were discussed, no blockers. # Handling MYPY Updates We support mypy in our gate jobs, but having the version unpinned causes occasional breakages. We pin hacking and similar tools, and we agreed that mypy should be included. The caveat being we need to remember to periodically unpin and update these version numbers to stay current. # New volume driver LVM+Clone Jan Horstmann proposed a derivative of our LVM driver to add support for using device mapper clone target for transparently migrating volumes. This would all an admin to leverage local storage for performance and still be able to live migrate instances across compute hosts. A POC/Demo is hopefully coming soon, no fundamental blockers were identified, very interested to see how this comes together. # Reviews for backup service It is very difficult to get reviews for cinder-backup related changes. Tobias asked if any cores can focus some time on cinder-backup specifically (an potential backports). We agreed that mention of the backup review queue and review dashboard at the weekly meeting may help raise awareness. # New Location API adoption Rajat proposed a patch that adds the call for the `add_image_location` method which triggers the new location API workflow that is more secure and robust than the old location workflow. This call is made when glance is using cinder as a backend and we want to perform an optimized "upload volume to image" operation. Additional patches are up that add support for testing this feature. We may still need some input from the Glance team, to be followed up on in our weekly meeting. # Adding `service` query parameter to AZ API Tobias proposed an API change, adding a query parameter to an existing API. Specifically, adding `service` query parameter to os-availability-zone API. The patch is thought to be complete and ready for review. # Adding a backup summary API Tobias proposed adding a backup summary API (same way that is available for volumes). He also found a volume summary API bug when implementing this feature. The patch is ready for review. # Horizon Feature Gap We had the privilege to meet with the Horizon team and discuss their efforts on closing existing feature gaps. Several specifics were discussed and clarity was achieved. Concerns were raised over possible incompatibility issues between the openstack client and the SDK. Further, Horizon updates microversion only when necessary and value was identified in reviewing microversions to see if any additional functionality can be leveraged. # Optimize RBD update from Cinder to Glance Currently the path to upload a cinder volume to glance image is not optimized when cinder and glance are both using RBD. We can leverage RBD layering (or copy-on-write cloning) to complete the operation in seconds irrespective of the volume size (since no actual copy is happening). There are 3 features implemented that were needed to support this feature: 1. RBD clone dependency deletion: We have landed changes in glance and cinder that handle the RBD dependency chains so dependency from volumes to images pools shouldn't be an issue 2. Service role: Currently the attachment delete API already uses service role + service token and we can use similar changes for this feature 3. New location APIs: This new glance feature eliminates the security risk as we will be calculating the image hash even in this optimized path (in background so our operation performance won't be affected) * Previously several blockers, appear to be all resolved now * Significant performance increase * Glance -> Cinder already supported * Fallback to unoptimized codepath * Related, recent patches to improve RBD cinder/glance, need to be considered/reviewed Eric has committed to help with the reviews. # State of volume replication Some time was spent discussing the current state of replication. We do not have testing coverage in tempest, so the current state of any particular driver implementation is difficult to know. An idea was raised to poll vendors for use cases and testing results. There is a general lack of clarity for this feature across projects and what our overall disaster recovery solution is. # Image Encryption (Glance Cross-Project) The goal with this effort is to standardize encrypted images between Glance, Cinder and Nova, in an effort to make them consistent and accessible to users. Here is a recap: 1. We merged a patchset in os-brick so far that extends key handling to support more types of Barbican keys for image encryption in addition to Cinder's existing implementation (which strictly uses binascii.hexlify() conversion). 2. Patchset for Glance has been written to add Secret Consumer API from Barbican for encrypted images and to handle the new standardized format. 3. Patchset for Cinder has been written, making it able to produce and use the new standardized format + additionally support qcow2+LUKS in addition to raw LUKS format as well as non-hexlified keys from Barbican. After the initial discussion with Glance, we agreed on *not* introducing any new container_format or disk_format and instead handle this with `os_encrypt_*` properties instead (e.g. os_encrypt_format=luks); this is how the patchsets are currently implemented. However, due to the recent CVEs around image format parsing and conversion, valid concerns were raised that this approach might lead to new vulnerabilities due to potential conflicts and ambiguities between container_format, disk_format and the encryption attributes. We need to agree on a new way of handling the classification of encrypted images in the metadata. We have at least two cases to consider: 1. Raw LUKS (as currently produced by Cinder from encrypted volumes) 2. qcow2+LUKS (as produced by Nova for Ephemeral Storage); both could also be produced by the user (see the gist link in the reference section). # Volume type metadata progress We received a progress update on the volume type metadata effort. Consensus that the direction is sound. A new API endpoint in addition to changes to the response of volume type show / create will require a new microversion. The tempest test still need to be adjusted. Patches need review. # Status of OpenAPI effort Very little traction on initial patches, needs help getting reviews for the first set of patches. Patches also cleanup technical debt and fix several bugs, would be very good to have. The next set of patches is more meaningful, but being held until initial reviews are available. # Removal of uWSGI scripts (stephenfin) The patch to remove uWSGI scripts has been up and needs review. Relatively small patch. Update: this has been reviewed and merged. # Eventlet A dedicated etherpad was created to help document current status. Cinder is still in the investigatory phase. Not only do we use eventlet directly, some drivers such as RBD use C libraries that spawn native threads. We must take care in how we handle these cases. Our goal is only preparatory changes in Epoxy (because it is a SLURP release), and hold big changes for F (which most people won't upgrade to, so will give us more time to detect & fix issues) [1] https://etherpad.opendev.org/p/epoxy-ptg-cinder [2] https://wiki.openstack.org/wiki/CinderEpoxyPTGSummary -- Jon

1 0

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-21)
by Goutham Pacha Ravi 04 Nov '24

04 Nov '24

Hello Stackers, Thanks to all moderators and volunteers for compiling virtual PTG notes and sharing summaries through this mailing list. A lot of exciting work seems to be tracked through the action items from these summaries. I hope you were able to review the TC's PTG summary [1]. Next week is milestone-1 of the 2025.1 "Epoxy" release cycle [2]. This is typically a bug-targeting milestone for project teams. The following is a summary of this past week's happenings within the OpenStack Technical Committee. === Weekly Meeting === During the past week, the OpenStack Technical Committee met over IRC [3]. We followed up on action items from weeks preceding the Project Teams Gathering. We finalized PTL and DPL appointments for the "mistral" and "watcher" project teams. We spent some time reviewing action items related to translating documentation and user strings within OpenStack services. Automated jobs posting translation updates from Zanata are broken due to incompatibility with newer Python versions. Migration to Weblate remains a priority and the i18n team may require more support to complete this migration. We're collaborating over IRC to identify the needs and next steps. The TC also discussed progress on the community goal to migrate integration testing to Python 3.12 and to the latest LTS version of the operating system used in DevStack test jobs. Like me, you may have wondered why we’re combining these efforts. The reason is that it allows us to have a simpler test matrix despite the number of Python versions and OS distributions that OpenStack software is tested with. While we maintain unit test jobs for each supported Python version, it is impractical to duplicate each integration job for every OS distribution available. Therefore, we choose the latest LTS (e.g., Ubuntu 24.04 LTS / "Noble Numbat"), which includes the upper bound of the supported Python versions (Python 3.12). Additionally, each project will have at least one test job for the minimum supported Python version (Python 3.9), so you'll see a test job for the previous LTS release of the OS that will begin in this release and continue running as it transitions to a "stable" release branch. The next meeting of the OpenStack Technical Committee is on 2024-11-05 (today) at 1800 UTC. This is our monthly video meeting too! This meeting will occur simultaneously on Video (Zoom) and IRC (OFTC's #openstack-tc channel). Allison Price (aprice) and Jimmy McArthur (jimmymcarthur) from the OpenStack VMware Migration Working Group [4] will be joining us to discuss the working group's efforts. Meeting information, including other agenda items, is on the meeting wiki [5]. I hope you'll be able to join us. Please remember that you can always propose items to the TC's weekly meeting agenda. === Governance Proposals === ==== Merged ==== Add Watcher DPL for Epoxy | https://review.opendev.org/c/openstack/governance/+/933018 Adding Axel Vanzaghi as PTL for Mistral | https://review.opendev.org/c/openstack/governance/+/927962 ==== Open for Review ==== Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/931254 === How to Contact the TC === You can reach the TC in several ways: 1. **Email:** Send an email with the tag [tc] on this mailing list. 2. **IRC:** Use the 'tc-members' keyword on the #openstack-tc IRC channel on OFTC. 3. **Weekly Meeting:** Join us every Tuesday at 1800 UTC [5]. Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" TC PTG Summary: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [2] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [3] TC Meeting IRC Log 2024-10-29: https://meetings.opendev.org/meetings/tc/2024/tc.2024-10-29-18.00.log.html [4] OpenStack VMware Migration Group: https://www.openstack.org/vmware-migration-to-openstack/ [5] TC Meeting Agenda, 2024-11-05: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

Re: GPU PCI passthrough woes.
by Andy Speagle 04 Nov '24

04 Nov '24

Aye, that seems to have done it. Strange that this wasn't necessary for my old 1080 cards. Thanks! On Wed, 2024-10-30 at 08:12 +0000, Alexander Dibbo - STFC UKRI wrote: > ⚠️CAUTION⚠️ > This email originated outside of Toyon's network. Do not download > attachments or click links unless you know the content is safe. > > Hi Andy, > > We've seen many issues with pci passthrough and I suspect that the > l40s will have a seen similar issues with the a100s > > Here is an alias string that works for our a100s: > `alias = {"vendor_id": "10de", "product_id": "20f1", "device_type": > "type-PF", "name": "nvidia-tesla-a100-pcie-vga"}` > > I believe it is the `device_type` that will be your issue > > Regards > > Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader > For STFC Cloud Documentation visit > https://urldefense.us/v3/__https://stfc.atlassian.net/wiki/spaces/CLOUDKB/o… > To raise a support ticket with the cloud team please email > cloud-support(a)stfc.ac.uk > To receive notifications about the service please subscribe to our > mailing list at: > https://urldefense.us/v3/__https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=S… > To receive fast notifications or to discuss usage of the cloud please > join our Slack: > https://urldefense.us/v3/__https://stfc-cloud.slack.com/__;!!A4KLn-w15EM4BR… > > -----Original Message----- > From: Andy Speagle <aspeagle(a)toyon.com> > Sent: 30 October 2024 02:15 > To: openstack-discuss(a)lists.openstack.org > Subject: GPU PCI passthrough woes. > > Hey Folks, > > I could use a little assistance getting GPU passthrough working. I > had > this working already for one flavor of nvidia gpu... and I've added > some hosts with a much newer gpu... > > I've updated the pci_alias and pci_passthrough variables and those > seem > to be getting set properly in nova.conf > > passthrough_whitelist = [{"vendor_id":"10de", > "product_id":"1b06"},{"vendor_id":"10de", "product_id":"26b9"}] > > alias = {"name": "gpu", "product_id": "1b06", "vendor_id": "10de"} > alias = {"name": "gpu-l40s", "product_id": "26b9", "vendor_id": > "10de"} > > I believe I have all of the iommu stuff configured and have the pci- > stub module entries... dmesg output shows that the GPUs are being > claimed by the stub module. > > $ openstack flavor show t1.small_gpu_l40s > +----------------------------+--------------------------------------+ > > Field | Value | > +----------------------------+--------------------------------------+ > > OS-FLV-DISABLED:disabled | False | > > OS-FLV-EXT-DATA:ephemeral | 0 | > > access_project_ids | None | > > description | None | > > disk | 0 | > > id | af70c94e-0026-4a39-bc1e-dfb93b286a54 | > > name | t1.small_gpu_l40s | > > os-flavor-access:is_public | True | > > properties | pci_passthrough:alias='gpu-l40s:1' | > > ram | 2048 | > > rxtx_factor | 1.0 | > > swap | 0 | > > vcpus | 1 | > +----------------------------+--------------------------------------+ > > Yet... I can't seem to get an instance to run using that new > flavor... > keeps complaining about there not being enough hosts available. > > Fault: > code: 500 > created: 2024-10-30T02:04:33Z > message: "No valid host was found. There are not enough hosts > available." > details: | > Traceback (most recent call last): > File "/usr/lib/python3/dist- > packages/nova/conductor/manager.py", > line 1580, in schedule_and_build_instances > host_lists = self._schedule_instances(context, > request_specs[0], > File "/usr/lib/python3/dist- > packages/nova/conductor/manager.py", > line 940, in _schedule_instances > host_lists = self.query_client.select_destinations( > File "/usr/lib/python3/dist- > packages/nova/scheduler/client/query.py", line 41, in > select_destinations > return self.scheduler_rpcapi.select_destinations(context, > spec_obj, > File "/usr/lib/python3/dist-packages/nova/scheduler/rpcapi.py", > line 160, in select_destinations > return cctxt.call(ctxt, 'select_destinations', **msg_args) > File "/usr/lib/python3/dist- > packages/oslo_messaging/rpc/client.py", line 189, in call > result = self.transport._send( > File "/usr/lib/python3/dist- > packages/oslo_messaging/transport.py", line 123, in _send > return self._driver.send(target, ctxt, message, > File "/usr/lib/python3/dist- > packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send > return self._send(target, ctxt, message, wait_for_reply, > timeout, > File "/usr/lib/python3/dist- > packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send > raise result > nova.exception_Remote.NoValidHost_Remote: No valid host was > found. > There are not enough hosts available. > > Traceback (most recent call last): > File "/usr/lib/python3/dist- > packages/oslo_messaging/rpc/server.py", line 241, in inner > return func(*args, **kwargs) > > File "/usr/lib/python3/dist- > packages/nova/scheduler/manager.py", > line 223, in select_destinations > selections = self._select_destinations( > > File "/usr/lib/python3/dist- > packages/nova/scheduler/manager.py", > line 250, in _select_destinations > selections = self._schedule( > > File "/usr/lib/python3/dist- > packages/nova/scheduler/manager.py", > line 416, in _schedule > self._ensure_sufficient_hosts( > > File "/usr/lib/python3/dist- > packages/nova/scheduler/manager.py", > line 455, in _ensure_sufficient_hosts > raise exception.NoValidHost(reason=reason) > > nova.exception.NoValidHost: No valid host was found. There are > not > enough hosts available. > > Any clues on how/where to dig into this more to see what might be > missing? Thanks. > > -- > Andy Speagle > -- Andy Speagle Sr. Site Reliability Engineer Toyon Research Corporation 316.617.2431

3 2

Is HPET recommended for x86 instance?
by Jiatong Shen 04 Nov '24

04 Nov '24

Hello Community Experts, I found that to enable the HPET timer for x86 guests, the operator should set hw_time_hpet on the given image. By default, HPET is disabled. My question is what are the pros and cons for using HPET? Thanks. -- Best Regards, Jiatong Shen

2 3

[cloudbase-init] This project doesn't seem to be well managed.
by dogyun kim 04 Nov '24

04 Nov '24

Hi, I wanted to contribute to that project, so we created a CR. https://review.opendev.org/c/x/cloudbase-init/+/933629 But no one managed this project, so i can't get review. If you have an administrator, I would like to ask you to confirm it. Thanks

2 1

[cloudbase-init] This project doesn't seem to be well managed.
by 김도균 04 Nov '24

04 Nov '24

3 2

[keystone] Weekly Meeting Canceled 06-Nov-2024
by Dave Wilde 04 Nov '24

04 Nov '24

Hello all, Due to some scheduling conflicts I have to cancel this weeks Keystone meeting. Please feel free to add to the agenda [0] or reach out in #openstack-keystone if you need anything from us. Thanks, /Dave 0: https://etherpad.opendev.org/p/keystone-weekly-meeting

1 0

[neutron] Bug deputy report (week starting on October 28)
by Bernard Cafarelli 04 Nov '24

04 Nov '24

Hey neutrinos, here is the list of bugs reported during 2024 week 44 While this was Halloween week, the number of reported bugs was not too scary, and bugs have patches or assignee High * [OVN] Revision number registers must be filtered by resource ID and type - https://bugs.launchpad.net/neutron/+bug/2085946 Patch by Rodolfo https://review.opendev.org/c/openstack/neutron/+/933752 W+1 Medium * ´openstack vpn service show´ command doesnt return external_v4_ip and external_v6_ip - https://bugs.launchpad.net/python-neutronclient/+bug/2086144 CLI issue for VPNaaS, Lajos looking into it * [ovn-octavia-provider] Load balancer enable/disable has a reverse effect - https://bugs.launchpad.net/neutron/+bug/2086194 Patch proposed https://review.opendev.org/c/openstack/ovn-octavia-provider/+/933879 * [OVN] The security group create command is not creating the SG rules revision number registers - https://bugs.launchpad.net/neutron/+bug/2086205 Patch proposed https://review.opendev.org/c/openstack/neutron/+/933969 Low * [neutron-tempest-plugin] Trunk tests are running out of VLAN IDs when creating a VLAN network - https://bugs.launchpad.net/neutron/+bug/2086602 Patch proposed https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/934027 Wishlist * Reboot breaks VPN connection (OVN) - https://bugs.launchpad.net/neutron/+bug/2086189 Marked as wishlist for VPNaaS, as this is for redundancy on reboot -- Bernard Cafarelli

1 0

[openstack-ansible][ptg] Epoxy Project Team Gathering summary for OpenStack-Ansible
by Dmitriy Rabotyagov 04 Nov '24

04 Nov '24

Hi folks, Despite low participation in the PTG this time, we still had quite productive discussions around a variety of topics. You can find some summary on our future plans below: * We agreed to proceed with upgrading RabbitMQ to version 4.0 for Dalmatian. This means that HA (replicated) Queues won't be available anymore so Caracal will be the only release having support for both HA Queues and Quorum Queues. Deployments are expected to accomplish their migration to Quorum Queues (or just Classic Queues v2) on Caracal before proceeding further. * Implementing unified limits At this point of time, support for unified limits across services is not clear enough for us as well as how from operator perspective a mix of unified limits and classic quota will look like, as these 2 options are very distinct from each other. While one use case for their implementation would be vGPUs, we need an interested party for implementing them at this point. * Reviving Watcher support We have retired the Watcher role in Stein. With the project itself getting more traction these days, we agreed to proceed with reviving the role code to re-add support for the Watcher. * uWSGI and eventlet replacement A big topic throughout this PTG was replacement of eventlet with more native async approaches. However, uWSGI has no support for any of them right now. Since uWSGI is also in a maintenance mode, there is no expectations for adding async support to the project either. With that we need to start looking at alternatives, like uvicorn, which is already used in Skyline role. With that we potentially want to re-purpose the uWSGI role to be more generic at serving WSGI, meaning we will look into implementing service selection (uwsgi/gunicorn/uvicorn/etc) and renaming the role to address the more generic nature of it. * PKI role improvements This was another big topic we have discussed this time. There are multiple issues we need to address with the role. It could be split into multiple sub-topics: - Improving CA private key storage method on the deploy host to be able to store them encrypted with ansible-vault/sops or some other lookup mechanism - We need to implement better way for creating and storing private keys for generated certificates as transferring them from the deploy host is not really a best practice. - Adding support for hashicorp vault to store it in an encrypted manner. With that we should also add support to have CA in hashi-vault directly [1] and issuing certs from there. - Improve certs rollout process to allow short-term certificates to be issued and rotated regularly. This includes dynamically reloading libvirt config to address VNC consoles and reducing runtime for certificate rotation. * Would be great to have finally another round on ProxySQL efforts * Document of making user_secrets not a plain-texted file. Also address upgrade script failure in case user_secrets contain ansible-vault/sops/etc rather than plain text. * Document 2 ways of doing "pretty endpoints" - domain based and subpath based. * We need to work closely on deprecation of gather_subset starting with ansible-core 2.18, as there is no straightforward replacement. * Some repositories are missing deb822_repository migration, ie: frrouting, zun, qdrouterd [1] https://developer.hashicorp.com/vault/tutorials/secrets-management/pki-engi…

1 0

[neutron][ovn-octavia-provider][octavia][ovn] Octavia OVN DBs sync cmd
by Rico Lin 04 Nov '24

04 Nov '24

Hi all I would like to raise attention to the patch set that introduces Octavia OVN DBs sync cmd. patches: https://review.opendev.org/q/topic:%22add-ovn-db-sync-utils%22 With command `octavia-ovn-db-sync-util {--debug}` It should allows us to do the OVN DB sync similar to what `neutron-ovn-db-sync-util` does. And thanks to Fernando Royo, the patches are small and earlier to review in focused functions. Would be great if we could have some more reviews for it. *Rico Lin*

2 1

[tacker][ptg] 2025.1 Epoxy vPTG summary
by Yasufumi Ogawa 01 Nov '24

01 Nov '24

Hi tacker team, Thank you for joining the vPTG in the last week. I'd like to share the summary of our two days discussion for the proposals on the etherpad[1]. ## [recap] Revise FTs(yasufum) We have revised functional test from Bobcat and it's a recap session for looking back the remained issues. All the issues are going to be fixed in Epoxy. * Revise directory structure * Names of functions, classes and files too long for maintenance to be done this release * Combine scenarios into one to reduce the number of tests if possible. In process * Reduce setting up infra for the tests to be done this release * Revising scenarios and reducing setups are going to be started in this release. ## Plan for Improvement FT in Epoxy release (yasuum) We discussed the items about strategies for improving functional tests. To complete the goal of this proposal, we will take several steps to focus on dominant ones. (1) Divide time consuming tests, (2) Move tests less frequently updated to non-voting (3) Drop tests almost not updated from zuul (4) Rename each test scinarios and revise directory structure. We agreed with this proposal, but which of tests will be divided or dropped was not decided in the discussion. So, continue the discussion in the next IRC meeting. ## Add option to enable/disable Cinder volume recreation in v1 Heal request (koba) In the current v1 Heal request, the related Cinder Volume for the specified VDU is recreated although it can be specified not to recreate it. To support it in v1, Clients specify a flag in the additional_params to indicate whether to recreate the volume or not when making a Heal request. * Tacker conductor checks this flag to decide whether to mark the storage resources when executing STACK_UPDATE. Heal process works by recreating the resources marked "mark-unhealthy" during STACK_UPDATE. *Consider ensuring version compatibility and consistency with v2 operations. * Default behavior will be the same as now (recreating the Cinde Volume). * The name of the flag parameter in the additonal_params can be configured by the admin in tacker.conf. ## Add default_secret_key Option to [vim_keys] for Multi-Master Tacker Deployment (koba) ## Enhancement of the Ansible Driver (sample mgmt driver) (koba) Tacker currently supports only a single version of Ansible. As a result, when tenants include Playbooks in the VNF-Package that only work with specific versions of Ansible, there may be compatibility problems with the Playbooks. Tacker administrators must modify the Ansible Driver’s source code directly if they want to enforce options related to ansible-playbook execution, such as specifying log storage locations or custom callback plugins. These issues lack flexibility for both tenants and administrators. We agreed the proposed approaches for supporting several versions of ansible, but still not sure it's appropriate to define all the supported versions in tacker.conf and select which version will be applied by specifying it in the definition file in VNF package. It going to be discussed the way of selecting version in the next meeting. ## Remove legacy engine facade (yasufum) It's one of the community goal, so we'll fix it in this release. ## Remove sphinxcontrib-*diag from your (docs) dependencies (yasufum) It's similar to the previous item, and fix it in this release. ## Ensuring Tacker Compatibility with TST and SOL Specifications (Renu Rani) Agreed to the approach proposed on the etherpad. * Addressing the Gap Between TST and SOL Specifications: Raise bugs on TST and request the TST team to update schemas and variables according to SOL specifications. * Support of Compliance Tests for Tacker V2 a. Resolving Issues with Interconnected Test Cases in Tacker: Update Tacker's code to ensure compatibility with the interconnected test cases in v4.5.1-dev. b. Supporting Tacker for TST Release-4: Implement new Tacker compliance test cases for TST Release-4. c. Validating Compliance Test Cases with Tacker V2: Seek the Tacker team's opinion on whether to use Release 3 or Release 4 for validation. ## Enhancing Tacker Testing Coverage in the OpenStack Helm Check Pipeline (Shivam Shukla) To further enhance Tacker testing within the OpenStack Helm Check pipeline by adding more testcases to ensure thorough verification of Tacker deployments. Although the test scenarios were proposed, but we were not sure all the scenario are really needed because some of them are existing scenarios. We agreed to accept the proposal itself, but still not all the scenario should be introduced. So, continue to discuss for that. 1. VIM management (create, update and delete VIMs) 2. VNFD management (create and delete VNF packages) 3. VNF Instance management (create, update and delete VNF instances) 4. VNF LCM operation (instantiate, terminate, heal and scale) 5. Subscription management (create and delete subscription for VNFs) [1] https://etherpad.opendev.org/p/oct2024-ptg-tacker

1 0

[ironic] Bare Metal SIG
by Julia Kreger 01 Nov '24

01 Nov '24

Greetings folks, During this past week's PTG, the Ironic community participants reached consensus to wrap up the Bare Metal SIG. There are many reasons, but the bulk of the reasoning revolves around spreading word and collaborating outward as opposed to focusing on our own vertical of interaction. Having discussions in different contexts helps bring strength to discussions and we encourage contributors to engage in adjacent community and group discussions where relevant then build connections between communities. Communities, when working together, are always stronger when we build these sorts of connections or bonds. We hope to continue having meetups with operators as we always have, even if it may not bear the Bare Metal SIG name. As always, operators of any bare metal provisioning software are welcome to join our discussions at any time in OFTC IRC #openstack-ironic or right here on the mailing list. Among the accomplishments of the Baremetal SIG, was a whitepaper[0] devoted to the Ironic use case. Special thanks goes to Chris Hoge, previously with the Open Infra Foundation who helped drive us to the finish line on that effort. Beyond the whitepaper, and likely even harder was to bring people together in person on multiple occasions! A special thank goes to Arne Wiebalck who championed the Bare Metal SIG and helped steward several meetups including the most recent meetup at CERN in late Spring of this year. The success of that event speaks volumes about our community and the power of collaboration! Many thanks also goes to everyone who helped with the videos we have posted to youtube[1]! Without everyone pitching in, we wouldn't have been able to achieve success! -Julia, on behalf of the Ironic team. [0]: https://www.openstack.org/use-cases/bare-metal/how-ironic-delivers-abstract… [1]: https://www.youtube.com/playlist?list=PLKqaoAnDyfgoBFAjUvZGjKXQjogWZBLL_

1 0

[devstack][nova][cinder][ironic][glance][swift][neutron][all] Deprecating/removing non-uWSGI deployment mechanisms?
by Stephen Finucane 01 Nov '24

01 Nov '24

o/ tl;dr: I'd like to remove the ability to deploy services under mod_wsgi from DevStack. I'd also like to remove the ability to deploy services under eventlet, where this feature is not currently being tested, and deprecate it where it is. I took a foray through DevStack last week and realised that, depending on the service in question, there are currently 3 possible ways an API can be deployed by DevStack: * Eventlet server * Apache + uWSGI * Apache + mod_wsgi In addition, there are a number of overlapping config options that could be set to modify which of these was in use: * ENABLE_HTTPD_MOD_WSGI_SERVICES * WSGI_MODE * Service-specific options like NOVA_USE_MOD_WSGI, CINDER_USE_MOD_WSGI etc. Potential confusion aside, there are two issues I see with this. Firstly, and most obviously, we are working on eventlet removal, which will necessarily involve eventlet server removal [1]. In addition, there is a goal [2][3] to remove use of '.wsgi' scripts in favour of Python module paths. Quite a few services have migrated to this already, and those that haven't are going to need to do so by time pip 25.0 comes out (since that removes the deprecated legacy editable install flow [4] meaning these files will not be generated by pbr and DevStack deployments will start failing...you have been warned). However, mod_wsgi need .wsgi scripts. I didn't realise we cared about mod_wsgi in DevStack still before last week, so to complete the wsgi script -> module migration for services that can be deployed with mod_wsgi (e.g. Keystone), we're going to have to either (a) package a .wsgi script in DevStack or (b) remove the ability to deploy with mod_wsgi. My immediate thought on this was to simply remove all non-uWSGI deployment paths immediately and deprecate the config options for removal to force DevStack plugins to adapt. However, some services still want to test eventlet server for their API while they stabilise support for standard WSGI-based deployments (neutron), while others explicitly rely on evenlet to generate a single standalone binary (ironic, glance, ...). That means we can't just go remove everything now and must instead take a phased approach. I would therefore like to propose we do the following: * Remove the ability to deploy services under mod_wsgi from DevStack and deprecate/remove the WSGI_MODE flag. To be clear, other deployment tools can continue to use mod_wsgi but DevStack won't. * Entirely remove the ability to deploy under eventlet from services that are not currently testing this. This includes Nova and Cinder. * Invert the default for services that are testing eventlet, so that they test with uWSGI by default now and jobs will need to opt-in to eventlet rather than opt-in to uWSGI. I think this just affects Neutron. * Optionally, and likely only one Neutron is no longer testing eventlet (next cycle?), deprecate ENABLE_HTTPD_MOD_WSGI_SERVICES and {SERVICE}_USE_MOD_WSGI config options entirely and replace it with {SERVICE}_STANDALONE options like Glance has. This will affect Ironic and maybe more. I have proposed a series that will do much of this. The WIP patches indicate the things that I am less clear about, such as Swift's readiness for deploying under uWSGI. I would be grateful if folks from the individual service teams could take a look and let me know what they think: https://review.opendev.org/c/openstack/devstack/+/932193/1 Cheers, Stephen [1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [2] https://review.opendev.org/c/openstack/governance/+/902807 [3] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [4] https://ichard26.github.io/blog/2024/08/whats-new-in-pip-24.2/

6 10