- openstack-discuss - lists.openstack.org

[watcher] Core team updates
by Douglas Viroel 13 Mar '26

13 Mar '26

Hi all, Following up on previous Sean's email[1], from Sep 8th 2025, regarding Watcher core team updates, I'd like to propose the following updates for 2026.2 cycle: 1. Core Removals As we start a new release cycle, the watcher core members mentioned in Sean's email for potential removal due to inactivity (*chenker[2], Dan Smith[3], Marios Andreous[4], and Slawek Kaplonski[5]*) will now be removed, effective immediately. I'd like to take this opportunity to thank all of them for their contributions to Watcher over the years. Maintaining an active core team is essential for the health of the project, and in the case of any of these contributors becoming active again in the future, we would welcome proposals to re-add them to the core team. 2. New Core Member Proposal I'd like to propose adding *Alfredo Moralejo (amoralej)* to both the *watcher-core* and *watcher-dashboard-core* groups. He has been consistently contributing to the Watcher project and has demonstrated a solid understanding of the codebase through quality reviews[6][7] and patches. Alfredo provides valuable feedback, helps with manual testing when needed, proactively file new bugs and assist on the bug triage process. I believe he would be a valuable addition to the core team. If there are no objections, the new core additions will be applied right after the next Watcher IRC meeting, next Thursday, Mar 19th, Please feel free to share any feedback or concern around these 2 proposed changes in the core team. Best regards, [1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [2] https://www.stackalytics.io/?module=watcher-group&release=all&user_id=chenk… [3] https://www.stackalytics.io/?module=watcher-group&release=all&user_id=danms [4] https://review.opendev.org/q/reviewedby:marios.andreou@gmail.com [5] https://www.stackalytics.io/?module=watcher-group&release=all&user_id=slaweq [5] https://www.stackalytics.io/?module=watcher-group&release=all&user_id=amora… [6] https://www.stackalytics.io/?module=opendev.org/openstack/watcher-dashboard… -- Douglas Viroel - dviroel

1 0

[ironic][nova][ptg] Cross-project session around nova-compute startup
by Jay Faulkner 13 Mar '26

13 Mar '26

Hey all, A few releases back we swapped up the model for nova-compute HA in Ironic -- mainly, moved it in line with other nova drivers where the loss of a single nova-compute service takes down a portion of the cloud, but HA is achieved by having other portions of the cloud still available. This model has serious limitations in an Ironic world; including but not limited to: * Use cases around rebuilding into the same physical machine can never be isolated from an outage (including upgrades) * Even with a reasonable number of nodes managed, nova-compute processes managing Ironic nodes can take a long, long time to get online. We're talking 15-20 minutes with reports of even worse in some scenarios. I'd like to improve this story. It's operationally painful to eat a multi-minute long outage for a deployment of code to a server -- and given my use case of in-place rebuilds, there's no amount of aggregation of nodes which can get us clear of this issue. It's also clear that going back to a model which would run multiple nova-compute processes to manage the same machines is too incompatible with the general nova model; so instead I think we can try to tackle a major pain point: the slow startup. Ideas I have thought about to address this include: * Improving how resources are added to placement for Ironic, doing it incrementally from Ironic may reduce the number of calls needed at startup * Improving placement API interfaces to perhaps allow bulk updates, so we aren't doing "N" calls for "N" ironic nodes at startup, as I believe we currently do * Some method of "handoff" between a staged nova-compute service and one that is shutting down, so we can avoid the outage in properly coordinated update scenarios. (I'm not sure this is possible) One thing that's clear to me: I don't have the knowledge to pursue this alone. My hope is that we can work together at the PTG to determine one or more good, potential directions, then put together a spec during the Hibiscus cycle which could be implemented for "I" (not Icehouse :D). As long as we have a commitment from nova cores to perform reviews and help with the design, I should be also to provide resources (myself + maybe others) from the GR-OSS team to help implement it. Thanks! Jay Faulkner Open Source Developer G-Research Open Source Software

1 0

[ops][keystone]Migration to Epoxy and KeyError: '$6$'
by federica fanzago 13 Mar '26

13 Mar '26

Dear all, we are running an OpenStack deployment Caracal on AlmaLinux 9.7 and we are attempting the upgrade to Epoxy release. Our cloud has been running since 2018, and over the years we have performed several release updates (Rocky-Train-Yoga-Caracal) without modifying the default value for the parameter "password_hash_algorithm" in the keystone.conf file During the update to Epoxy, Keystone shows an error in /var/log/httpd/keystone_wsgi_internal_error.log that seems related to the deprecation of older encryption algorithms 026-03-13 15:31:15.359556 ValueError: Unsupported password hashing algorithm ident: $6$ 2026-03-13 15:31:17.703201 mod_wsgi (pid=32730): Exception occurred processing WSGI script '/usr/bin/keystone-wsgi-public'. 2026-03-13 15:31:17.703935 Traceback (most recent call last): 2026-03-13 15:31:17.703966 File "/usr/lib/python3.9/site-packages/keystone/common/password_hashing.py", line 72, in _get_hasher_from_ident 2026-03-13 15:31:17.703972 return _HASHER_IDENT_MAP[hashed[0 : hashed.index('$', 1) + 1]] 2026-03-13 15:31:17.703994 KeyError: '$6$' In our db there are some passwords starting with "$6$round....". Do you any have suggestions about how to modify the encryption algorithm for these passwords to $2$? Is there a keystone script to perform this migration and fix the error? Thanks, cheers Federica -- Federica Fanzago INFN Sezione di Padova Via Marzolo, 8 35131 Padova - Italy Tel: +39 049.967.7367 --

2 1

[release] Release countdown for week R-2, Mar 16 - 20
by Elõd Illés 13 Mar '26

13 Mar '26

Development Focus ----------------- At this point we should have release candidates (RC1 or recent intermediary release) for all the 2026.1 Gazpacho deliverables. Teams should be working on any release-critical bugs that would require another RC or intermediary release before the final release. Actions ------- Early in the week, the release team will be proposing stable/2026.1 branch creation for all deliverables that have not branched yet, using the latest available 2026.1 Gazpacho release as the branch point. If your team is ready to go for creating that branch, please let us know by leaving a +1 on these patches. If you would like to wait for another release before branching, you can -1 the patch and update it later in the week with the new release you would like to use. By the end of the week the release team will merge those patches though, unless an exception is granted. Once stable/2026.1 branches are created, if a release-critical bug is detected, you will need to fix the issue in the master branch first, then backport the fix to the stable/2026.1 branch before releasing out of the stable/2026.1 branch. After all of the cycle-with-rc projects have branched we will branch devstack, grenade, and the requirements repos. This will effectively open them up for 2026.2 Hibiscus development, though the focus should still be on finishing up 2026.1 Gazpacho until the final release. Opening up the new development cycle is also a good opportunity to teams to revise their zuul jobs and used nodesets and clean up all unnecessary ones. For projects with translations, watch for any translation patches coming through and merge them quickly. A new release should be produced so that translations are included in the final 2026.1 Gazpacho release. Finally, now is a good time to finalize release notes. In particular, consider adding any relevant "prelude" content. Release notes are targetted for the downstream consumers of your project, so it would be great to include any useful information for those that are going to pick up and use or deploy the 2026.1 Gazpacho version of your project. Upcoming Deadlines & Dates -------------------------- Final RC deadline: March 26th, 2026 (R-1 week) Final 2026.1 "Gazpacho" release: April 1st, 2026 2026.2 "Hibiscus" (virtual) PTG: April 20-24, 2026 Előd Illés irc: elodilles @ #openstack-release

1 0

[ci infra] local PyPi cache/mirroring/proxy
by Radomir Dopieralski 13 Mar '26

13 Mar '26

Hello, I don't really have much more than surface experience with our CI infrastructure, but I wanted to start a discussion on a topic that maybe could be picked up by more knowledgeable colleagues. Our tests install a lot of python packages. It almost feels like more than half the time of any test run is spent installing packages. By default, all those packages are being installed from PyPi directly. I wonder if we could gain some advantages by running a local PyPi cache or mirror inside of the CI infrastructure, and making all those tests use it by default for all its package installs. It would: - speed up the downloading of packages, - make tests more resilient in the face of outside network or PyPi outages, - make us better citizens by decreasing the traffic to PyPi and conserving their resources, - give us more control over which packages are available (so, for example, we could easily pin setuptools for all tests, or ban a particular known bad package version, or even do pre-releases of our own packages before we release them on PyPi), Of course nothing is for free, and such a service would require some extra work to set up and maintain. I don't have the experience to be able to tell if the overhead is worth it, but maybe it's something worth considering? Or maybe we are already doing this, and I'm just wasting everyone's time? There is a number of ready to use tools we could utilize for this: https://packaging.python.org/en/latest/guides/index-mirrors-and-caches/#exi… What do you think? -- Radomir Dopieralski

2 2

[neutron] OVN version used in the functional gate job
by Jakub Libosvar 13 Mar '26

13 Mar '26

Hi all, I wanted to bring this topic up for a discussion again. I’ve been working on a Neutron feature [1] for some time, the feature requires the BGP support in OVN. The functional job currently uses a moving OVN branch based on the OVN LTS version - which is 24.03 - almost 2 years old. I tried to change this and requested a patch to switch to using a pinned OVN mostly for two benefits: - we won’t get gate broken if a bad patch is backported to the branch-24.03 - I get the BGP features needed for the work on [1] The patch was denied to keep the policy we set on ourselves - ie. to go with the branch based on the OVN LTS version. The workaround was to skip multiple tests that rely on the OVN changes specific to BGP. This is not ideal as now majority of the functionality is not tested and we’re just waiting for the LTS release, which may bring some surprises. This was a concrete example but if we put it into a more general situation, in Neutron we heavily rely on OVN and its features and any development of a potential feature that requires some work in OVN gets cumbersome because we can’t validate our code against OVN until the point a next OVN LTS version is released, which in the worst case scenario can take up to 2 years. Hence I propose a change and use a pinned hash from the OVN tree that is updated on two occasions: - A next OVN version is released (non-LTS) - We’re developing a feature that relies on bleeding edge OVN - then we pick a good hash and pin to it in the functional job definition (not in the scripts) This would apply solely to the functional tests because tempest jobs can define their own OVN version and unit tests should be isolated from the used OVN version, and only to the master branch - because that is the development branch after all. Such a change brings multiple benefits, we won’t need to skip tests if the required functionality is not present in current OVN version and have the development go smoother. We may uncover issues in OVN early. We will not get a broken gate if a change containing a bug is backported to OVN stable branch. At this point I can’t see any benefit of using the moving LTS branch, but if there is one please bring it up here. Any thoughts on this? Thanks, Kuba [1] https://opendev.org/openstack/neutron-specs/src/branch/master/specs/2025.2/… [2] https://review.opendev.org/c/openstack/neutron/+/961243

5 9

[kolla] Persistent `cannot fork child process` when launching any VM in all-in-one deployment
by Dennis Martin 13 Mar '26

13 Mar '26

Hi everyone, I've been debugging an issue for several weeks and have exhausted the obvious possibilities. I'm hoping the collective expertise here can point me in the right direction. ## Environment - **OpenStack**: Kolla-Ansible 2025.1 (Epoxy), all-in-one deployment - **OS**: Ubuntu 24.04 on a VM (nested virtualization) - **OKD target**: Trying to deploy OKD 4.18, but even a simple Cirros test VM fails ## The Problem Any attempt to create a VM (even `--image cirros --flavor m1.tiny`) fails with: ``` libvirt.libvirtError: cannot fork child process: Resource temporarily unavailable ``` The VM goes straight to `ERROR` state with no useful details in `nova-compute.log` beyond the same error. ## What Works ✅ - OpenStack services are up (`nova-compute`, `neutron`, etc. all healthy) - Can create networks, subnets, routers, keypairs via CLI - Direct `qemu-system-x86_64 -enable-kvm` test succeeds (KVM itself works) - `/dev/kvm` exists with correct permissions - KVM modules loaded (`kvm_intel`), nested virt enabled (`Y`) ## What We've Checked (All Fine) | Check | What We Found | Status | |-------|---------------|--------| | System PID limit (`pid_max`) | 4,194,304 | ✅ OK | | Kernel threads max (`threads-max`) | 722,077 | ✅ OK | | User process limit (`ulimit -u`) | 361,038 | ✅ OK | | Host thread count (`ps -eLf \| wc -l`) | ~111,000 | ✅ OK | | `nova_compute` container `pids.max` | 108,000 | ✅ OK | | `nova_compute` PIDs in container | 27 | ✅ OK | | `nova_libvirt` container `PidsLimit` | `<nil>` (unlimited) | ✅ OK | | AppArmor blocking libvirt | No profiles loaded | ✅ OK | | Host `libvirtd` running | Inactive | ✅ OK | | Libvirt log volume size | 12KB | ✅ OK | | RAM available | 88 GB total, plenty free | ✅ OK | | vCPUs | 32 cores | ✅ OK | ## What We Can't Check (Missing Commands) Inside the `nova_libvirt` container, the `ulimit` command is missing, so we cannot determine: - `nproc` limit inside the container - `nofile` (file descriptor) limit ## What We Haven't Checked - Kernel parameters like `vm.max_map_count` (default is 65530, could this be an issue?) - cgroup v2 limits on the `nova_libvirt` container (Ubuntu 24.04 uses cgroup v2) - `libvirtd` logs inside the container (the log file may not exist or be empty) ## The Ask Has anyone seen this in a Kolla-Ansible all-in-one deployment where all the obvious limits are large but libvirt still refuses to fork? Could it be: 1. A cgroup v2 limit we missed? 2. A kernel parameter that needs tuning (`vm.max_map_count`)? 3. Something else entirely? I'm happy to run any additional diagnostics or provide more logs. Any guidance would be hugely appreciated. Thanks, Dennis

2 1

[kolla-ansible][flamingo][vpnaas] Problem and solution
by Franck VEDEL (UGA) 13 Mar '26

13 Mar '26

2 3

Re: [Devel] OpenStack RPMs
by Neal Gompa 13 Mar '26

13 Mar '26

On Thu, Mar 12, 2026 at 8:08 PM Francesco Di Nucci via Devel <devel(a)lists.almalinux.org> wrote: > > Hi all, > > given that RDO will soon stop releasing RPMs of OpenStack, it has been > suggested on IRC etc that AlmaLinux and Rocky Linux communities might be > interested in building the RPMs instead, is there already a discussion > about it? > I think we'd generally prefer to see AlmaLinux contributors helping out in upstream RDO. If there are folks from the AlmaLinux community interested, please engage with RDO to help them. -- 真実はいつも一つ！/ Always, there's only one truth!

1 0

[kolla] Help debugging cannot fork child process when launching VMs in all-in-one deployment
by Dennis Martin 12 Mar '26

12 Mar '26

Hi Kolla team, I've been troubleshooting an issue for weeks and would really appreciate your insight. *Environment:* - Kolla-Ansible 2025.1 (Epoxy) all-in-one - Ubuntu 24.04 on VM (nested virtualization) - OpenStack services seem healthy (nova-compute, neutron, etc. all up) *The Problem:* Any attempt to create a VM (even a tiny Cirros test VM) fails with: text libvirt.libvirtError: cannot fork child process: Resource temporarily unavailable *What Works:* - ✅ Can create networks, subnets, routers, keypairs via CLI - ✅ Direct qemu-system-x86_64 -enable-kvm test succeeds - ✅ /dev/kvm exists with correct permissions - ✅ KVM modules loaded (kvm_intel), nested virt enabled (Y) *Resource Limits (all seem fine):* - RAM: 88 GB, vCPUs: 32 - pid_max: 4,194,304 - kernel.threads-max: 722,077 - ulimit -u: 361,038 - Total host threads: ~111k (well below limits) *Container-Specific Checks:* - nova_compute: pids.max=108k, Max processes=unlimited, only 27 PIDs running - nova_libvirt: No explicit PidsLimit (<nil>), logs not showing obvious errors - Libvirt volume only 12KB (no runaway logs) - Host libvirtd is inactive (no conflict) *What I've Tried:* - Recreated networks/routers multiple times - Restarted containers - Verified all endpoints are correctly set to VIP (192.168.58.50) - Checked kernel logs – no relevant errors *Full error from nova-compute log:* text libvirt.libvirtError: cannot fork child process: Resource temporarily unavailable Has anyone encountered this in an all-in-one deployment where all obvious limits are huge but libvirt still refuses to fork? Could it be a cgroup v2 limit I'm missing, or something in the libvirt container configuration? Any pointers would be hugely appreciated. Thanks, Dennis

1 0