September 2024 - openstack-discuss

[publiccloud-sig] Heads-Up: next meeting September 25th - 0700 UTC
by Felix Kronlage-Dammers 23 Sep '24

23 Sep '24

Hi everyone, We are adjusting the schedule for the publiccloud-sig and will switch to odd instead of even weeks. This Wednesday (September 25th) the next meeting of the publiccloud-sig is going to happen. Thanks to Artem (gtema) and Joern (joek-office) who helped prepare the meeting agenda for this wednesday: - Housekeeping - Upcoming vPTG - Reminder to sign up for PTG: https://openinfra.dev/ptg/ - IAM Federation - combination of functional requirements (every CSP is having unique use case so we want to find similarities and differencies to determine the roadmap for Keystone) If you’re interesting in IAM federation, be sure to join! We’ll be meeting at 0700 UTC here: https://meetpad.opendev.org/publiccloud-sig <https://meetpad.opendev.org/publiccloud-sig> See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG <https://wiki.openstack.org/wiki/PublicCloudSIG> See you on Wednesday! felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

Openstack power off/on process
by Michael Knox 23 Sep '24

23 Sep '24

Hey Folks. I am working through a scenario that requires us to have Openstack in a Pelican Case and deployed in the "field" on a semi regular basis. I guess, portal "edge cloud" if you will? The problem I am trying to clean up is the startup process. We use kolla-ansible and rocky with zed (though we have one with ubuntu and wallaby) in this specific use case. We do the shutdown nicely, kolla-ansible -i inventory stop, then power off the control nodes (we have 3 in the setup). The problem is on the start up. All the containers start and then we still *have* to run the kolla-ansible -i inventory mariadb_recovery step to get the cloud back into a usable state. No matter the order of shut down or start up, MariaDB seems to always need a recovery. Are there any people here doing this or something similar? Is there a more efficient startup process I am not aware of or should consider? The folks in the field aren't Openstack admins, so hoping to figure out a solution that is less dependent on their knowledge of Openstack's undercloud tooling. Thanks in advance Michael

2 2

[cinder][CI] Can't start zuul service
by xuq.fnstxz＠fujitsu.com 23 Sep '24

23 Sep '24

Hello every one, I'm building a CI system on my ubuntu 22 system. Now I've installed the zuul using 'pip install zuul' and configured it. But after running command 'zuul-scheduler', I saw a lot of errors like following in the log. It seems that there is something error in my config file that prevents access to the repo. > 2024-09-20 13:05:06,010 ERROR zuul.TenantParser: Error processing cat job > Traceback (most recent call last): > File "/home/stack/.local/lib/python3.10/site-packages/zuul/configloader.py", line 2157, in _cacheTenantYAML > self._processCatJob(abide, tenant, loading_errors, job, > File "/home/stack/.local/lib/python3.10/site-packages/zuul/configloader.py", line 2249, in _processCatJob > raise TimeoutError(f"Cat job {job} timed out; consider setting " > TimeoutError: Cat job <zuul.zk.event_queues.JobResultFuture object at 0x7fb770341510> timed out; consider setting merger.git_timeout in zuul.conf Following is my zuul.conf. > [zookeeper] > hosts=ZOOKEEPER_HOST > tls_cert=/cert/client-cert.pem > tls_key=/cert/client-key.pem > tls_ca=/cert/ca-cert.pem > > [database] > # Use variable interpolation to supply the password from the > # docker-compose file. > # https://zuul-ci.org/docs/zuul/latest/configuration.html > dburi=mysql+pymysql://DB_USER:DB_PASS@DB_HOST/ZUUL_DB > > [keystore] > password=secret > > [web] > listen_address=0.0.0.0 > port=9000 > root=http://localhost:9000 > log_config=/etc/zuul/logging.conf > > [webclient] > url=http://localhost:9000 > verify_ssl=false > > [scheduler] > tenant_config=/etc/zuul/main.yaml > log_config=/etc/zuul/logging.conf > [merger] > git_user_email=git_user(a)example.com > git_user_name=git_user > log_config=/etc/zuul/logging.conf > > [executor] > private_key_file=/home/stack/.ssh/id_rsa > default_username=executor_user > trusted_rw_paths=/srv/static/logs > > [connection gerrit] > driver=gerrit > server=review.opendev.org > user=git_user > sshkey=/ssh/id_rsa And the following is the tenant_config file. > - tenant: > name: myCI > source: > gerrit: > untrusted-projects: > - openstack/cinder Could you please tell me what I need to modify or add in the config file? Thanks and Regards, Inori.

2 2

[kolla] Weird upgrade procudere ?
by Michal Arbet 23 Sep '24

23 Sep '24

Hi Koalas, I am not sure but is this normal ? In rabbitmq.yml is this +++ b/ansible/rabbitmq.yml @@ -0,0 +1,115 @@ +--- +# For RabbitMQ we need to be careful about restarting services, to avoid losing quorum. +- name: Apply role rabbitmq + gather_facts: false + hosts: + - rabbitmq + - '&enable_rabbitmq_True' But I can see during the upgrade restart every node at the same moment and then ther is some try to do it better ... I mean is that restart before OK ? Thanks TASK [rabbitmq : Enable all stable feature flags] ***************************************************************************************************************************************************************** ok: [controller1] ok: [controller0] ok: [controller2] TASK [rabbitmq : Check rabbitmq containers] *********************************************************************************************************************************************************************** changed: [controller0] => (item={'key': 'rabbitmq', 'value': {'container_name': 'rabbitmq', 'group': 'rabbitmq', 'enabled': True, 'image': 'dockerhub.ultimum.io/kolla-dev/debian-rabbitmq:KB-2023.1', 'bootstrap_environment': {'KOLLA_BOOTSTRAP': None, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'volumes': ['/etc/kolla/rabbitmq/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'rabbitmq:/var/lib/rabbitmq/', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'healthcheck': {'interval': '30', 'retries': '3', 'start_period': '5', 'test': ['CMD-SHELL', 'healthcheck_rabbitmq'], 'timeout': '30'}, 'haproxy': {'rabbitmq_management': {'enabled': 'yes', 'mode': 'http', 'port': '15672', 'host_group': 'rabbitmq'}, 'rabbitmq_outward_management': {'enabled': False, 'mode': 'http', 'port': '15674', 'host_group': 'outward-rabbitmq'}, 'rabbitmq_outward_external': {'enabled': False, 'mode': 'tcp', 'external': True, 'port': '5674', 'host_group': 'outward-rabbitmq', 'frontend_tcp_extra': ['timeout client 1h'], 'backend_tcp_extra': ['timeout server 1h']}}}}) changed: [controller2] => (item={'key': 'rabbitmq', 'value': {'container_name': 'rabbitmq', 'group': 'rabbitmq', 'enabled': True, 'image': 'dockerhub.ultimum.io/kolla-dev/debian-rabbitmq:KB-2023.1', 'bootstrap_environment': {'KOLLA_BOOTSTRAP': None, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'volumes': ['/etc/kolla/rabbitmq/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'rabbitmq:/var/lib/rabbitmq/', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'healthcheck': {'interval': '30', 'retries': '3', 'start_period': '5', 'test': ['CMD-SHELL', 'healthcheck_rabbitmq'], 'timeout': '30'}, 'haproxy': {'rabbitmq_management': {'enabled': 'yes', 'mode': 'http', 'port': '15672', 'host_group': 'rabbitmq'}, 'rabbitmq_outward_management': {'enabled': False, 'mode': 'http', 'port': '15674', 'host_group': 'outward-rabbitmq'}, 'rabbitmq_outward_external': {'enabled': False, 'mode': 'tcp', 'external': True, 'port': '5674', 'host_group': 'outward-rabbitmq', 'frontend_tcp_extra': ['timeout client 1h'], 'backend_tcp_extra': ['timeout server 1h']}}}}) changed: [controller1] => (item={'key': 'rabbitmq', 'value': {'container_name': 'rabbitmq', 'group': 'rabbitmq', 'enabled': True, 'image': 'dockerhub.ultimum.io/kolla-dev/debian-rabbitmq:KB-2023.1', 'bootstrap_environment': {'KOLLA_BOOTSTRAP': None, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'volumes': ['/etc/kolla/rabbitmq/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'rabbitmq:/var/lib/rabbitmq/', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'healthcheck': {'interval': '30', 'retries': '3', 'start_period': '5', 'test': ['CMD-SHELL', 'healthcheck_rabbitmq'], 'timeout': '30'}, 'haproxy': {'rabbitmq_management': {'enabled': 'yes', 'mode': 'http', 'port': '15672', 'host_group': 'rabbitmq'}, 'rabbitmq_outward_management': {'enabled': False, 'mode': 'http', 'port': '15674', 'host_group': 'outward-rabbitmq'}, 'rabbitmq_outward_external': {'enabled': False, 'mode': 'tcp', 'external': True, 'port': '5674', 'host_group': 'outward-rabbitmq', 'frontend_tcp_extra': ['timeout client 1h'], 'backend_tcp_extra': ['timeout server 1h']}}}}) RUNNING HANDLER [rabbitmq : Restart rabbitmq container] *********************************************************************************************************************************************************** [WARNING]: Failure using method (v2_playbook_on_handler_task_start) in callback plugin (<ansible.plugins.callback.ara_default.CallbackModule object at 0x7f66542e0100>): 'NoneType' object is not subscriptable changed: [controller0] changed: [controller1] changed: [controller2] PLAY [Restart rabbitmq services] ********************************************************************************************************************************************************************************** TASK [rabbitmq : Get info on RabbitMQ container] ****************************************************************************************************************************************************************** ok: [controller0] TASK [rabbitmq : Put RabbitMQ node into maintenance mode] ********************************************************************************************************************************************************* changed: [controller0] TASK [rabbitmq : Restart rabbitmq container] ********************************************************************************************************************************************************************** changed: [controller0] TASK [rabbitmq : Waiting for rabbitmq to start] ******************************************************************************************************************************************************************* changed: [controller0] PLAY [Restart rabbitmq services] ********************************************************************************************************************************************************************************** TASK [rabbitmq : Get info on RabbitMQ container] ****************************************************************************************************************************************************************** ok: [controller1] TASK [rabbitmq : Put RabbitMQ node into maintenance mode] ********************************************************************************************************************************************************* changed: [controller1] TASK [rabbitmq : Restart rabbitmq container] ********************************************************************************************************************************************************************** changed: [controller1] TASK [rabbitmq : Waiting for rabbitmq to start] ******************************************************************************************************************************************************************* changed: [controller1] PLAY [Restart rabbitmq services] ********************************************************************************************************************************************************************************** TASK [rabbitmq : Get info on RabbitMQ container] ****************************************************************************************************************************************************************** ok: [controller2] TASK [rabbitmq : Put RabbitMQ node into maintenance mode] ********************************************************************************************************************************************************* changed: [controller2] TASK [rabbitmq : Restart rabbitmq container] ********************************************************************************************************************************************************************** changed: [controller2] TASK [rabbitmq : Waiting for rabbitmq to start] ******************************************************************************************************************************************************************* changed: [controller2] PLAY [Apply rabbitmq post-configuration] ************************************************************************************************************************************************************************** TASK [Include rabbitmq post-deploy.yml] *************************************************************************************************************************************************************************** TASK [rabbitmq : Enable all stable feature flags] ************************* Michal Arbet Openstack Engineer Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic +420 604 228 897 michal.arbet(a)ultimum.io *https://ultimum.io <https://ultimum.io/>* LinkedIn <https://www.linkedin.com/company/ultimum-technologies> | Twitter <https://twitter.com/ultimumtech> | Facebook <https://www.facebook.com/ultimumtechnologies/timeline>

1 0

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2024.2/R-1)
by Goutham Pacha Ravi 22 Sep '24

22 Sep '24

Hello Stackers, We're in the home stretch of the 2024.2 ("Dalmatian") release cycle this week [1]. OpenStack services will be tagging their final release candidates in preparation for the final release next week. Since the release train never stops, I'd direct your attention to the 2025.1 ("Epoxy") release cycle's schedule [2], active feature development has already begun across the board. We concluded elections for four seats on the OpenStack Technical Committee last week [3]. I'm pleased to welcome first-time TC members Doug Goldstein (cardoe) and Sylvain Bauza (bauzas), and welcome back Ghanshyam Mann (gmann) and Dr. Jens Harbott (frickler). I'd also like to extend congratulations to the PTLs of 37 project teams who volunteered to lead their teams. Most of these leaders have served in the past cycle, and on behalf of the TC, I'd like to thank them for their dedication. I also welcome Guillaume Boutry (gboutry), who will be the PTL for the Sunbeam project for the first time. Last week, the TC merged a goal proposal to move all Ubuntu-based testing in the Zuul CI system to the latest LTS release: 24.04 LTS (Noble Numbat). Ghanshyam Mann (gmann) is expected to lead this goal and is asking project teams to help resolve test failures and incompatibilities when transitioning both unit and functional test jobs to this distribution. This goal may be selected for implementation during the 2025.1 ("Epoxy") release cycle. Several minor governance changes were also merged and proposed, as listed below. === Weekly Meeting === The OpenStack TC weekly meeting was held over IRC on 2024-09-17. The IRC meeting logs are available on eavesdrop [4]. We continued to populate topics for the upcoming virtual Project Teams Gathering [5] and began addressing leadership appointments for two projects with volunteers (Swift and Mistral). We're also checking the community's interest in maintaining the "Watcher" project [6], and waiting on the Tacker team to drop its dependency on "kuryr-kubernetes" so that project can be retired. Amy Marrich (spotz) and Sylvain Bauza (bauzas) recapped the OpenInfra Summit in Suwon, noting the packed rooms and commending the organizers for a great job. Feedback from hallway discussions highlighted the need for improved documentation translation, as the lack of translated materials makes it difficult for new users and contributors, especially in non-English-speaking regions. We’ll discuss this further at the PTG in October. bauzas also raised the need for better tracking and communication regarding the etherpads used in the community for various tasks and contributor work. The next OpenStack TC meeting will be held over IRC in OFTC's #openstack-tc channel on 2024-09-24 at 1800 UTC. I hope you'll be able to attend and participate in the discussions. The agenda for the meeting is on the meeting wiki [7]. Please remember that you can always edit the wiki and propose new topics. === Governance Proposals === ==== Merged ==== - Add goal to migrate CI/CD jobs to Ubuntu Noble | https://review.opendev.org/c/openstack/governance/+/928078 - PTI: Project-first wording | https://review.opendev.org/c/openstack/governance/+/929644 - Charms: Retire unmaintained charms | https://review.opendev.org/c/openstack/governance/+/926793 - Add TC/PTL results from the 2025.1 election | https://review.opendev.org/c/openstack/governance/+/929940 ==== Open for Review ==== - Appoint Tim Burke as PTL for Swift | https://review.opendev.org/c/openstack/governance/+/928881 - Add Axel Vanzaghi as PTL for Mistral | https://review.opendev.org/c/openstack/governance/+/927962 - Retire all single charm repositories | https://review.opendev.org/c/openstack/governance/+/903490 - Add gouthamr nomination for 2025.1 TC chair | https://review.opendev.org/c/openstack/governance/+/929971 - Add chair nomination folder for 2025.1 | https://review.opendev.org/c/openstack/governance/+/929970 - Select community-wide goal: Migrate CI/CD jobs to Ubuntu Noble | https://review.opendev.org/c/openstack/governance/+/929703 - Mark kuryr-kubernetes and kuryr-tempest-plugin inactive | https://review.opendev.org/c/openstack/governance/+/929698 === Upcoming Events === - 2024-10-08: OpenInfra Monthly Board Meeting: https://board.openinfra.dev/ - 2024-10-15: OpenInfra Days NA, Indianapolis: https://ittraining.iu.edu/explore-topics/titles/oid-iu/ - 2024-10-21: OpenInfra Project Teams Gathering: https://openinfra.dev/ptg/ Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2024.2 "Dalmatian" Release Schedule: https://releases.openstack.org/dalmatian/schedule.html [2] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [3] 2025.1 TC and PTL Election Results: [4] TC Meeting IRC Log, 2024-09-17: https://meetings.opendev.org/meetings/tc/2024/tc.2024-09-17-18.00.log.html [5] 2025.1 OpenStack TC/PTG planning etherpad: https://etherpad.opendev.org/p/oct2024-ptg-os-tc [6] Retirement of openstack/watcher: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [7] TC Meeting Agenda, 2024-09-24: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee

1 0

[upgrade][openstack] OpenStack Upgrade Procedure
by Ammar Al-Sayegh 21 Sep '24

21 Sep '24

Hello, I need to upgrade an OpenStack installation for the first time. The system Admin who performed the original installation is no longer available. How can I determine the current OpenStack version, modules installed, type of installation, and networking paradigm from OpenStack Horizon? Once determined, where I can find the appropriate upgrade procedure? Thank you. -ammar

2 1

[networking-mlnx] Project Health
by William Szumski 20 Sep '24

20 Sep '24

Hi all, I've noticed that there hasn't been any stable releases of networking-mlnx since stable/zed and was just wondering if this project still has any active maintainers. I would be interested in helping to keep this project going, so please give me a shout if you require any assistance. Thanks for all your hard work, Will

3 3

[unmaintained][cinder] question about zuul jobs for unmaintained branches
by Brian Rosmaita 20 Sep '24

20 Sep '24

Hello Unmaintained maintenance team, I was reviewing a cinder-tempest-plugin patch (which is in an un-branched repository), and noticed that we have a bunch of Zuul cinder jobs defined for stable branches that no longer exist, for example, stable/wallaby [0]. Where this impacts the unmaintained branches is that, for example, unmaintained/wallaby's .zuul.yaml is configured to run (for example) cinder-tempest-plugin-lvm-lio-barbican in the check [1], but you can see on this review, it's not actually running: https://review.opendev.org/c/openstack/cinder/+/926279?tab=change-view-tab-… So, my question is what to do about these unused jobs: (a) delete them (b) only delete stable/x for x <= ussuri, and update the jobs for stable/x for victoria <= x <= zed to unmaintained/x (c) delete them all, but propose a patch to the appropriate unmaintained branches zuul.yaml files that contain the job definition so that the Unmaintained team can make any changes needed to get them running without having to go through the cinder-tempest-plugin-core team's review/approval process (d) something I haven't thought of Thanks! brian [0] https://opendev.org/openstack/cinder-tempest-plugin/src/branch/master/.zuul… [1] https://opendev.org/openstack/cinder/src/branch/unmaintained/wallaby/.zuul.…

2 1

New to OpenStack. Test environment with 1 control node and 2 compute nodes. Unable to migrate or resize instances
by collinl＠churchofjesuschrist.org 20 Sep '24

20 Sep '24

As the subject says, I am new to OpenStack, and have spun up a test cluster with one control node, and two compute nodes. It seems to work for several tasks (i.e. I can spin up a cirros instance on both of the compute nodes, which seems to work just fine) However, when I attempt to migrate an instance from one compute node to the other, the instance goes in to an error state. (In order to get it running again, I have to set the state to active, then reboot the instance) I am not sure which logs will have the required information to help debug this issue, but I have looked through nova and cinder and keystone logs. Nothing I am seeing is obvious to me as to the cause of the failures. Also taking a snapshot seems to fail as well, with this error: Error: Unable to create snapshot. Details Invalid input received: Invalid volume: Volume e12b3ff8-9870-45f9-b59a-c6fb194ed517 status must be available, in-use, but current status is: attaching. (HTTP 400) (Request-ID: req-499d3bf9-97a0-48b2-8565-254a87287cff) (HTTP 400) (Request-ID: req-d0cd5aed-894e-40b2-8a2f-4b22188ec372) Looking at the volumes in my environment, both volumes show a status of "attaching" (both are for a test cirros instance, both are type nfs, and both instances seem to work just fine (meaning I can connect to them either on the console or via ssh and run commands without errors) Is the reason that each of those tasks is failing because of the "attaching" status of the volumes? I spun up the cluster using packstack and all the nodes are installed on Rocky Linux 9.4. I added an nfs backend after packstack was all done, and that does seem to be working, as the ephemeral volumes used to spin up the test instances does seem to be on the nfs storage, rather than the default lvm storage backend. I have installed openstack using the bobcat release of openstack (from the centos-openstack-bobcat repository). I have been crawling through the standalone install documentation to see if I can find something that explains why the instance migration isn't working, but, again, nothing is jumping out at me. Any guidance on where I should be looking, or advice on what I may be missing would be greatly appreciated.

4 10

upper-constraints (< $latest) considered harmful
by Stephen Finucane 20 Sep '24

20 Sep '24

o/ I'll jump straight into it. For those of you not in the loop we're currently going through a situation where it looks like OSC will be held back due to a bug that the Nova gates are highlighted. The actual details aren't hugely important, but what is important is the fact that this bug was first included in the OSC 7.0.0 release, which was released on August 6th. For those of you who are better at mental calendar maths than I am (or already looked at your calendar), you'll know this was over 6 weeks ago, well before the various feature freeze deadlines . and you might ask why did we only spot this last week? That would be because the upper-constraint (henceforth referred to as u-c) bump failed [1] due a different issue, meaning u-c kept us pinned at <= 6.6.1 which in turn meant every CI job *except* the ones running in the OSC gate itself kept testing the old release instead of the thing we were supposed to be released as part of Dalmatian. That only changed last week when 7.1.0 was released and the u-c bump for *that* merged [2], at which point people started shouting and hollering. Now, there is one other important consideration to factor into this, namely that, due to a flaw in our tooling, we didn't end up cutting a release branch for OSC alongside the rest of the client-library deliverables. Had we done so, we'd have seen 7.1.0 much earlier than we actually did [3]. That 7.1.0 release fixed the initial issue that prevented us releasing 7.0.0. However, this doesn't change the fact that 7.0.0 came out 6 weeks ago and not once did we get to test it outside of OSC's gate, and what's worse, we (the SDK team) were totally oblivious to the fact). Had it been known, we'd have cut a new release as soon as the fix [4] for that issue had merged (August 12th, which for the record is over 5 weeks ago). This is not the first time this has happened to us with one of our own dependencies. We had a very similar issue in Bobcat (2023.2) where we cut an oslo.db 13.0.0 release very early in that cycle, followed by a number of subsequent releases, only to have none of them end up making their way into u-c due to issues with some services. We then got to the end of the cycle, realized this, and had to do a frenzied mega revert to get us back to a pre-13.x state and allow us to release an oslo.db deliverable for Bobcat. It also happened with Castellan, whose 4.2.0 release sat waiting to merge for 4 months [4] before undergoing a similar mass revert. Adding more fuel to this fire, it's not just a problem with our own dependencies either. In fact, the problem is probably worse there. Need I remind people that it took 5 releases or 2 years, 8 months and change to uncap SQLAlchemy 2.x (a contributing factor in the oslo.db issue, fwiw). We had to struggle to get Sphinx uncapped in Bobcat [5] and fight to not revert that cap. You currently can't build docs for many projects on Fedora hosts since we're capping, among many other things, the Pillow library. The list goes on. Which brings me to my main points, of which there are three. Firstly: * There needs to be a clear time limit for how long a u-c bump can remain unmerged and a plan for what happens if affected projects have not resolved their issues within that time. There will always be exceptions - no one could rightly have asked services to switch to SQLAlchemy 2.x in a fortnight, and libraries can have legit bugs that justify a block - but those exceptions should be well-understood and, as with the bumps themselves, time-limited. A project that needs 4 releases to adapt to a crucial new library version is not a healthy project. secondly: * Caps or u-c bump reverts should similarly be clearly time-limited. We should not be defaulting to e.g re-capping Sphinx because your docs build has started to fail and then leaving that cap in place for months. Cap, fix the issue, and remove the cap asap. and, finally: * A failing u-c bump for an openstack deliverable should be treated with the highest priority 🚨🚨🚨 and should be something that the corresponding team should be made aware of immediately so they can start coordinating a resolution. We should not have u-c patches languishing for weeks, let alone months, only for there to be a last minute panic like this. I have ideas on the above (which basically warrant to more stick/[7]) but, perhaps fortunately, this isn't something I can decide on my own. Nor, should I add, is it something I expect the understaffed and oversubscribed release team to be able to do themselves. Instead, I think it's something that the TC and community as a whole need to settle on. So as soon as we get Dalmatian out the door, let's do that. Cheers, Stephen [1] https://review.opendev.org/c/openstack/requirements/+/925763 [2] https://review.opendev.org/c/openstack/requirements/+/928948 [3] https://review.opendev.org/c/openstack/releases/+/928838 [4] https://review.opendev.org/c/openstack/requirements/+/883141 [5] https://review.opendev.org/c/openstack/requirements/+/891694 [6] https://review.opendev.org/c/openstack/requirements/+/927102 [7] https://y.yarn.co/dbdc0d5b-04e6-4c39-9cd3-44c3be3729ab.mp4

8 9