- openstack-discuss - lists.openstack.org

[ceilometer] What's the general status / recommendation for Telemetry in OpenStack
by Waines, Greg 26 Sep '24

26 Sep '24

Hey ... I am working in OpenStack StarlingX which supports a Kubernetes cluster and a containerized OpenStack cluster (via openstack-helm). Our OpenStack offering is fairly simple ... only running Nova, Cinder, Neutron, Keystone, Glance, Horizon and Heat ... @ Antelope. In a predecessor of StarlingX, maybe in the Queens(?) timeframe ... we had a Telemetry solution that consisted of: - Ceilometer (for basic data polling, notifications and collection) - Panko (for event DB) - Gnocchi (for time-series DB) - Aodh (for alarming based on metric thresholds, etc.) HOWEVER ... today (Caracal) ... - I think Gnocchi is still alive and well ... https://github.com/gnocchixyz/gnocchi * but maintained outside of OpenStack - I think OpenStack Panko was deprecated in ~ Wallaby ? * and don't think(?) it is maintained outside of OpenStack ? - The OpenStack Landscape at https://www.openstack.org/software/ * shows only Ceilometer and Aodh MY QUESTIONs : ----------------------- - so what are the current recommendations for OpenStack Telemetry ? - is it just Ceilometer and Aodh ? * Does Ceilometer provide DBs ? (like it used to a long time ago) - is Gnocchi (although maintained outside of OpenStack) still supported in the OpenStack environment as a DB option ? - what is used for event DB ? ... now that panko is gone ... Gnocchi ? Ceilometer ? - are there actually multiple options/solutions for OpenStack Telemetry ? e.g. * Ceilometer / Gnocchi / Aodh * Ceilometer / External Elastic Solution / Aodh * Others ? Greg.

5 5

[powerflex][glance]Dell PowerFlex: After setting the Glance backend to PowerFlex Cinder, uploading an image and creating an image volume does not result in the image volume being created correctly.
by cuiye.liu＠dell.com 25 Sep '24

25 Sep '24

Description of customer infrastructure (Environment) - MANDATORY PowerFlex 3.6.2, 4.5.2 2023.2(Bobcat) Issue/problem detail and customer impact (Description, Impact, Steps to reproduce the issue, How often it happens) - MANDATORY Provide a clear description of the issue After setting the Glance backend to PowerFlex Cinder, uploading an image and creating an image volume does not result in the image volume being created correctly. Clear statement of customer impact unable to use PowerFlex as the backend for Glance, so additional storage is needed Steps to reproduce the issue i.e. commands #openstack volume create --size 8 --type powerflex --image cirros_flex --bootable test_image_volume2 Frequency of the issue everytime The error information is as following: 2024-09-19 07:03:46.320 24 DEBUG cinder.volume.flows.manager.create_volume [None req-c0a6a1fb-e26e-4925-a338-a4c9a68952f1 9c7543d67e184d51a6a77d5046c9e8a1 379ea5a5d4824c7f85e51e77d7c12add - - - -] Volume 37a4ef47-93b5-4395-8cdc-d04498fde0a0: re-scheduling SchedulerAPI.create_volume attempt 3 due to Failed to download image 87ea548b-80ff-4f7b-802b-042debaae696, reason: IOError: 32 Corrupt image download. Hash was cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e expected 1b03ca1bc3fafe448b90583c12f367949f8b0e665685979d95b004e48574b953316799e23240f4f739d1b5eb4c4ca24d38fdc6f4f9d8247a2bc64db25d6bbdb2 _reschedule /var/lib/kolla/venv/lib64/python3.9/site-packages/cinder/volume/flows/manager/create_volume.py:162 It is similar to this bug that is submmited in 2021: 1936314 – Glance image download fails with the error [Errno 32] Corrupt image download. (redhat.com) https://bugzilla.redhat.com/show_bug.cgi?id=1936314 Does anyone knows if it is fixed?

1 0

Openstack deployment on a single node server
by Gyan Dev Mishra 25 Sep '24

25 Sep '24

Hi, I am a newbie to OpenStack deployment, we are looking for a deployment strategy for OpenStack to test all our cloud-based apps to save on testing expenses. I tried it with microstack where I faced some issues related to Available Zones which was always empty. So, is it possible to use only one node to deploy and use complete OpenStack or do we need to follow some different strategy. I am having one server machine on which I want to deploy the complete cloud solutions. Kindly provide me with the necessary insights. I really appreciate any help you can provide. Gyan dev, Sr. Software Developer at eCosmos Solutions DISCLAIMER: This communication may contain confidential or copyright information. If you are not an intended recipient, you must not keep, forward, copy, use, save or rely on this communication. Any such action is unauthorized and prohibited. If you have received this communication in error please delete it. Reasonable precautions have been taken to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.

3 2

Retire sqlalchemy-migrate
by Thomas Goirand 25 Sep '24

25 Sep '24

Hi, As much as I know, there's no active OpenStack project using sqlalchemy-migrate anymore. Can someone retire the project, or at least push some indication in the README.rst that the project is retired? When looking at: https://opendev.org/x/sqlalchemy-migrate nothing says so... Cheers, Thomas Goirand (zigo)

3 3

[trove][zaqar][vitrage] What to do with projects not supporting SQLAlchemy 2.x ?
by Thomas Goirand 25 Sep '24

25 Sep '24

Hi, A number of project still don't have support for SQLAlchemy 2.x, which is currently in Debian testing/unstable, and which will be the version in Debian 13 (aka: Trixie). If these project don't get quickly support for SQLAlchemy 2.x, they are going to drop (well, in fact, they were already kicked out of Debian Testing). Are these project still maintained? IMO, no support for SQLAlchemy 2.x is a reason good enough to declare they aren't... Cheers, Thomas Goirand (zigo)

2 2

Network MTU setting in ovn/ovs
by Łukasz Chrustek 25 Sep '24

25 Sep '24

Hi, I have a question about increasing MTU for tenant network. Now it set to 1442 by default on geneve baked tenant network. Is there any drawbacks or it is some kind of bad practice to increase this MTU to 1500 bytes or higher ? We had an issue with one migrated instance from older version of openstack (where was linuxbrigde configured), there mtu was set to 1500, so the settings for i.e. docker inside the vm was set respectively to the higher value. -- Best regards Lukasz Chrustek

4 11

[oslo.db][magnum] QueuePool hangs in Caracal
by Mohammed Naser 24 Sep '24

24 Sep '24

Hi there! The community has identified a bug (https://bugs.launchpad.net/magnum/+bug/2067345) which makes Magnum unusable / entirely broken in Caracal. There is a proposed workaround here: https://bugs.launchpad.net/magnum/+bug/2067345/comments/35 The majority of the issue seems to be around when the changes to remove autocommit were done, I've attempted to switch to enginefacade to prevent these issues (same as Ironic) but with no avail: https://review.opendev.org/c/openstack/magnum/+/927446 There is also a greenthread stacktrace there which shows where/how it's hanging, would it be possible to have someone who might have a bit more insight to help point us in the right direction? I'm more than happy to take this on and keep pushing it forward but I'm not sure if I know where the fix can go at this point. Thanks, Mohammed

2 1

[release][zun][kuryr-libnetwork][kuryr-kubernetes] Immediate action required for 2024.2 (Dalmatian) release
by Dmitriy Rabotyagov 24 Sep '24

24 Sep '24

Hi, As you might know, R-2 week where Release Candidates for all projects following cycle-with-rc cadence should be created has already passed. Nevertheless there are a couple of projects, which do have broken Zuul Jobs[1], which prevents them from being released (tagged with RC1 and create a stable/2024.2 branch). There is a hard deadline by the end of this week (September 27, 2024) to merge required fixes into projects and update corresponsive release patches [2][3][4] in order to be included into the Dalmatian release. Otherwise the Technical Committee will have to take action on the mentioned projects, which can include preventing them from being released and marking them as Inactive ones. So everyone interested in the future of these projects - please take a look at the gate health and help core teams out fixing outstanding issues. Core teams - please prioritise review of proposed patches or come up with fixes during this week. [1] https://review.opendev.org/q/topic:dalmatian-rc1-deadline+is:open [2] https://review.opendev.org/c/openstack/releases/+/928531 [3] https://review.opendev.org/c/openstack/releases/+/928532 [4] https://review.opendev.org/c/openstack/releases/+/928575

1 0

[manila] Propose Gireesh Awasthi to manila core
by Carlos Silva 24 Sep '24

24 Sep '24

Hello everyone, I would like to add Gireesh Awasthi to the OpenStack Manila core team. Gireesh started his OpenStack journey a couple of cycles ago, and caused great impact from the get go while working on some complex features that impacted all of the manila code base. He's also been consistently participative in the latest PTGs. With the knowledge acquired in the code base, he then started contributing more to reviews, and became a go-to person for providing feedback on the NetApp ONTAP driver changes, and/or third party driver changes. It is important to note that this is yet another step into maintaining a healthy community with people from different affiliations sharing the responsibility of maintaining an open source project. In case there are no objections, I will add him to the core team in 1 week from now. [1] https://bit.ly/manila-reviews-90-days Thank you! Regards, carloss

2 2

[cyborg][zun][masakari][tacker][venus] Pending removal of LegacyEngineFacade
by Stephen Finucane 24 Sep '24

24 Sep '24

o/ Just a heads up to the listed projects that oslo.db's LegacyEngineFacade is pending removal in the Epoxy cycle. I would encourage you to migrate asap to avoid a broken CI. For a minimal example of how to do this, I'd encourage you to look at [1]. Cheers, Stephen [1] https://review.opendev.org/c/openstack/mistral/+/926420

1 0

Looking forward to work with Real-world project!
by Artemio Mendoza 23 Sep '24

23 Sep '24

Hi OpenStackers, As a new OpenStack community member, I read your email very carefully and looking for an opportunity that I can support. After graduating from university, I got a BS degree in computer science. I have been working on web/mobile app development for 10 years and have some reasonable experience in AWS, Azure, and other cloud environments. But I was really interested in OpenStack cloud computing so I started researching this last year and I'm actively keeping an eye on this community email, and trying to find some clues and new ideas about what I am going to do. I'm especially interested in Ceph and VDI development based on OpenStack. For my experience, I have tried to set up devstack on my laptop(20 core i7th, 32GB RAM), and tried run Bumblebee ( https://github.com/NeCTAR-RC/bumblebee) but failed. :( So for now, I'm still going to learn OpenStack doc. But I want to practice myself by jumping into real-world projects. If you are interested in me or looking for someone who can collaborate/research, then I will absolutely dedicate myself to that as a volunteer. Looking forward to hearing many good suggestions. Please feel free to reach out to me through Skype (huknamatata21) or this number (+15124877569) Regards, Artemio (Hony)

1 0

[Neutron] Bug deputy Sept 16 - 23
by Jakub Libosvar 23 Sep '24

23 Sep '24

Hi, I was the bug deputy last week. Was calm, only 3 bugs reported. Here is the breakdown: High ---- * security-groups-rules-belongs-to-default-sg slows down SG GET operations - https://bugs.launchpad.net/neutron/+bug/2081173 - I marked it as high because it’s a performance regression Medium ------ * Tighten / improve rally SG tests to catch performance regressions - mostly a placeholder to improve rally scenarios to prevent regressing in performance - https://bugs.launchpad.net/neutron/+bug/2081108 Wishlist -------- * Neutron OVN support for CARP - https://bugs.launchpad.net/neutron/+bug/2081643 - sounds like a new feature but it also requires some enhancements in the core OVN first

1 0

[publiccloud-sig] Heads-Up: next meeting September 25th - 0700 UTC
by Felix Kronlage-Dammers 23 Sep '24

23 Sep '24

Hi everyone, We are adjusting the schedule for the publiccloud-sig and will switch to odd instead of even weeks. This Wednesday (September 25th) the next meeting of the publiccloud-sig is going to happen. Thanks to Artem (gtema) and Joern (joek-office) who helped prepare the meeting agenda for this wednesday: - Housekeeping - Upcoming vPTG - Reminder to sign up for PTG: https://openinfra.dev/ptg/ - IAM Federation - combination of functional requirements (every CSP is having unique use case so we want to find similarities and differencies to determine the roadmap for Keystone) If you’re interesting in IAM federation, be sure to join! We’ll be meeting at 0700 UTC here: https://meetpad.opendev.org/publiccloud-sig <https://meetpad.opendev.org/publiccloud-sig> See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG <https://wiki.openstack.org/wiki/PublicCloudSIG> See you on Wednesday! felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

Openstack power off/on process
by Michael Knox 23 Sep '24

23 Sep '24

Hey Folks. I am working through a scenario that requires us to have Openstack in a Pelican Case and deployed in the "field" on a semi regular basis. I guess, portal "edge cloud" if you will? The problem I am trying to clean up is the startup process. We use kolla-ansible and rocky with zed (though we have one with ubuntu and wallaby) in this specific use case. We do the shutdown nicely, kolla-ansible -i inventory stop, then power off the control nodes (we have 3 in the setup). The problem is on the start up. All the containers start and then we still *have* to run the kolla-ansible -i inventory mariadb_recovery step to get the cloud back into a usable state. No matter the order of shut down or start up, MariaDB seems to always need a recovery. Are there any people here doing this or something similar? Is there a more efficient startup process I am not aware of or should consider? The folks in the field aren't Openstack admins, so hoping to figure out a solution that is less dependent on their knowledge of Openstack's undercloud tooling. Thanks in advance Michael

2 2

[cinder][CI] Can't start zuul service
by xuq.fnstxz＠fujitsu.com 22 Sep '24

22 Sep '24

Hello every one, I'm building a CI system on my ubuntu 22 system. Now I've installed the zuul using 'pip install zuul' and configured it. But after running command 'zuul-scheduler', I saw a lot of errors like following in the log. It seems that there is something error in my config file that prevents access to the repo. > 2024-09-20 13:05:06,010 ERROR zuul.TenantParser: Error processing cat job > Traceback (most recent call last): > File "/home/stack/.local/lib/python3.10/site-packages/zuul/configloader.py", line 2157, in _cacheTenantYAML > self._processCatJob(abide, tenant, loading_errors, job, > File "/home/stack/.local/lib/python3.10/site-packages/zuul/configloader.py", line 2249, in _processCatJob > raise TimeoutError(f"Cat job {job} timed out; consider setting " > TimeoutError: Cat job <zuul.zk.event_queues.JobResultFuture object at 0x7fb770341510> timed out; consider setting merger.git_timeout in zuul.conf Following is my zuul.conf. > [zookeeper] > hosts=ZOOKEEPER_HOST > tls_cert=/cert/client-cert.pem > tls_key=/cert/client-key.pem > tls_ca=/cert/ca-cert.pem > > [database] > # Use variable interpolation to supply the password from the > # docker-compose file. > # https://zuul-ci.org/docs/zuul/latest/configuration.html > dburi=mysql+pymysql://DB_USER:DB_PASS@DB_HOST/ZUUL_DB > > [keystore] > password=secret > > [web] > listen_address=0.0.0.0 > port=9000 > root=http://localhost:9000 > log_config=/etc/zuul/logging.conf > > [webclient] > url=http://localhost:9000 > verify_ssl=false > > [scheduler] > tenant_config=/etc/zuul/main.yaml > log_config=/etc/zuul/logging.conf > [merger] > git_user_email=git_user(a)example.com > git_user_name=git_user > log_config=/etc/zuul/logging.conf > > [executor] > private_key_file=/home/stack/.ssh/id_rsa > default_username=executor_user > trusted_rw_paths=/srv/static/logs > > [connection gerrit] > driver=gerrit > server=review.opendev.org > user=git_user > sshkey=/ssh/id_rsa And the following is the tenant_config file. > - tenant: > name: myCI > source: > gerrit: > untrusted-projects: > - openstack/cinder Could you please tell me what I need to modify or add in the config file? Thanks and Regards, Inori.

2 2

[kolla] Weird upgrade procudere ?
by Michal Arbet 22 Sep '24

22 Sep '24

Hi Koalas, I am not sure but is this normal ? In rabbitmq.yml is this +++ b/ansible/rabbitmq.yml @@ -0,0 +1,115 @@ +--- +# For RabbitMQ we need to be careful about restarting services, to avoid losing quorum. +- name: Apply role rabbitmq + gather_facts: false + hosts: + - rabbitmq + - '&enable_rabbitmq_True' But I can see during the upgrade restart every node at the same moment and then ther is some try to do it better ... I mean is that restart before OK ? Thanks TASK [rabbitmq : Enable all stable feature flags] ***************************************************************************************************************************************************************** ok: [controller1] ok: [controller0] ok: [controller2] TASK [rabbitmq : Check rabbitmq containers] *********************************************************************************************************************************************************************** changed: [controller0] => (item={'key': 'rabbitmq', 'value': {'container_name': 'rabbitmq', 'group': 'rabbitmq', 'enabled': True, 'image': 'dockerhub.ultimum.io/kolla-dev/debian-rabbitmq:KB-2023.1', 'bootstrap_environment': {'KOLLA_BOOTSTRAP': None, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'volumes': ['/etc/kolla/rabbitmq/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'rabbitmq:/var/lib/rabbitmq/', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'healthcheck': {'interval': '30', 'retries': '3', 'start_period': '5', 'test': ['CMD-SHELL', 'healthcheck_rabbitmq'], 'timeout': '30'}, 'haproxy': {'rabbitmq_management': {'enabled': 'yes', 'mode': 'http', 'port': '15672', 'host_group': 'rabbitmq'}, 'rabbitmq_outward_management': {'enabled': False, 'mode': 'http', 'port': '15674', 'host_group': 'outward-rabbitmq'}, 'rabbitmq_outward_external': {'enabled': False, 'mode': 'tcp', 'external': True, 'port': '5674', 'host_group': 'outward-rabbitmq', 'frontend_tcp_extra': ['timeout client 1h'], 'backend_tcp_extra': ['timeout server 1h']}}}}) changed: [controller2] => (item={'key': 'rabbitmq', 'value': {'container_name': 'rabbitmq', 'group': 'rabbitmq', 'enabled': True, 'image': 'dockerhub.ultimum.io/kolla-dev/debian-rabbitmq:KB-2023.1', 'bootstrap_environment': {'KOLLA_BOOTSTRAP': None, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'volumes': ['/etc/kolla/rabbitmq/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'rabbitmq:/var/lib/rabbitmq/', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'healthcheck': {'interval': '30', 'retries': '3', 'start_period': '5', 'test': ['CMD-SHELL', 'healthcheck_rabbitmq'], 'timeout': '30'}, 'haproxy': {'rabbitmq_management': {'enabled': 'yes', 'mode': 'http', 'port': '15672', 'host_group': 'rabbitmq'}, 'rabbitmq_outward_management': {'enabled': False, 'mode': 'http', 'port': '15674', 'host_group': 'outward-rabbitmq'}, 'rabbitmq_outward_external': {'enabled': False, 'mode': 'tcp', 'external': True, 'port': '5674', 'host_group': 'outward-rabbitmq', 'frontend_tcp_extra': ['timeout client 1h'], 'backend_tcp_extra': ['timeout server 1h']}}}}) changed: [controller1] => (item={'key': 'rabbitmq', 'value': {'container_name': 'rabbitmq', 'group': 'rabbitmq', 'enabled': True, 'image': 'dockerhub.ultimum.io/kolla-dev/debian-rabbitmq:KB-2023.1', 'bootstrap_environment': {'KOLLA_BOOTSTRAP': None, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'volumes': ['/etc/kolla/rabbitmq/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'rabbitmq:/var/lib/rabbitmq/', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'healthcheck': {'interval': '30', 'retries': '3', 'start_period': '5', 'test': ['CMD-SHELL', 'healthcheck_rabbitmq'], 'timeout': '30'}, 'haproxy': {'rabbitmq_management': {'enabled': 'yes', 'mode': 'http', 'port': '15672', 'host_group': 'rabbitmq'}, 'rabbitmq_outward_management': {'enabled': False, 'mode': 'http', 'port': '15674', 'host_group': 'outward-rabbitmq'}, 'rabbitmq_outward_external': {'enabled': False, 'mode': 'tcp', 'external': True, 'port': '5674', 'host_group': 'outward-rabbitmq', 'frontend_tcp_extra': ['timeout client 1h'], 'backend_tcp_extra': ['timeout server 1h']}}}}) RUNNING HANDLER [rabbitmq : Restart rabbitmq container] *********************************************************************************************************************************************************** [WARNING]: Failure using method (v2_playbook_on_handler_task_start) in callback plugin (<ansible.plugins.callback.ara_default.CallbackModule object at 0x7f66542e0100>): 'NoneType' object is not subscriptable changed: [controller0] changed: [controller1] changed: [controller2] PLAY [Restart rabbitmq services] ********************************************************************************************************************************************************************************** TASK [rabbitmq : Get info on RabbitMQ container] ****************************************************************************************************************************************************************** ok: [controller0] TASK [rabbitmq : Put RabbitMQ node into maintenance mode] ********************************************************************************************************************************************************* changed: [controller0] TASK [rabbitmq : Restart rabbitmq container] ********************************************************************************************************************************************************************** changed: [controller0] TASK [rabbitmq : Waiting for rabbitmq to start] ******************************************************************************************************************************************************************* changed: [controller0] PLAY [Restart rabbitmq services] ********************************************************************************************************************************************************************************** TASK [rabbitmq : Get info on RabbitMQ container] ****************************************************************************************************************************************************************** ok: [controller1] TASK [rabbitmq : Put RabbitMQ node into maintenance mode] ********************************************************************************************************************************************************* changed: [controller1] TASK [rabbitmq : Restart rabbitmq container] ********************************************************************************************************************************************************************** changed: [controller1] TASK [rabbitmq : Waiting for rabbitmq to start] ******************************************************************************************************************************************************************* changed: [controller1] PLAY [Restart rabbitmq services] ********************************************************************************************************************************************************************************** TASK [rabbitmq : Get info on RabbitMQ container] ****************************************************************************************************************************************************************** ok: [controller2] TASK [rabbitmq : Put RabbitMQ node into maintenance mode] ********************************************************************************************************************************************************* changed: [controller2] TASK [rabbitmq : Restart rabbitmq container] ********************************************************************************************************************************************************************** changed: [controller2] TASK [rabbitmq : Waiting for rabbitmq to start] ******************************************************************************************************************************************************************* changed: [controller2] PLAY [Apply rabbitmq post-configuration] ************************************************************************************************************************************************************************** TASK [Include rabbitmq post-deploy.yml] *************************************************************************************************************************************************************************** TASK [rabbitmq : Enable all stable feature flags] ************************* Michal Arbet Openstack Engineer Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic +420 604 228 897 michal.arbet(a)ultimum.io *https://ultimum.io <https://ultimum.io/>* LinkedIn <https://www.linkedin.com/company/ultimum-technologies> | Twitter <https://twitter.com/ultimumtech> | Facebook <https://www.facebook.com/ultimumtechnologies/timeline>

1 0

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2024.2/R-1)
by Goutham Pacha Ravi 22 Sep '24

22 Sep '24

Hello Stackers, We're in the home stretch of the 2024.2 ("Dalmatian") release cycle this week [1]. OpenStack services will be tagging their final release candidates in preparation for the final release next week. Since the release train never stops, I'd direct your attention to the 2025.1 ("Epoxy") release cycle's schedule [2], active feature development has already begun across the board. We concluded elections for four seats on the OpenStack Technical Committee last week [3]. I'm pleased to welcome first-time TC members Doug Goldstein (cardoe) and Sylvain Bauza (bauzas), and welcome back Ghanshyam Mann (gmann) and Dr. Jens Harbott (frickler). I'd also like to extend congratulations to the PTLs of 37 project teams who volunteered to lead their teams. Most of these leaders have served in the past cycle, and on behalf of the TC, I'd like to thank them for their dedication. I also welcome Guillaume Boutry (gboutry), who will be the PTL for the Sunbeam project for the first time. Last week, the TC merged a goal proposal to move all Ubuntu-based testing in the Zuul CI system to the latest LTS release: 24.04 LTS (Noble Numbat). Ghanshyam Mann (gmann) is expected to lead this goal and is asking project teams to help resolve test failures and incompatibilities when transitioning both unit and functional test jobs to this distribution. This goal may be selected for implementation during the 2025.1 ("Epoxy") release cycle. Several minor governance changes were also merged and proposed, as listed below. === Weekly Meeting === The OpenStack TC weekly meeting was held over IRC on 2024-09-17. The IRC meeting logs are available on eavesdrop [4]. We continued to populate topics for the upcoming virtual Project Teams Gathering [5] and began addressing leadership appointments for two projects with volunteers (Swift and Mistral). We're also checking the community's interest in maintaining the "Watcher" project [6], and waiting on the Tacker team to drop its dependency on "kuryr-kubernetes" so that project can be retired. Amy Marrich (spotz) and Sylvain Bauza (bauzas) recapped the OpenInfra Summit in Suwon, noting the packed rooms and commending the organizers for a great job. Feedback from hallway discussions highlighted the need for improved documentation translation, as the lack of translated materials makes it difficult for new users and contributors, especially in non-English-speaking regions. We’ll discuss this further at the PTG in October. bauzas also raised the need for better tracking and communication regarding the etherpads used in the community for various tasks and contributor work. The next OpenStack TC meeting will be held over IRC in OFTC's #openstack-tc channel on 2024-09-24 at 1800 UTC. I hope you'll be able to attend and participate in the discussions. The agenda for the meeting is on the meeting wiki [7]. Please remember that you can always edit the wiki and propose new topics. === Governance Proposals === ==== Merged ==== - Add goal to migrate CI/CD jobs to Ubuntu Noble | https://review.opendev.org/c/openstack/governance/+/928078 - PTI: Project-first wording | https://review.opendev.org/c/openstack/governance/+/929644 - Charms: Retire unmaintained charms | https://review.opendev.org/c/openstack/governance/+/926793 - Add TC/PTL results from the 2025.1 election | https://review.opendev.org/c/openstack/governance/+/929940 ==== Open for Review ==== - Appoint Tim Burke as PTL for Swift | https://review.opendev.org/c/openstack/governance/+/928881 - Add Axel Vanzaghi as PTL for Mistral | https://review.opendev.org/c/openstack/governance/+/927962 - Retire all single charm repositories | https://review.opendev.org/c/openstack/governance/+/903490 - Add gouthamr nomination for 2025.1 TC chair | https://review.opendev.org/c/openstack/governance/+/929971 - Add chair nomination folder for 2025.1 | https://review.opendev.org/c/openstack/governance/+/929970 - Select community-wide goal: Migrate CI/CD jobs to Ubuntu Noble | https://review.opendev.org/c/openstack/governance/+/929703 - Mark kuryr-kubernetes and kuryr-tempest-plugin inactive | https://review.opendev.org/c/openstack/governance/+/929698 === Upcoming Events === - 2024-10-08: OpenInfra Monthly Board Meeting: https://board.openinfra.dev/ - 2024-10-15: OpenInfra Days NA, Indianapolis: https://ittraining.iu.edu/explore-topics/titles/oid-iu/ - 2024-10-21: OpenInfra Project Teams Gathering: https://openinfra.dev/ptg/ Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2024.2 "Dalmatian" Release Schedule: https://releases.openstack.org/dalmatian/schedule.html [2] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [3] 2025.1 TC and PTL Election Results: [4] TC Meeting IRC Log, 2024-09-17: https://meetings.opendev.org/meetings/tc/2024/tc.2024-09-17-18.00.log.html [5] 2025.1 OpenStack TC/PTG planning etherpad: https://etherpad.opendev.org/p/oct2024-ptg-os-tc [6] Retirement of openstack/watcher: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [7] TC Meeting Agenda, 2024-09-24: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee

1 0

[upgrade][openstack] OpenStack Upgrade Procedure
by Ammar Al-Sayegh 21 Sep '24

21 Sep '24

Hello, I need to upgrade an OpenStack installation for the first time. The system Admin who performed the original installation is no longer available. How can I determine the current OpenStack version, modules installed, type of installation, and networking paradigm from OpenStack Horizon? Once determined, where I can find the appropriate upgrade procedure? Thank you. -ammar

2 1

[networking-mlnx] Project Health
by William Szumski 20 Sep '24

20 Sep '24

Hi all, I've noticed that there hasn't been any stable releases of networking-mlnx since stable/zed and was just wondering if this project still has any active maintainers. I would be interested in helping to keep this project going, so please give me a shout if you require any assistance. Thanks for all your hard work, Will

3 3

[unmaintained][cinder] question about zuul jobs for unmaintained branches
by Brian Rosmaita 20 Sep '24

20 Sep '24

Hello Unmaintained maintenance team, I was reviewing a cinder-tempest-plugin patch (which is in an un-branched repository), and noticed that we have a bunch of Zuul cinder jobs defined for stable branches that no longer exist, for example, stable/wallaby [0]. Where this impacts the unmaintained branches is that, for example, unmaintained/wallaby's .zuul.yaml is configured to run (for example) cinder-tempest-plugin-lvm-lio-barbican in the check [1], but you can see on this review, it's not actually running: https://review.opendev.org/c/openstack/cinder/+/926279?tab=change-view-tab-… So, my question is what to do about these unused jobs: (a) delete them (b) only delete stable/x for x <= ussuri, and update the jobs for stable/x for victoria <= x <= zed to unmaintained/x (c) delete them all, but propose a patch to the appropriate unmaintained branches zuul.yaml files that contain the job definition so that the Unmaintained team can make any changes needed to get them running without having to go through the cinder-tempest-plugin-core team's review/approval process (d) something I haven't thought of Thanks! brian [0] https://opendev.org/openstack/cinder-tempest-plugin/src/branch/master/.zuul… [1] https://opendev.org/openstack/cinder/src/branch/unmaintained/wallaby/.zuul.…

2 1

New to OpenStack. Test environment with 1 control node and 2 compute nodes. Unable to migrate or resize instances
by collinl＠churchofjesuschrist.org 20 Sep '24

20 Sep '24

As the subject says, I am new to OpenStack, and have spun up a test cluster with one control node, and two compute nodes. It seems to work for several tasks (i.e. I can spin up a cirros instance on both of the compute nodes, which seems to work just fine) However, when I attempt to migrate an instance from one compute node to the other, the instance goes in to an error state. (In order to get it running again, I have to set the state to active, then reboot the instance) I am not sure which logs will have the required information to help debug this issue, but I have looked through nova and cinder and keystone logs. Nothing I am seeing is obvious to me as to the cause of the failures. Also taking a snapshot seems to fail as well, with this error: Error: Unable to create snapshot. Details Invalid input received: Invalid volume: Volume e12b3ff8-9870-45f9-b59a-c6fb194ed517 status must be available, in-use, but current status is: attaching. (HTTP 400) (Request-ID: req-499d3bf9-97a0-48b2-8565-254a87287cff) (HTTP 400) (Request-ID: req-d0cd5aed-894e-40b2-8a2f-4b22188ec372) Looking at the volumes in my environment, both volumes show a status of "attaching" (both are for a test cirros instance, both are type nfs, and both instances seem to work just fine (meaning I can connect to them either on the console or via ssh and run commands without errors) Is the reason that each of those tasks is failing because of the "attaching" status of the volumes? I spun up the cluster using packstack and all the nodes are installed on Rocky Linux 9.4. I added an nfs backend after packstack was all done, and that does seem to be working, as the ephemeral volumes used to spin up the test instances does seem to be on the nfs storage, rather than the default lvm storage backend. I have installed openstack using the bobcat release of openstack (from the centos-openstack-bobcat repository). I have been crawling through the standalone install documentation to see if I can find something that explains why the instance migration isn't working, but, again, nothing is jumping out at me. Any guidance on where I should be looking, or advice on what I may be missing would be greatly appreciated.

4 10

upper-constraints (< $latest) considered harmful
by Stephen Finucane 20 Sep '24

20 Sep '24

o/ I'll jump straight into it. For those of you not in the loop we're currently going through a situation where it looks like OSC will be held back due to a bug that the Nova gates are highlighted. The actual details aren't hugely important, but what is important is the fact that this bug was first included in the OSC 7.0.0 release, which was released on August 6th. For those of you who are better at mental calendar maths than I am (or already looked at your calendar), you'll know this was over 6 weeks ago, well before the various feature freeze deadlines . and you might ask why did we only spot this last week? That would be because the upper-constraint (henceforth referred to as u-c) bump failed [1] due a different issue, meaning u-c kept us pinned at <= 6.6.1 which in turn meant every CI job *except* the ones running in the OSC gate itself kept testing the old release instead of the thing we were supposed to be released as part of Dalmatian. That only changed last week when 7.1.0 was released and the u-c bump for *that* merged [2], at which point people started shouting and hollering. Now, there is one other important consideration to factor into this, namely that, due to a flaw in our tooling, we didn't end up cutting a release branch for OSC alongside the rest of the client-library deliverables. Had we done so, we'd have seen 7.1.0 much earlier than we actually did [3]. That 7.1.0 release fixed the initial issue that prevented us releasing 7.0.0. However, this doesn't change the fact that 7.0.0 came out 6 weeks ago and not once did we get to test it outside of OSC's gate, and what's worse, we (the SDK team) were totally oblivious to the fact). Had it been known, we'd have cut a new release as soon as the fix [4] for that issue had merged (August 12th, which for the record is over 5 weeks ago). This is not the first time this has happened to us with one of our own dependencies. We had a very similar issue in Bobcat (2023.2) where we cut an oslo.db 13.0.0 release very early in that cycle, followed by a number of subsequent releases, only to have none of them end up making their way into u-c due to issues with some services. We then got to the end of the cycle, realized this, and had to do a frenzied mega revert to get us back to a pre-13.x state and allow us to release an oslo.db deliverable for Bobcat. It also happened with Castellan, whose 4.2.0 release sat waiting to merge for 4 months [4] before undergoing a similar mass revert. Adding more fuel to this fire, it's not just a problem with our own dependencies either. In fact, the problem is probably worse there. Need I remind people that it took 5 releases or 2 years, 8 months and change to uncap SQLAlchemy 2.x (a contributing factor in the oslo.db issue, fwiw). We had to struggle to get Sphinx uncapped in Bobcat [5] and fight to not revert that cap. You currently can't build docs for many projects on Fedora hosts since we're capping, among many other things, the Pillow library. The list goes on. Which brings me to my main points, of which there are three. Firstly: * There needs to be a clear time limit for how long a u-c bump can remain unmerged and a plan for what happens if affected projects have not resolved their issues within that time. There will always be exceptions - no one could rightly have asked services to switch to SQLAlchemy 2.x in a fortnight, and libraries can have legit bugs that justify a block - but those exceptions should be well-understood and, as with the bumps themselves, time-limited. A project that needs 4 releases to adapt to a crucial new library version is not a healthy project. secondly: * Caps or u-c bump reverts should similarly be clearly time-limited. We should not be defaulting to e.g re-capping Sphinx because your docs build has started to fail and then leaving that cap in place for months. Cap, fix the issue, and remove the cap asap. and, finally: * A failing u-c bump for an openstack deliverable should be treated with the highest priority 🚨🚨🚨 and should be something that the corresponding team should be made aware of immediately so they can start coordinating a resolution. We should not have u-c patches languishing for weeks, let alone months, only for there to be a last minute panic like this. I have ideas on the above (which basically warrant to more stick/[7]) but, perhaps fortunately, this isn't something I can decide on my own. Nor, should I add, is it something I expect the understaffed and oversubscribed release team to be able to do themselves. Instead, I think it's something that the TC and community as a whole need to settle on. So as soon as we get Dalmatian out the door, let's do that. Cheers, Stephen [1] https://review.opendev.org/c/openstack/requirements/+/925763 [2] https://review.opendev.org/c/openstack/requirements/+/928948 [3] https://review.opendev.org/c/openstack/releases/+/928838 [4] https://review.opendev.org/c/openstack/requirements/+/883141 [5] https://review.opendev.org/c/openstack/requirements/+/891694 [6] https://review.opendev.org/c/openstack/requirements/+/927102 [7] https://y.yarn.co/dbdc0d5b-04e6-4c39-9cd3-44c3be3729ab.mp4

8 9

[release] Release countdown for week R-1, Sep 23-27
by Elõd Illés 20 Sep '24

20 Sep '24

Development Focus ----------------- We are on the final mile of the 2024.2 Dalmatian development cycle! Remember that the 2024.2 Dalmatian final release will include the latest release candidate (for cycle-with-rc deliverables) or the latest intermediary release (for cycle-with-intermediary deliverables) available. September 26th, 2024 is the deadline for final 2024.2 Dalmatian release candidates as well as any last cycle-with-intermediary deliverables. We will then enter a quiet period until we tag the final release on October 2nd, 2024. Teams should be prioritizing fixing release-critical bugs, before that deadline. Otherwise it's time to start planning the 2025.1 Epoxy development cycle, including discussing PTG sessions content, in preparation of Project Teams Gathering. Actions ------- Watch for any translation patches coming through on the stable/2024.2 branch and merge them quickly. If you discover a release-critical issue, please make sure to fix it on the master branch first, then backport the bugfix to the stable/2024.2 branch before triggering a new release. Please drop by #openstack-release with any questions or concerns about the upcoming release ! Upcoming Deadlines & Dates -------------------------- Final 2024.2 Dalmatian release: October 2nd, 2024 Project Teams Gathering: October 21-25, 2024 Előd Illés irc: elodilles @ #openstack-release

1 0

[cinder] Retype encrypted cinder volume to non-encrypted volume
by Jan Wasilewski 20 Sep '24

20 Sep '24

Hi, I would like to ask about the procedure to retype a Cinder volume that is in-use (bootable volume, connected to a VM) from an encrypted (LUKS) type to a non-encrypted type (1000 IOPS). I tried to do this via the CLI[1], and while it seemed to work (at least according to the CLI output), the process didn’t complete successfully. Cinder logs[2] indicate that everything looks more or less fine: the volume is created, the volume migration is OK, and the retype is marked as completed. However, in the end, it doesn’t work (final output of [1]). I reviewed the blueprint at https://blueprints.launchpad.net/cinder/+spec/retype-encrypted-volume, which appears to be implemented. Am I doing something wrong, or does this functionality not work as expected? Thanks in advance for your help and guidance. /JW *[1] Retype from CLI: https://paste.openstack.org/show/bVpFT82oURnanXkixreT/ <https://paste.openstack.org/show/bVpFT82oURnanXkixreT/>[2] cinder-logs: https://paste.openstack.org/show/b5rZpWNMqdi6qdpjaDfg/ <https://paste.openstack.org/show/b5rZpWNMqdi6qdpjaDfg/>*

4 6

[glance] Still not working over uwsgi
by Thomas Goirand 20 Sep '24

20 Sep '24

Hi, I've been told in the past in this very list, that Glance would now work over uwsgi. Though that's not the case. Normal image upload works, but doing a "openstack server backup" fails, as well as shelving (it's the same operation...). We get: "Failed to upload image data due to HTTP error: webob.exc.HTTPRequestEntityTooLarge: Image storage media is full: Object PUT failed" even though the Swift user Glance is configured for isn't over quota. Is there any configuration options that we missed? Or should we still consider that Glance over uwsgi is broken? Cheers, Thomas Goirand (zigo)

3 2