July 2025 - openstack-discuss - lists.openstack.org

Instance Freeze During Snapshot in OpenStack 2024.1 with Ceph Octopus
by mohammad kokabi 21 Jul '25

21 Jul '25

Hello, I hope this message finds you well. I am currently using OpenStack version 2024.1, with Ceph Octopus as the backend storage. The configuration is as follows: Glance is configured to use the "images" pool on Ceph. The virtual machine disks are stored on the "vms" pool. When I attempt to create a snapshot of a running instance, the instance appears to freeze for a few seconds, during which it loses network connectivity (e.g., I cannot ping it). This behavior is unexpected, as I understand that snapshot operations, especially with Ceph backend, should not significantly interrupt the instance's runtime. Could you please advise what might be causing this issue? Are there any known limitations, configurations, or best practices I should be aware of regarding snapshot behavior with Ceph in this version of OpenStack? Looking forward to your insights. Best regards,

2 1

RE: Create image error
by berndbausch＠gmail.com 21 Jul '25

21 Jul '25

Violating limits imposed by the operator should lead to a useful error message rather than an HTTP 5xx error. In any case, there should have been a message in the Glance API log. Perhaps you are using a prerelease version of OpenStack? When you get Devstack from the Git, you may want to specify the version; otherwise, you obtain software that is currently under development. For example, the following gets you the first version released this year (Epoxy): git clone https://opendev.org/openstack/devstack -b stable/2025.1 ------------------------------------------------------------------------------------------------------------------- From: BEDDA Fadhel <fadhel.bedda(a)gmail.com> Sent: Saturday, May 31, 2025 6:00 PM To: berndbausch(a)gmail.com Subject: Re: Create image error Hello, I was able to work around the problem by disabling the limits in the file: /etc/glance/glance-api.conf False instead of True Thanks Le sam. 31 mai 2025 à 09:55, <mailto:berndbausch@gmail.com> a écrit : My guess is that the image service is down. To check its status: systemctl status devstack@g-api Also check the image service log: journalctl -e -u devstack@g-api ------------------------------------------------------------------------ From: BEDDA Fadhel <mailto:fadhel.bedda@gmail.com> Sent: Saturday, May 31, 2025 3:08 PM To: mailto:openstack-discuss@lists.openstack.org Subject: Create image error Hi everyone, I installed OpenStack (DevStack). At first, I was able to create three images (Ubuntu, Debian, and Windows) in addition to the Cirros image. When I tried to add another image, everything got stuck. I couldn't add any images, and I got the infamous error 550. HttpException: 502: Server Error for url:

1 0

[Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches �C Suspected Regression in Neutron Master Branch
by Renu Rani 21 Jul '25

21 Jul '25

Hi Neutron Team, We are currently facing an issue in Tacker Zuul patch jobs build (with master branch), where the DevStack setup fails during the initial Neutron network creation step. Problem statement: The failure occurs at this point in the stack.sh script, for devstack environment creation in build process. Below command execution is failing in stack.sh [openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc d777d04b-93ff-4e94-b6a9-cc8de3e23921] Error Message: This results in a 500 Internal Server Error, as seen in the logs: [HttpException: 500: Server Error for url: http://10.x.x.x/networking/v2.0/routers/20b264df-5f19-4599-a83d-2e76e7cbf5d…] Analysis for the issue: Below error is coming in Neutron API logs: [ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on un-acquired lock] 1. Tried overriding the required-projects in Zuul to use stable/2025.1 branches (instead of master) to identify project(master branch) which is causing root cause. 2. Tacker team has created DNM patch (952892) to invetsigate this in detail. https://review.opendev.org/c/openstack/tacker/+/952892 1. In DNM patch Patchset 11 (on June 29), we have taken devstack stable/2025.1. In stable branch the devstack environment created sucessfully. 2. In DNM patch Patchset 25 (on July 8), with the same devstack override (stable/2025.1), the job now fails earlier during DevStack setup. The logs show that Neutron failed to start, causing the job to fail across all nodes. As per observations and anlysis seems the recent changes in the Neutron master branch between June 29 and July 8 might be causing this issue. We’d appreciate your help in reviewing the changes in Neutron's master branch during that window that could cause this behavior and suggesting any debugging approaches or patches we can test on tacker. Please let us know if you'd like additional logs or info from the Zuul job runs. Thanks in advance for your support! Thanks & Regards, Renu Rani (レヌラニ) Mobile: +91-9311888962 E-mail: renu.rani(a)india.nec.com<mailto:renu.rani@india.nec.com> The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.

3 5

[tacker] Skip today's IRC meeting
by Yasufumi Ogawa 20 Jul '25

20 Jul '25

Hi Tacker team, Sorry for the late notice. Due to a holiday in Japan and most of us will not join, I would like to skip today's IRC meeting. Thanks, Yasufumi

1 0

[ops][largescale-sig][scientific-sig] Ops Radio Hour on July 25, 2025
by Ildiko Vancsa 20 Jul '25

20 Jul '25

Hi OpenStack community, I'm reaching out to you with a friendly reminder that the next OpenStack Ops Radio Hour[1] is scheduled to happen on __July 25th, 2025 at 1300UTC.__ The call series is a casual setting for operators, engineers, and contributors to come together, share knowledge, and collaborate on the complexities of managing and deploying OpenStack environments. Topics the next gathering will dive into: - OVN Migration +Documentation proposal: https://github.com/mariojmdavid/migrate-to-ovn - (SWIFT) Compliance processes - GPUs - VMware migration - vTPM (live migrations or not?): reboot, start What other topics should we add to the agenda? A few reminders: - The group on the call agreed to try out Matrix as a chat platform for more synchronous communication outside calls + You can access the Matrix room here: https://matrix.to/#/#openstack-ops:opendev.org <https://matrix.to/#/%23openstack-ops:opendev.org> + Further details are here: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… - If you would like to read and catch up on notes from previous calls, you can find them in this etherpad: https://etherpad.opendev.org/p/openstack_ops_radiohour See you on the call on July 25th! Bets Regards, Ildikó [1] https://wiki.openstack.org/wiki/Operations/Meetups

1 0

[kolla-ansible][proxysql] "Wait for backup proxysql container to start" fails
by Bursztynowski Dariusz 20 Jul '25

20 Jul '25

Hello OpenStack community, Installing Kolla-ansible OpenStack 2025.1, I've met a problem with proxysql which does not start (in task [loadbalancer : Wait for backup proxysql to start]) because of unsupported system page size: sudo cat /var/log/kolla/proxysql/proxysql.log <jemalloc>: Unsupported system page size <jemalloc>: Error in malloc(): out of memory I'm working with aarch64 CPU (Raspberry Pi 5 cluster, instructive tool for educational purposes). I have posted the bug here including more detailed description: https://bugs.launchpad.net/kolla-ansible/+bug/2117309 . What is interesting, version 2023.1 works just fine on the same HW/OS platform. I would be grateful for any suggestions on how to resolve this. Best, Dariusz Bursztynowski

1 1

[Nova] Unable to fetch CPU/RAM allocation ratios via /os-hypervisors API (Nova API v2.91 vs v2.1 support)
by Thamanna Farhath 19 Jul '25

19 Jul '25

Hello OpenStack community, I'm working with OpenStack (2023.1 kolla ansible) Nova and using the /os-hypervisors API to gather hypervisor information. I noticed that the CPU and RAM allocation ratios (which are shown in the Horizon dashboard) are not included in the response from: GET /os-hypervisors GET /os-hypervisors/{id} We're currently using Nova API version 2.91, but I see that the /os-hypervisors API appears to support only up to version 2.1, I’d like to clarify the following points: Is it expected behavior that the /os-hypervisors API does not return CPU and RAM allocation ratios? Is there any API endpoint that exposes these ratios programmatically? As we are using Nova API version 2.91, but /os-hypervisors seems limited to v2.1, is this a known limitation or design choice? If the only source of truth is the nova.conf configuration file (e.g., cpu_allocation_ratio, ram_allocation_ratio), is there any plan to expose this via API in future releases? It would be helpful to know if this is something we should work around by manually collecting config values, or if there's a more appropriate method in modern Nova setups. Thamanna Farhath Associate Engineer R&D mailto:manoj.kumar@zybisys.com thamanna.f(a)zybisys.com https://www.zybisys.com https://in.linkedin.com/company/zybisys https://www.facebook.com/Zybisys/ https://www.instagram.com/ZyBiSys/ https://www.youtube.com/@zybisys https://www.facebook.com/Zybisys/ Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

1 0

[Kolla-ansible][epoxy]nova problems at restart
by Franck VEDEL (UGA) 18 Jul '25

18 Jul '25

Hello everyone, I need your help. II will try to explain my problem by providing a lot of information. I have been trying to solve it for 3 days, without success. In May, my old cluster became too small for the 200 students using it, so I started installing a new cluster. Six nodes, three in one data center, three in another. After testing with Bobcat and Caracal (unsatisfactory, with numerous issues, especially with authentication and display in Horizon), I tried Epoxy. I installed everything I needed and tested it. It was a real success. It was very fast in its new configuration, and I thought everything would be ready by September. In May, my old cluster was becoming too small for the 200 students using it, so I started installing a new cluster. 6 nodes, 3 in one data center, 3 in another. After testing with Bobcat and Caracal (unsatisfactory, with numerous issues, especially with authentication and display in Horizon), I tried Epoxy. I installed everything I needed and tested it. It was a real success. It was very fast in its new configuration; I thought everything would be ready by September. At the beginning of July, I was told to shut down everything, due to electrical maintenance in a data center. I shut everything down. 3 days later, I restarted it, and nothing worked anymore. However, it was simply a shutdown/restart of the servers. The problems are with Nova. Here's some information and logs, if you can help me restart the cluster. So Epoxy on Ubuntu 2404. (venv2) user1@ordi1:~/OPENSTACK/INSTALL$ kolla-ansible --version kolla-ansible 20.0.1.dev4 Since nova didn't want to work I did this: (venv2) user1@ordi1:~/OPENSTACK/INSTALL$ kolla-ansible reconfigure -I multinode -t nova ERROR with a task: TASK [nova-cell : Waiting for nova-compute services to register themselves] **************** skipping: [ordi2] skipping: [ordi3] skipping: [ordi4] skipping: [ordi5] skipping: [ordi6] An exception occurred during task execution. To see the full traceback, use -vvv. The error was: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) fatal: [ordi1]: FAILED! => {"msg": "Unexpected failure during module execution: Expecting value: line 1 column 1 (char 0)", "stdout": ""} I search with Google and find this (BUG2 in the link): https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… But I am in 20.0.1.dev4 There is a lot of logs (with nova)….. total 533340 -rw-r--r-- 1 42436 42436 0 juil. 17 06:42 nova-api.log -rw-r--r-- 1 42436 42436 105512247 juil. 17 06:42 nova-api.log.1 -rw-r--r-- 1 42436 42436 2460643 juil. 6 06:40 nova-api.log.2.gz -rw-r--r-- 1 42436 42436 7305818 juil. 5 06:31 nova-api.log.3.gz -rw-r--r-- 1 42436 42436 7234505 juil. 2 06:48 nova-api.log.4.gz -rw-r--r-- 1 42436 42436 2356307 juin 29 06:35 nova-api.log.5.gz -rw-r--r-- 1 42436 42436 7181854 juin 28 06:49 nova-api.log.6.gz -rw-r--r-- 1 42436 42436 56065234 juil. 18 09:06 nova-api-uwsgi.log -rw-r--r-- 1 42436 42436 113689390 juil. 17 06:42 nova-api-uwsgi.log.1 -rw-r--r-- 1 42436 42436 2150056 juil. 6 06:40 nova-api-uwsgi.log.2.gz -rw-r--r-- 1 42436 42436 6393602 juil. 5 06:31 nova-api-uwsgi.log.3.gz -rw-r--r-- 1 42436 42436 6383183 juil. 2 06:48 nova-api-uwsgi.log.4.gz -rw-r--r-- 1 42436 42436 2083376 juin 29 06:35 nova-api-uwsgi.log.5.gz -rw-r--r-- 1 42436 42436 6351824 juin 28 06:49 nova-api-uwsgi.log.6.gz -rw-r--r-- 1 42436 42436 1467596 juil. 18 09:06 nova-compute.log -rw-r--r-- 1 42436 42436 2626598 juil. 18 09:06 nova-conductor.log -rw-r--r-- 1 42436 42436 44883046 juil. 18 06:45 nova-conductor.log.1 -rw-r--r-- 1 42436 42436 7154 juil. 18 08:58 nova-manage.log -rw-r--r-- 1 42436 42436 221050 juil. 16 15:08 nova-metadata.log -rw-r--r-- 1 42436 42436 28094556 juil. 18 09:06 nova-metadata-uwsgi.log -rw-r--r-- 1 42436 42436 74174166 juil. 17 06:42 nova-metadata-uwsgi.log.1 -rw-r--r-- 1 42436 42436 5650339 juil. 5 06:31 nova-metadata-uwsgi.log.2.gz -rw-r--r-- 1 42436 42436 1861983 juin 29 06:35 nova-metadata-uwsgi.log.3.gz -rw-r--r-- 1 42436 42436 5601477 juin 27 06:53 nova-metadata-uwsgi.log.4.gz -rw-r--r-- 1 42436 42436 5640648 juin 21 06:36 nova-metadata-uwsgi.log.5.gz -rw-r--r-- 1 42436 42436 1871248 juin 15 06:43 nova-metadata-uwsgi.log.6.gz -rw-r--r-- 1 42436 42436 111880 juil. 16 15:08 nova-novncproxy.log -rw-r--r-- 1 42436 42436 2668833 juil. 18 09:06 nova-scheduler.log -rw-r--r-- 1 42436 42436 45997165 juil. 18 06:45 nova-scheduler.log.1 -rw-r--r-- 1 42436 42436 0 mai 15 08:47 privsep-helper.log 2 containers have problems: docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6231ee0f3c97 quay.io/openstack.kolla/nova-scheduler:2025.1-ubuntu-noble "dumb-init --single-…" 10 minutes ago Exited (1) 5 seconds ago nova_scheduler 57707a9d136e quay.io/openstack.kolla/nova-conductor:2025.1-ubuntu-noble "dumb-init --single-…" 39 hours ago Exited (1) 4 seconds ago nova_conductor Have a look to nova_conductor.log (same thing with nova_scheduler): 2025-07-18 09:08:25.170 7 ERROR nova.scheduler.client.report [-] Failed to initialize placement client (is keystone available?): openstack.exceptions.NotSupported: The placement service for 192.168.100.100:RegionOne exists but does not have any supported versions. 2025-07-18 09:08:25.171 7 ERROR nova.conductor.manager [-] Fatal error initializing placement client: The placement service for 192.168.100.100:RegionOne exists but does not have any supported versions.: openstack.exceptions.NotSupported: The placement service for 192.168.100.100:RegionOne exists but does not have any supported versions. 2025-07-18 09:08:25.172 7 CRITICAL nova [-] Unhandled error: openstack.exceptions.NotSupported: The placement service for 192.168.100.100:RegionOne exists but does not have any supported versions. 2025-07-18 09:08:25.172 7 ERROR nova Traceback (most recent call last): 2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/bin/nova-conductor", line 8, in <module> 2025-07-18 09:08:25.172 7 ERROR nova sys.exit(main()) 2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^ 2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/cmd/conductor.py", line 45, in main 2025-07-18 09:08:25.172 7 ERROR nova server = service.Service.create(binary='nova-conductor', 2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/service.py", line 256, in create 2025-07-18 09:08:25.172 7 ERROR nova service_obj = cls(host, binary, topic, manager, 2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/service.py", line 120, in __init__ 2025-07-18 09:08:25.172 7 ERROR nova self.manager = manager_class(host=self.host, *args, **kwargs) 2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/conductor/manager.py", line 124, in __init__ 2025-07-18 09:08:25.172 7 ERROR nova self.compute_task_mgr = ComputeTaskManager() 2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^ 2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/conductor/manager.py", line 255, in __init__ 2025-07-18 09:08:25.172 7 ERROR nova self.report_client 2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/conductor/manager.py", line 283, in report_client 2025-07-18 09:08:25.172 7 ERROR nova return report.report_client_singleton() 2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/scheduler/client/report.py", line 91, in report_client_singleton 2025-07-18 09:08:25.172 7 ERROR nova PLACEMENTCLIENT = SchedulerReportClient() 2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^^ 2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/scheduler/client/report.py", line 237, in __init__ 2025-07-18 09:08:25.172 7 ERROR nova self._client = self._create_client() 2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^ 2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/scheduler/client/report.py", line 280, in _create_client 2025-07-18 09:08:25.172 7 ERROR nova client = self._adapter or utils.get_sdk_adapter( 2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^ 2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/utils.py", line 1035, in get_sdk_adapter 2025-07-18 09:08:25.172 7 ERROR nova return getattr(conn, service_type.replace('-', '_')) 2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/openstack/service_description.py", line 88, in __get__ 2025-07-18 09:08:25.172 7 ERROR nova proxy = self._make_proxy(instance) 2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/openstack/service_description.py", line 290, in _make_proxy 2025-07-18 09:08:25.172 7 ERROR nova raise exceptions.NotSupported( 2025-07-18 09:08:25.172 7 ERROR nova openstack.exceptions.NotSupported: The placement service for 192.168.100.100:RegionOne exists but does not have any supported versions. 2025-07-18 09:08:25.172 7 ERROR nova Another thing is strange: Autre point bizarre: tail -f /var/log/kolla/placement/placement-api-uwsgi.log --- no python application found, check your startup logs for errors --- [pid: 1092|app: -1|req: -1/307186] 192.168.100.4 () {22 vars in 215 bytes} [Fri Jul 18 09:11:28 2025] GET / => generated 21 bytes in 0 msecs (HTTP/1.0 500) 3 headers in 102 bytes (0 switches on core 0) What can I do ? Can you help me to find a solution to restart my Openstack ? Thanks a lot if you can. Franck

3 7

[mariadb][keystone][blazar] Deadlock error on Galera cluster
by Winicius Allan 18 Jul '25

18 Jul '25

Hi community o/ Suddenly, I noticed that the login on the dashboard wasn't working, so I checked the keystone logs. I saw some entries like that: 2025-07-10 00:53:45.120 23 ERROR sqlalchemy.pool.impl.QueuePool oslo_db.exception.DBDeadlock: (pymysql.err.OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') Nova-conductor was also showing messages like these. At the same time, blazer-manager was logging after every _process_event calling: ERROR sqlalchemy.exc.TimeoutError: QueuePool limit of size 1 overflow 50 reached, connection timed out, timeout 30.00 I do not know if it's related to the _process_event method that batches events concurrently[1], and the DB was locked. Well... I increased the max_pool_size from 1 to 2 on blazar-manager, and that message stops showing. Does anyone know what can cause this deadlock generally? I'd like to ask another question: I have three controllers in my environment. Why do controllers with no active VIP address bind connections on the MySQL port, for example: tcp ESTAB 0 0 10.0.0.105:3306 10.0.0.107:50642 users:(("mariadbd",pid=977010,fd=1089)) [1] https://github.com/openstack/blazar/blob/stable/2024.1/blazar/manager/servi… Best regards.

3 5

[ops] Very basic newby questions
by Leroy Tennison 17 Jul '25

17 Jul '25

I am looking for a solution to automatically move (upon request) running VMs from one hypervisor to another. The current hypervisors are KVM/QEMU. Just looking at the documentation I am becoming overwhelmed. Questions (A URL which directly addresses the question is quite acceptable): Does OpenStack work with existing hypervisors or would existing hypervisors have to be "converted" (somehow) to OpenStack? If it can be used with existing hypervisors, is there a how-to for this specific task (I understand that OpenStack can do many things). If OpenStack can't be used with existing hypervisors, are you aware of solutions for this need? I have looked at ProxMox but it appears to want centralized storage and very fast networks which doesn't match the environment. Thank you for your help.

5 10