[openstack][rabbitmq]computes cannot register to nova 2025.0master
Hello. I deployed OpenStack 2025.1 Master with Kolla-Ansible, but the compute nodes cannot register with Nova. Error: Cannot update service status on host "compute05" due to an unexpected exception.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 3dad7dbe8ea9429c83958b98bccc7094 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver Traceback (most recent call last): 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/virt/libvirt/driver.py", line 5687, in _set_host_enabled 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver service = objects.Service.get_by_compute_host(ctx, CONF.host) 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File "/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_versionedobjects/base.py", line 175, in wrapper 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = cls.indirection_api.object_class_action_versions( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return cctxt.call(context, 'object_class_action_versions', 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File "/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/rpc/client.py", line 190, in call 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = self.transport._send( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File "/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/transport.py", line 123, in _send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return self._driver.send(target, ctxt, message, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File "/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 800, in send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File "/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 789, in _send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = self._waiter.wait(msg_id, timeout, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File "/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 654, in wait 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver message = self.waiters.get(msg_id, timeout=timeout) 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File "/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 520, in get 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver raise oslo_messaging.MessagingTimeout( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 3dad7dbe8ea9429c83958b98bccc7094 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver All services are in a "down" state, but this issue only occurs when using a three-controller cluster. When I use a single controller, everything works fine. My Env: OS: Ubuntu 22.04 OpenStack Version: 2025.1 Master Deployment Tool: Kolla-Ansible 19 Thank you. Nguyen Huu Khoi
This looks like your rabbit cluster is not ready. I haven’t worked with quorum queues yet, so my debugging advice is limited here. But I would start investigating rabbitmq. Zitat von Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com>:
Hello. I deployed OpenStack 2025.1 Master with Kolla-Ansible, but the compute nodes cannot register with Nova.
Error: Cannot update service status on host "compute05" due to an unexpected exception.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 3dad7dbe8ea9429c83958b98bccc7094 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver Traceback (most recent call last): 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/virt/libvirt/driver.py", line 5687, in _set_host_enabled 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver service = objects.Service.get_by_compute_host(ctx, CONF.host) 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File "/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_versionedobjects/base.py", line 175, in wrapper 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = cls.indirection_api.object_class_action_versions( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return cctxt.call(context, 'object_class_action_versions', 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File "/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/rpc/client.py", line 190, in call 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = self.transport._send( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File "/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/transport.py", line 123, in _send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return self._driver.send(target, ctxt, message, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File "/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 800, in send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File "/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 789, in _send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = self._waiter.wait(msg_id, timeout, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File "/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 654, in wait 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver message = self.waiters.get(msg_id, timeout=timeout) 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File "/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 520, in get 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver raise oslo_messaging.MessagingTimeout( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 3dad7dbe8ea9429c83958b98bccc7094 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver
All services are in a "down" state, but this issue only occurs when using a three-controller cluster. When I use a single controller, everything works fine.
My Env:
OS: Ubuntu 22.04
OpenStack Version: 2025.1 Master
Deployment Tool: Kolla-Ansible 19
Thank you.
Nguyen Huu Khoi
Hello.i get it, you r right. Ty. I go with 2024.2 ok. On Sun, Mar 23, 2025, 3:57 PM Eugen Block <eblock@nde.ag> wrote:
This looks like your rabbit cluster is not ready. I haven’t worked with quorum queues yet, so my debugging advice is limited here. But I would start investigating rabbitmq.
Zitat von Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com>:
Hello. I deployed OpenStack 2025.1 Master with Kolla-Ansible, but the compute nodes cannot register with Nova.
Error: Cannot update service status on host "compute05" due to an unexpected exception.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 3dad7dbe8ea9429c83958b98bccc7094 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver Traceback (most recent call last): 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/nova/virt/libvirt/driver.py",
line 5687, in _set_host_enabled 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver service = objects.Service.get_by_compute_host(ctx, CONF.host) 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_versionedobjects/base.py",
line 175, in wrapper 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = cls.indirection_api.object_class_action_versions( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/nova/conductor/rpcapi.py",
line 240, in object_class_action_versions 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return cctxt.call(context, 'object_class_action_versions', 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/rpc/client.py",
line 190, in call 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = self.transport._send( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/transport.py",
line 123, in _send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return self._driver.send(target, ctxt, message, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 800, in send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 789, in _send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = self._waiter.wait(msg_id, timeout, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 654, in wait 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver message = self.waiters.get(msg_id, timeout=timeout) 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 520, in get 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver raise oslo_messaging.MessagingTimeout( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 3dad7dbe8ea9429c83958b98bccc7094 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver
All services are in a "down" state, but this issue only occurs when using a three-controller cluster. When I use a single controller, everything works fine.
My Env:
OS: Ubuntu 22.04
OpenStack Version: 2025.1 Master
Deployment Tool: Kolla-Ansible 19
Thank you.
Nguyen Huu Khoi
Hello. After doing some tests, I see that rabbitmq v3.13.7 cannot create cluster properly but v3.11.28 works well. Nguyen Huu Khoi On Sun, Mar 23, 2025 at 4:47 PM Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hello.i get it, you r right. Ty. I go with 2024.2 ok.
On Sun, Mar 23, 2025, 3:57 PM Eugen Block <eblock@nde.ag> wrote:
This looks like your rabbit cluster is not ready. I haven’t worked with quorum queues yet, so my debugging advice is limited here. But I would start investigating rabbitmq.
Zitat von Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com>:
Hello. I deployed OpenStack 2025.1 Master with Kolla-Ansible, but the compute nodes cannot register with Nova.
Error: Cannot update service status on host "compute05" due to an unexpected exception.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 3dad7dbe8ea9429c83958b98bccc7094 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver Traceback (most recent call last): 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/nova/virt/libvirt/driver.py",
line 5687, in _set_host_enabled 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver service = objects.Service.get_by_compute_host(ctx, CONF.host) 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_versionedobjects/base.py",
line 175, in wrapper 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = cls.indirection_api.object_class_action_versions( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/nova/conductor/rpcapi.py",
line 240, in object_class_action_versions 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return cctxt.call(context, 'object_class_action_versions', 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/rpc/client.py",
line 190, in call 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = self.transport._send( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/transport.py",
line 123, in _send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return self._driver.send(target, ctxt, message, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 800, in send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 789, in _send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = self._waiter.wait(msg_id, timeout, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 654, in wait 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver message = self.waiters.get(msg_id, timeout=timeout) 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 520, in get 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver raise oslo_messaging.MessagingTimeout( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 3dad7dbe8ea9429c83958b98bccc7094 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver
All services are in a "down" state, but this issue only occurs when using a three-controller cluster. When I use a single controller, everything works fine.
My Env:
OS: Ubuntu 22.04
OpenStack Version: 2025.1 Master
Deployment Tool: Kolla-Ansible 19
Thank you.
Nguyen Huu Khoi
Alright, that's good. Do you have more details on what goes wrong? Have you checked launchpad for existing bugs? Or maybe this is a new bug that you could report. I'm not there yet, I just upgraded our own version to Zed last Friday, so it's going to take a couple more weeks until I get to the latest version. Zitat von Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com>:
Hello. After doing some tests, I see that rabbitmq v3.13.7 cannot create cluster properly but v3.11.28 works well.
Nguyen Huu Khoi
On Sun, Mar 23, 2025 at 4:47 PM Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hello.i get it, you r right. Ty. I go with 2024.2 ok.
On Sun, Mar 23, 2025, 3:57 PM Eugen Block <eblock@nde.ag> wrote:
This looks like your rabbit cluster is not ready. I haven’t worked with quorum queues yet, so my debugging advice is limited here. But I would start investigating rabbitmq.
Zitat von Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com>:
Hello. I deployed OpenStack 2025.1 Master with Kolla-Ansible, but the compute nodes cannot register with Nova.
Error: Cannot update service status on host "compute05" due to an unexpected exception.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 3dad7dbe8ea9429c83958b98bccc7094 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver Traceback (most recent call last): 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/nova/virt/libvirt/driver.py",
line 5687, in _set_host_enabled 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver service = objects.Service.get_by_compute_host(ctx, CONF.host) 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_versionedobjects/base.py",
line 175, in wrapper 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = cls.indirection_api.object_class_action_versions( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/nova/conductor/rpcapi.py",
line 240, in object_class_action_versions 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return cctxt.call(context, 'object_class_action_versions', 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/rpc/client.py",
line 190, in call 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = self.transport._send( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/transport.py",
line 123, in _send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return self._driver.send(target, ctxt, message, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 800, in send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 789, in _send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = self._waiter.wait(msg_id, timeout, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 654, in wait 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver message = self.waiters.get(msg_id, timeout=timeout) 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 520, in get 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver raise oslo_messaging.MessagingTimeout( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 3dad7dbe8ea9429c83958b98bccc7094 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver
All services are in a "down" state, but this issue only occurs when using a three-controller cluster. When I use a single controller, everything works fine.
My Env:
OS: Ubuntu 22.04
OpenStack Version: 2025.1 Master
Deployment Tool: Kolla-Ansible 19
Thank you.
Nguyen Huu Khoi
I'm going to upgrade from Zed to Antelope in a few weeks, so if you have more details about what went wrong and if you can share with us, I'd appreciate it. Em seg., 24 de mar. de 2025 às 04:32, Eugen Block <eblock@nde.ag> escreveu:
Alright, that's good. Do you have more details on what goes wrong? Have you checked launchpad for existing bugs? Or maybe this is a new bug that you could report. I'm not there yet, I just upgraded our own version to Zed last Friday, so it's going to take a couple more weeks until I get to the latest version.
Zitat von Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com>:
Hello. After doing some tests, I see that rabbitmq v3.13.7 cannot create cluster properly but v3.11.28 works well.
Nguyen Huu Khoi
On Sun, Mar 23, 2025 at 4:47 PM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Hello.i get it, you r right. Ty. I go with 2024.2 ok.
On Sun, Mar 23, 2025, 3:57 PM Eugen Block <eblock@nde.ag> wrote:
This looks like your rabbit cluster is not ready. I haven’t worked with quorum queues yet, so my debugging advice is limited here. But I would start investigating rabbitmq.
Zitat von Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com>:
Hello. I deployed OpenStack 2025.1 Master with Kolla-Ansible, but the compute nodes cannot register with Nova.
Error: Cannot update service status on host "compute05" due to an unexpected exception.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 3dad7dbe8ea9429c83958b98bccc7094 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver Traceback (most recent call last): 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/nova/virt/libvirt/driver.py",
line 5687, in _set_host_enabled 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver service = objects.Service.get_by_compute_host(ctx, CONF.host) 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_versionedobjects/base.py",
line 175, in wrapper 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = cls.indirection_api.object_class_action_versions( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/nova/conductor/rpcapi.py",
line 240, in object_class_action_versions 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return cctxt.call(context, 'object_class_action_versions', 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/rpc/client.py",
line 190, in call 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = self.transport._send( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/transport.py",
line 123, in _send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return self._driver.send(target, ctxt, message, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 800, in send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 789, in _send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = self._waiter.wait(msg_id, timeout, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 654, in wait 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver message = self.waiters.get(msg_id, timeout=timeout) 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 520, in get 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver raise oslo_messaging.MessagingTimeout( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 3dad7dbe8ea9429c83958b98bccc7094 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver
All services are in a "down" state, but this issue only occurs when using a three-controller cluster. When I use a single controller, everything works fine.
My Env:
OS: Ubuntu 22.04
OpenStack Version: 2025.1 Master
Deployment Tool: Kolla-Ansible 19
Thank you.
Nguyen Huu Khoi
Hi, I am still not sure if others encounter this bug like me. I see that problem happened with v4 https://github.com/rabbitmq/rabbitmq-server/discussions/13257 and one more thing, I testest with rabbitmq cluster with kolla-ansible deployment not on bare metal. we need make sure this point. Nguyen Huu Khoi On Mon, Mar 24, 2025 at 2:32 PM Eugen Block <eblock@nde.ag> wrote:
Alright, that's good. Do you have more details on what goes wrong? Have you checked launchpad for existing bugs? Or maybe this is a new bug that you could report. I'm not there yet, I just upgraded our own version to Zed last Friday, so it's going to take a couple more weeks until I get to the latest version.
Zitat von Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com>:
Hello. After doing some tests, I see that rabbitmq v3.13.7 cannot create cluster properly but v3.11.28 works well.
Nguyen Huu Khoi
On Sun, Mar 23, 2025 at 4:47 PM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Hello.i get it, you r right. Ty. I go with 2024.2 ok.
On Sun, Mar 23, 2025, 3:57 PM Eugen Block <eblock@nde.ag> wrote:
This looks like your rabbit cluster is not ready. I haven’t worked with quorum queues yet, so my debugging advice is limited here. But I would start investigating rabbitmq.
Zitat von Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com>:
Hello. I deployed OpenStack 2025.1 Master with Kolla-Ansible, but the compute nodes cannot register with Nova.
Error: Cannot update service status on host "compute05" due to an unexpected exception.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 3dad7dbe8ea9429c83958b98bccc7094 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver Traceback (most recent call last): 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/nova/virt/libvirt/driver.py",
line 5687, in _set_host_enabled 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver service = objects.Service.get_by_compute_host(ctx, CONF.host) 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_versionedobjects/base.py",
line 175, in wrapper 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = cls.indirection_api.object_class_action_versions( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/nova/conductor/rpcapi.py",
line 240, in object_class_action_versions 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return cctxt.call(context, 'object_class_action_versions', 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/rpc/client.py",
line 190, in call 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = self.transport._send( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/transport.py",
line 123, in _send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return self._driver.send(target, ctxt, message, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 800, in send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 789, in _send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = self._waiter.wait(msg_id, timeout, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 654, in wait 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver message = self.waiters.get(msg_id, timeout=timeout) 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 520, in get 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver raise oslo_messaging.MessagingTimeout( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 3dad7dbe8ea9429c83958b98bccc7094 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver
All services are in a "down" state, but this issue only occurs when using a three-controller cluster. When I use a single controller, everything works fine.
My Env:
OS: Ubuntu 22.04
OpenStack Version: 2025.1 Master
Deployment Tool: Kolla-Ansible 19
Thank you.
Nguyen Huu Khoi
Hello. Sorry, my bad, it happened because time mismatch on my node OS. Nguyen Huu Khoi On Tue, Mar 25, 2025 at 6:07 AM Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hi,
I am still not sure if others encounter this bug like me. I see that problem happened with v4
https://github.com/rabbitmq/rabbitmq-server/discussions/13257
and one more thing, I testest with rabbitmq cluster with kolla-ansible deployment not on bare metal. we need make sure this point.
Nguyen Huu Khoi
On Mon, Mar 24, 2025 at 2:32 PM Eugen Block <eblock@nde.ag> wrote:
Alright, that's good. Do you have more details on what goes wrong? Have you checked launchpad for existing bugs? Or maybe this is a new bug that you could report. I'm not there yet, I just upgraded our own version to Zed last Friday, so it's going to take a couple more weeks until I get to the latest version.
Zitat von Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com>:
Hello. After doing some tests, I see that rabbitmq v3.13.7 cannot create cluster properly but v3.11.28 works well.
Nguyen Huu Khoi
On Sun, Mar 23, 2025 at 4:47 PM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Hello.i get it, you r right. Ty. I go with 2024.2 ok.
On Sun, Mar 23, 2025, 3:57 PM Eugen Block <eblock@nde.ag> wrote:
This looks like your rabbit cluster is not ready. I haven’t worked with quorum queues yet, so my debugging advice is limited here. But I would start investigating rabbitmq.
Zitat von Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com>:
Hello. I deployed OpenStack 2025.1 Master with Kolla-Ansible, but the compute nodes cannot register with Nova.
Error: Cannot update service status on host "compute05" due to an unexpected exception.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 3dad7dbe8ea9429c83958b98bccc7094 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver Traceback (most recent call last): 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/nova/virt/libvirt/driver.py",
line 5687, in _set_host_enabled 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver service = objects.Service.get_by_compute_host(ctx, CONF.host) 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_versionedobjects/base.py",
line 175, in wrapper 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = cls.indirection_api.object_class_action_versions( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/nova/conductor/rpcapi.py",
line 240, in object_class_action_versions 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return cctxt.call(context, 'object_class_action_versions', 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/rpc/client.py",
line 190, in call 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = self.transport._send( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/transport.py",
line 123, in _send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return self._driver.send(target, ctxt, message, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 800, in send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 789, in _send 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver result = self._waiter.wait(msg_id, timeout, 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 654, in wait 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver message = self.waiters.get(msg_id, timeout=timeout) 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver File
"/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 520, in get 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver raise oslo_messaging.MessagingTimeout( 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 3dad7dbe8ea9429c83958b98bccc7094 2025-03-23 10:31:26.974 7 ERROR nova.virt.libvirt.driver
All services are in a "down" state, but this issue only occurs when using a three-controller cluster. When I use a single controller, everything works fine.
My Env:
OS: Ubuntu 22.04
OpenStack Version: 2025.1 Master
Deployment Tool: Kolla-Ansible 19
Thank you.
Nguyen Huu Khoi
participants (3)
-
Eugen Block
-
Nguyễn Hữu Khôi
-
Winicius Allan