<div dir="ltr"><div><p style="margin:0px 0px 1.2em;padding:0px;width:auto;max-width:45em;color:rgb(51,51,51);font-family:monospace;font-size:12px">Title:<br>  Openstack cluster cannot create when 1 of 3 rabbitmq cluster<br>  node down</p><p style="margin:0px 0px 1.2em;padding:0px;width:auto;max-width:45em;color:rgb(51,51,51);font-family:monospace;font-size:12px">Bug description:<br>  Description<br>  ===========<br>  I set up 3 controllers and 3 compute nodes. My system cannot work when 1 rabbit node in cluster rabbitmq is down, cannot create volume or launch instance. It stucked at creating and scheduling respectively.</p><p style="margin:0px 0px 1.2em;padding:0px;width:auto;max-width:45em;color:rgb(51,51,51);font-family:monospace;font-size:12px">  Steps to reproduce<br>  ===========<br>  Openstack nodes point rabbit://node1:5672,node2:5672,node3:5672//<br>  * Reboot 1 of 3 rabbitmq node.<br>  * Create volume or launch instance then it stucked at creating and scheduling respectively.</p><p style="margin:0px 0px 1.2em;padding:0px;width:auto;max-width:45em;color:rgb(51,51,51);font-family:monospace;font-size:12px">  Workaround<br>  ===========<br>  I need reboot cinder and nova services to create volume and launch instance .</p><p style="margin:0px 0px 1.2em;padding:0px;width:auto;max-width:45em;color:rgb(51,51,51);font-family:monospace;font-size:12px">More Info:<br> I see in cinder_scheduler, it looks like cinder cannot change to another rabbitmq node. I hope we have ideas for that..</p><p id="gmail-yui_3_10_3_1_1666604240270_2068" style="margin:0px 0px 1.2em;padding:0px;width:auto;max-width:45em;color:rgb(51,51,51);font-family:monospace;font-size:12px">2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task Traceback (most recent call last):<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 441, in get<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return self._queues[msg_id].get(block=True, timeout=timeout)<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/queue.py", line 322, in get<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return waiter.wait()<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/queue.py", line 141, in wait<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return get_hub().switch()<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 313, in switch2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return self.greenlet.switch()<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task _queue.Empty<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task During handling of the above exception, another exception occurred:<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task Traceback (most recent call last):<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task task(self, context)<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 9716, in _sync_power_states<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task db_instances = objects.InstanceList.get_by_host(context, self.host,<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_versionedobjects/base.py", line 175, in wrapper<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task result = cls.indirection_api.object_class_action_versions(<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return cctxt.call(context, 'object_class_action_versions',<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/client.py", line 189, in call<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task result = self.transport._send(<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/transport.py", line 123, in _send<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return self._driver.send(target, ctxt, message,<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return self._send(target, ctxt, message, wait_for_reply, timeout,<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 678, in _send<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task result = self._waiter.wait(msg_id, timeout,<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task message = self.waiters.get(msg_id, timeout=timeout)<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 443, in get<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task raise oslo_messaging.MessagingTimeout(<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID c8a676a9709242908dcff97046d7976d</p></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Nguyen Huu Khoi<br></div></div></div></div>