Open Stack

Title:
  Openstack cluster cannot create when 1 of 3 rabbitmq cluster
  node down

Bug description:
  Description
  ===========
  I set up 3 controllers and 3 compute nodes. My system cannot work when 1
rabbit node in cluster rabbitmq is down, cannot create volume or launch
instance. It stucked at creating and scheduling respectively.

  Steps to reproduce
  ===========
  Openstack nodes point rabbit://node1:5672,node2:5672,node3:5672//
  * Reboot 1 of 3 rabbitmq node.
  * Create volume or launch instance then it stucked at creating and
scheduling respectively.

  Workaround
  ===========
  I need reboot cinder and nova services to create volume and launch
instance .

More Info:
 I see in cinder_scheduler, it looks like cinder cannot change to another
rabbitmq node. I hope we have ideas for that..

2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task Traceback (most
recent call last):
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File
"/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 441, in get
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return
self._queues[msg_id].get(block=True, timeout=timeout)
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File
"/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/queue.py", line
322, in get
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return
waiter.wait()
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File
"/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/queue.py", line
141, in wait
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return
get_hub().switch()
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File
"/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/hubs/hub.py",
line 313, in switch2022-10-24 14:23:01.945 7 ERROR
oslo_service.periodic_task return self.greenlet.switch()
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task _queue.Empty
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task During handling
of the above exception, another exception occurred:
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task Traceback (most
recent call last):
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File
"/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_service/periodic_task.py",
line 216, in run_periodic_tasks
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task task(self,
context)
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File
"/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py",
line 9716, in _sync_power_states
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task db_instances =
objects.InstanceList.get_by_host(context, self.host,
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File
"/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_versionedobjects/base.py",
line 175, in wrapper
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task result =
cls.indirection_api.object_class_action_versions(
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File
"/var/lib/kolla/venv/lib/python3.8/site-packages/nova/conductor/rpcapi.py",
line 240, in object_class_action_versions
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return
cctxt.call(context, 'object_class_action_versions',
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File
"/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/client.py",
line 189, in call
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task result =
self.transport._send(
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File
"/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/transport.py",
line 123, in _send
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return
self._driver.send(target, ctxt, message,
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File
"/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 689, in send
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return
self._send(target, ctxt, message, wait_for_reply, timeout,
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File
"/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 678, in _send
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task result =
self._waiter.wait(msg_id, timeout,
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File
"/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 567, in wait
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task message =
self.waiters.get(msg_id, timeout=timeout)
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File
"/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 443, in get
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task raise
oslo_messaging.MessagingTimeout(
2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task
oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply
to message ID c8a676a9709242908dcff97046d7976d
Nguyen Huu Khoi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20221024/9b51a645/attachment.htm>