<div dir="ltr">Hello. Sorry for that.<div>I just want to notice that both nova and cinder have this problem,</div><div>When diving to logs on both service I see:</div><div>ERROR oslo.messaging._drivers.impl_rabbit [-] [8634b511-7eee-4e50-8efd-b96d420e9914] AMQP server on [node was down]:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error><br></div><div><br></div><div>and </div><div><br></div><div><br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task Traceback (most recent call last):<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 441, in get<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return self._queues[msg_id].get(block=True, timeout=timeout)<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/queue.py", line 322, in get<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return waiter.wait()<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/queue.py", line 141, in wait<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return get_hub().switch()<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 313, in switch2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return self.greenlet.switch()<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task _queue.Empty<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task <br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task During handling of the above exception, another exception occurred:<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task <br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task Traceback (most recent call last):<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task task(self, context)<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 9716, in _sync_power_states<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task db_instances = objects.InstanceList.get_by_host(context, self.host,<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_versionedobjects/base.py", line 175, in wrapper<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task result = cls.indirection_api.object_class_action_versions(<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return cctxt.call(context, 'object_class_action_versions',<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/client.py", line 189, in call<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task result = self.transport._send(<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/transport.py", line 123, in _send<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return self._driver.send(target, ctxt, message,<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return self._send(target, ctxt, message, wait_for_reply, timeout,<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 678, in _send<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task result = self._waiter.wait(msg_id, timeout,<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task message = self.waiters.get(msg_id, timeout=timeout)<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 443, in get<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task raise oslo_messaging.MessagingTimeout(<br>2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID c8a676a9709242908dcff97046d7976d<br><br>*** I use cluster rabbitmq with ha-policy for exchange and queue. These logs are gone when I restart cinder and nova services.</div><div><br><br></div><div><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Nguyen Huu Khoi<br></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Oct 24, 2022 at 5:42 PM Eugen Block <<a href="mailto:eblock@nde.ag">eblock@nde.ag</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">You don't need to create a new thread with the same issue.<br>
Do the rabbitmq logs reveal anything? We create a cluster within <br>
rabbitmq and the output looks like this:<br>
<br>
---snip---<br>
control01:~ # rabbitmqctl cluster_status<br>
Cluster status of node rabbit@control01 ...<br>
Basics<br>
<br>
Cluster name: rabbit@rabbitmq-cluster<br>
<br>
Disk Nodes<br>
<br>
rabbit@control01<br>
rabbit@control02<br>
rabbit@control03<br>
<br>
Running Nodes<br>
<br>
rabbit@control01<br>
rabbit@control02<br>
rabbit@control03<br>
<br>
Versions<br>
<br>
rabbit@control01: RabbitMQ 3.8.3 on Erlang 22.2.7<br>
rabbit@control02: RabbitMQ 3.8.3 on Erlang 22.2.7<br>
rabbit@control03: RabbitMQ 3.8.3 on Erlang 22.2.7<br>
---snip---<br>
<br>
During failover it's not unexpected that a message gets lost, but it <br>
should be resent, I believe. How is your openstack deployed?<br>
<br>
<br>
Zitat von Nguyễn Hữu Khôi <<a href="mailto:nguyenhuukhoinw@gmail.com" target="_blank">nguyenhuukhoinw@gmail.com</a>>:<br>
<br>
> Hello.<br>
> 2 remain nodes still running, here is my output:<br>
> Basics<br>
><br>
> Cluster name: rabbit@controller01<br>
><br>
> Disk Nodes<br>
><br>
> rabbit@controller01<br>
> rabbit@controller02<br>
> rabbit@controller03<br>
><br>
> Running Nodes<br>
><br>
> rabbit@controller01<br>
> rabbit@controller03<br>
><br>
> Versions<br>
><br>
> rabbit@controller01: RabbitMQ 3.8.35 on Erlang 23.3.4.18<br>
> rabbit@controller03: RabbitMQ 3.8.35 on Erlang 23.3.4.18<br>
><br>
> Maintenance status<br>
><br>
> Node: rabbit@controller01, status: not under maintenance<br>
> Node: rabbit@controller03, status: not under maintenance<br>
><br>
> Alarms<br>
><br>
> (none)<br>
><br>
> Network Partitions<br>
><br>
> (none)<br>
><br>
> Listeners<br>
><br>
> Node: rabbit@controller01, interface: [::], port: 15672, protocol: http,<br>
> purpose: HTTP API<br>
> Node: rabbit@controller01, interface: 183.81.13.227, port: 25672, protocol:<br>
> clustering, purpose: inter-node and CLI tool communication<br>
> Node: rabbit@controller01, interface: 183.81.13.227, port: 5672, protocol:<br>
> amqp, purpose: AMQP 0-9-1 and AMQP 1.0<br>
> Node: rabbit@controller03, interface: [::], port: 15672, protocol: http,<br>
> purpose: HTTP API<br>
> Node: rabbit@controller03, interface: 183.81.13.229, port: 25672, protocol:<br>
> clustering, purpose: inter-node and CLI tool communication<br>
> Node: rabbit@controller03, interface: 183.81.13.229, port: 5672, protocol:<br>
> amqp, purpose: AMQP 0-9-1 and AMQP 1.0<br>
><br>
> Feature flags<br>
><br>
> Flag: drop_unroutable_metric, state: enabled<br>
> Flag: empty_basic_get_metric, state: enabled<br>
> Flag: implicit_default_bindings, state: enabled<br>
> Flag: maintenance_mode_status, state: enabled<br>
> Flag: quorum_queue, state: enabled<br>
> Flag: user_limits, state: enabled<br>
> Flag: virtual_host_metadata, state: enabled<br>
><br>
> I used ha_queues mode all<br>
> But it is not better.<br>
> Nguyen Huu Khoi<br>
><br>
><br>
> On Tue, Oct 18, 2022 at 7:19 AM Nguyễn Hữu Khôi <<a href="mailto:nguyenhuukhoinw@gmail.com" target="_blank">nguyenhuukhoinw@gmail.com</a>><br>
> wrote:<br>
><br>
>> Description<br>
>> ===========<br>
>> I set up 3 controllers and 3 compute nodes. My system cannot work well<br>
>> when 1 rabbit node in cluster rabbitmq is down, cannot launch instances. It<br>
>> stucked at scheduling.<br>
>><br>
>> Steps to reproduce<br>
>> ===========<br>
>> Openstack nodes point rabbit://node1:5672,node2:5672,node3:5672//<br>
>> * Reboot 1 of 3 rabbitmq node.<br>
>> * Create instances then it stucked at scheduling.<br>
>><br>
>> Workaround<br>
>> ===========<br>
>> Point to rabbitmq VIP address. But We cannot share the load with this<br>
>> solution. Please give me some suggestions. Thank you very much.<br>
>> I did google and enabled system log's debug but I still cannot understand<br>
>> why.<br>
>><br>
>> Nguyen Huu Khoi<br>
>><br>
<br>
<br>
<br>
<br>
</blockquote></div>