[nova-compute] Failed to publish message to topic 'scheduler_fanout'
Hi, With Ussuri, failed to launch VM and saw such error from nova-compute. Didn't see any error in nova-api, nova-conductor and nova-scheduler. Did some search and not found any similar cases. Wondering anyone has ever hit this issue? ================================================= 2020-12-09 13:24:29.333 8 ERROR oslo.messaging._drivers.impl_rabbit [req-4226ac03-a5b7-446a-a74b-2785f9818927 - - - - -] [3830bc00-1541-4951-b4ed-e73faafaa78c] AMQP server on 10.6.20.21:5672 is unreachable: connection already closed. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: connection already closed 2020-12-09 13:24:30.361 8 INFO oslo.messaging._drivers.impl_rabbit [req-4226ac03-a5b7-446a-a74b-2785f9818927 - - - - -] [3830bc00-1541-4951-b4ed-e73faafaa78c] Reconnected to AMQP server on 10.6.20.21:5672 via [amqp] client with port 55524. 2020-12-09 13:26:28.377 8 ERROR oslo.messaging._drivers.impl_rabbit [req-4226ac03-a5b7-446a-a74b-2785f9818927 - - - - -] Failed to publish message to topic 'scheduler_fanout': : amqp.exceptions.MessageNacked 2020-12-09 13:26:28.377 8 ERROR oslo.messaging._drivers.impl_rabbit [req-4226ac03-a5b7-446a-a74b-2785f9818927 - - - - -] Unable to connect to AMQP server on 10.6.20.21:5672 after inf tries: : amqp.exceptions.MessageNacked 2020-12-09 13:26:28.378 8 ERROR oslo_service.periodic_task [req-4226ac03-a5b7-446a-a74b-2785f9818927 - - - - -] Error during ComputeManager._sync_scheduler_instance_info: oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.6.20.21:5672 after inf tries: ================================================= Thanks! Tony
Hi, apparently nova can't reach rabbitmq, I would check rabbit logs and 'rabbitmqctl cluster_status'. Regards, Eugen Zitat von Tony Liu <tonyliu0592@hotmail.com>:
Hi,
With Ussuri, failed to launch VM and saw such error from nova-compute. Didn't see any error in nova-api, nova-conductor and nova-scheduler. Did some search and not found any similar cases. Wondering anyone has ever hit this issue? ================================================= 2020-12-09 13:24:29.333 8 ERROR oslo.messaging._drivers.impl_rabbit [req-4226ac03-a5b7-446a-a74b-2785f9818927 - - - - -] [3830bc00-1541-4951-b4ed-e73faafaa78c] AMQP server on 10.6.20.21:5672 is unreachable: connection already closed. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: connection already closed 2020-12-09 13:24:30.361 8 INFO oslo.messaging._drivers.impl_rabbit [req-4226ac03-a5b7-446a-a74b-2785f9818927 - - - - -] [3830bc00-1541-4951-b4ed-e73faafaa78c] Reconnected to AMQP server on 10.6.20.21:5672 via [amqp] client with port 55524. 2020-12-09 13:26:28.377 8 ERROR oslo.messaging._drivers.impl_rabbit [req-4226ac03-a5b7-446a-a74b-2785f9818927 - - - - -] Failed to publish message to topic 'scheduler_fanout': : amqp.exceptions.MessageNacked 2020-12-09 13:26:28.377 8 ERROR oslo.messaging._drivers.impl_rabbit [req-4226ac03-a5b7-446a-a74b-2785f9818927 - - - - -] Unable to connect to AMQP server on 10.6.20.21:5672 after inf tries: : amqp.exceptions.MessageNacked 2020-12-09 13:26:28.378 8 ERROR oslo_service.periodic_task [req-4226ac03-a5b7-446a-a74b-2785f9818927 - - - - -] Error during ComputeManager._sync_scheduler_instance_info: oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.6.20.21:5672 after inf tries: =================================================
Thanks! Tony
Connection went down, not sure about the cause, no error from server side. Then successful reconnection followed by nack. Enabled debug in logging for rabbitmq, but restart rabbitmq cluster brought everything back to normal. Will keep logging level on debug and wait for the issue happens again. Thanks! Tony
-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Thursday, December 10, 2020 2:11 AM To: openstack-discuss@lists.openstack.org Subject: Re: [nova-compute] Failed to publish message to topic 'scheduler_fanout'
Hi,
apparently nova can't reach rabbitmq, I would check rabbit logs and 'rabbitmqctl cluster_status'.
Regards, Eugen
Zitat von Tony Liu <tonyliu0592@hotmail.com>:
Hi,
With Ussuri, failed to launch VM and saw such error from nova-compute. Didn't see any error in nova-api, nova-conductor and nova-scheduler. Did some search and not found any similar cases. Wondering anyone has ever hit this issue? ================================================= 2020-12-09 13:24:29.333 8 ERROR oslo.messaging._drivers.impl_rabbit [req-4226ac03-a5b7-446a-a74b-2785f9818927 - - - - -] [3830bc00-1541-4951-b4ed-e73faafaa78c] AMQP server on 10.6.20.21:5672 is unreachable: connection already closed. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: connection already closed 2020-12-09 13:24:30.361 8 INFO oslo.messaging._drivers.impl_rabbit [req-4226ac03-a5b7-446a-a74b-2785f9818927 - - - - -] [3830bc00-1541-4951-b4ed-e73faafaa78c] Reconnected to AMQP server on 10.6.20.21:5672 via [amqp] client with port 55524. 2020-12-09 13:26:28.377 8 ERROR oslo.messaging._drivers.impl_rabbit [req-4226ac03-a5b7-446a-a74b-2785f9818927 - - - - -] Failed to publish message to topic 'scheduler_fanout': : amqp.exceptions.MessageNacked 2020-12-09 13:26:28.377 8 ERROR oslo.messaging._drivers.impl_rabbit [req-4226ac03-a5b7-446a-a74b-2785f9818927 - - - - -] Unable to connect to AMQP server on 10.6.20.21:5672 after inf tries: : amqp.exceptions.MessageNacked 2020-12-09 13:26:28.378 8 ERROR oslo_service.periodic_task [req-4226ac03-a5b7-446a-a74b-2785f9818927 - - - - -] Error during ComputeManager._sync_scheduler_instance_info: oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.6.20.21:5672 after inf tries: =================================================
Thanks! Tony
participants (2)
-
Eugen Block
-
Tony Liu