Hi,

I cannot live-migrate any VM I have.

All my compute nodes are enabled. I have a few VMs, I am in the testing phase.

I tried to live-migrate toward an empty node, then toward a node which already had a VM in it. It is the same problem.

In the log files I have :

Source node :

Source

==> /var/log/kolla/nova/nova-compute.log <==
2023-04-09 11:32:49.419 7 INFO nova.compute.resource_tracker [req-4e91d562-d7ad-4585-a827-3db6277fd81e - - - - -] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from migration f60eb077-8e1c-404a-aca5-64e5609ca02c
2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [-] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Pre live migration failed at computehci02: oslo_messaging.rpc.client.RemoteError: Remote error: ClientException Gateway Time-out (HTTP 504)
['Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 72, in wrapped\n context, exc, binary)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", line 1439, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 213, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 201, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8367, in pre_live_migration\n bdm.save()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8304, in pre_live_migration\n connector=connector, mountpoint=bdm.device_name)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 433, in wrapper\n res = method(self, ctx, volume_id, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 801, in attachment_create\n volume_id, _connector, instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", line 421, in substitution\n return method.func(obj, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 39, in create\n retval = self._create(\'/attachments\', body, \'attachment\')\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", line 306, in _create\n resp, body = self.api.client.post(url, body=body)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 215, in _cs_request\n return self.request(url, method, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 201, in request\n raise exceptions.from_response(resp, body)\n', 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n'].
2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Traceback (most recent call last):
2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8501, in _do_pre_live_migration_from_source
2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] block_migration, disk, dest, migrate_data)
2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/rpcapi.py", line 948, in pre_live_migration
2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] **msg_args)
2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 192, in call
2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] retry=self.retry, transport_options=self.transport_options)
2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128, in _send
2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] transport_options=transport_options)
2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 691, in send
2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] transport_options=transport_options)
2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send
2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] raise result
2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] oslo_messaging.rpc.client.RemoteError: Remote error: ClientException Gateway Time-out (HTTP 504)
2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] ['Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 72, in wrapped\n context, exc, binary)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", line 1439, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 213, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 201, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8367, in pre_live_migration\n bdm.save()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8304, in pre_live_migration\n connector=connector, mountpoint=bdm.device_name)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 433, in wrapper\n res = method(self, ctx, volume_id, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 801, in attachment_create\n volume_id, _connector, instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", line 421, in substitution\n return method.func(obj, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 39, in create\n retval = self._create(\'/attachments\', body, \'attachment\')\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", line 306, in _create\n resp, body = self.api.client.post(url, body=body)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 215, in _cs_request\n return self.request(url, method, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 201, in request\n raise exceptions.from_response(resp, body)\n', 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n'].
2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7]
2023-04-09 11:33:30.157 7 INFO nova.compute.manager [-] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Swapping old allocation on dict_keys(['2857ae62-3519-4b13-b700-858c827dd78d']) held by migration f60eb077-8e1c-404a-aca5-64e5609ca02c for instance

In the destination node :

nova-compute.log
2023-04-09 11:32:30.052 7 WARNING os_brick.initiator.connectors.nvmeof [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error in _get_host_uuid: Unexpected error while running command.
Command: blkid overlay -s UUID -o value
Exit code: 2
Stdout: ''
Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.

2023-04-09 11:33:05.354 7 INFO nova.compute.resource_tracker [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from migration f60eb077-8e1c-404a-aca5-64e5609ca02c
2023-04-09 11:33:05.392 7 WARNING nova.compute.resource_tracker [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] Instance f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 has been moved to another host computehci01(computehci01). There are allocations remaining against the source host that might need to be removed: {'resources': {'VCPU': 1, 'MEMORY_MB': 2048}}.
2023-04-09 11:33:30.086 7 ERROR nova.volume.cinder [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Create attachment failed for volume 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb. Error: Gateway Time-out (HTTP 504) Code: 504: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Exception during message handling: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 72, in wrapped
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server context, exc, binary)
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 63, in wrapped
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", line 1439, in decorated_function
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 213, in decorated_function
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 201, in decorated_function
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8367, in pre_live_migration
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server bdm.save()
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8304, in pre_live_migration
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server connector=connector, mountpoint=bdm.device_name)
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 395, in wrapper
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = method(self, ctx, *args, **kwargs)
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 433, in wrapper
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = method(self, ctx, volume_id, *args, **kwargs)
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 813, in attachment_create
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server instance_uuid=instance_id)
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 801, in attachment_create
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server volume_id, _connector, instance_id)
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", line 421, in substitution
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return method.func(obj, *args, **kwargs)
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 39, in create
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server retval = self._create('/attachments', body, 'attachment')
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", line 306, in _create
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server resp, body = self.api.client.post(url, body=body)
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 227, in post
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return self._cs_request(url, 'POST', **kwargs)
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 215, in _cs_request
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return self.request(url, method, **kwargs)
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 201, in request
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, body)
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)
2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
2023-04-09 11:33:30.238 7 INFO nova.virt.block_device [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Attempting to driver detach volume 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb from mountpoint /dev/vda
2023-04-09 11:33:30.239 7 WARNING nova.virt.block_device [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Detaching volume from unknown instance
2023-04-09 11:33:30.240 7 WARNING nova.virt.libvirt.driver [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] During detach_volume, instance disappeared.: nova.exception.InstanceNotFound: Instance f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 could not be found.
2023-04-09 11:33:30.678 7 INFO nova.virt.libvirt.driver [-] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Instance destroyed successfully.
2023-04-09 11:33:30.682 7 INFO os_vif [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Successfully unplugged vif VIFBridge(active=True,address=fa:16:3e:9b:92:46,bridge_name='qbr46651303-65',has_traffic_filtering=True,id=46651303-65e1-4809-95cc-7c2d97acf16b,network=Network(532470c1-dd2f-4505-8557-ba24e9d4062b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap46651303-65')

All I can see is

b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error in _get_host_uuid: Unexpected error while running command.
Command: blkid overlay -s UUID -o value
Exit code: 2
Stdout: ''
Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.

Searching the web I didn't find anything useful about this error, and I don't know if it's the origin of the problem.

My default storage is ceph rbd, I have an HCI deployment, and all my VMs are for now created on Ceph only. But I have configured two other cinder backends, NFS and LVM, but they are node used for now.

Thanks in advance.

Regards.