<div dir="ltr"><div>Hi,</div><div><br></div><div>Indeed, there was a mistake in the cinder.conf file.</div><div>Thanks.<br></div><div><br></div><div>Regards.<br></div></div><div id="DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2"><br><table style="border-top:1px solid #d3d4de"><tr><td style="width:55px;padding-top:13px"><a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail" target="_blank"><img src="https://s-install.avcdn.net/ipm/preview/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif" alt="" width="46" height="29" style="width: 46px; height: 29px;"></a></td><td style="width:470px;padding-top:12px;color:#41424e;font-size:13px;font-family:Arial,Helvetica,sans-serif;line-height:18px">Virus-free.<a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail" target="_blank" style="color:#4453ea">www.avast.com</a></td></tr></table><a href="#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2" width="1" height="1"></a></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le mar. 11 avr. 2023 à 10:45, Pierre Riteau <<a href="mailto:pierre@stackhpc.com">pierre@stackhpc.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>You should check the status of your Cinder services.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, 9 Apr 2023 at 19:43, wodel youchi <<a href="mailto:wodel.youchi@gmail.com" target="_blank">wodel.youchi@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi,</div><div><br></div><div>I cannot live-migrate any VM I have.<br></div><div>All my compute nodes are enabled. I have a few VMs, I am in the testing phase.</div><div>I tried to live-migrate toward an empty node, then toward a node which already had a VM in it. It is the same problem. <br></div><div><br></div><div>In the log files I have :</div><div><br></div><div>Source node :</div><div>Source<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">==> /var/log/kolla/nova/nova-compute.log <==<br>2023-04-09 11:32:49.419 7 INFO nova.compute.resource_tracker [req-4e91d562-d7ad-4585-a827-3db6277fd81e - - - - -] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from migration f60eb077-8e1c-404a-aca5-64e5609ca02c<br>2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [-] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Pre live migration failed at computehci02: oslo_messaging.rpc.client.RemoteError: Remote error: ClientException Gateway Time-out (HTTP 504)<br>['Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 72, in wrapped\n context, exc, binary)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", line 1439, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 213, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 201, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8367, in pre_live_migration\n bdm.save()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8304, in pre_live_migration\n connector=connector, mountpoint=bdm.device_name)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 433, in wrapper\n res = method(self, ctx, volume_id, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 801, in attachment_create\n volume_id, _connector, instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", line 421, in substitution\n return method.func(obj, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 39, in create\n retval = self._create(\'/attachments\', body, \'attachment\')\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", line 306, in _create\n resp, body = <a href="http://self.api.client.post" target="_blank">self.api.client.post</a>(url, body=body)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 215, in _cs_request\n return self.request(url, method, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 201, in request\n raise exceptions.from_response(resp, body)\n', 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n'].<br>2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Traceback (most recent call last):<br>2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8501, in _do_pre_live_migration_from_source<br>2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] block_migration, disk, dest, migrate_data)<br>2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/rpcapi.py", line 948, in pre_live_migration<br>2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] **msg_args)<br>2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 192, in call<br>2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] retry=self.retry, transport_options=self.transport_options)<br>2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128, in _send<br>2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] transport_options=transport_options)<br>2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 691, in send<br>2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] transport_options=transport_options)<br>2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send<br>2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] raise result<br>2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] oslo_messaging.rpc.client.RemoteError: Remote error: ClientException Gateway Time-out (HTTP 504)<br>2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] ['Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 72, in wrapped\n context, exc, binary)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", line 1439, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 213, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 201, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8367, in pre_live_migration\n bdm.save()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8304, in pre_live_migration\n connector=connector, mountpoint=bdm.device_name)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 433, in wrapper\n res = method(self, ctx, volume_id, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 801, in attachment_create\n volume_id, _connector, instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", line 421, in substitution\n return method.func(obj, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 39, in create\n retval = self._create(\'/attachments\', body, \'attachment\')\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", line 306, in _create\n resp, body = <a href="http://self.api.client.post" target="_blank">self.api.client.post</a>(url, body=body)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 215, in _cs_request\n return self.request(url, method, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 201, in request\n raise exceptions.from_response(resp, body)\n', 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n'].<br>2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] <br>2023-04-09 11:33:30.157 7 INFO nova.compute.manager [-] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Swapping old allocation on dict_keys(['2857ae62-3519-4b13-b700-858c827dd78d']) held by migration f60eb077-8e1c-404a-aca5-64e5609ca02c for instance</blockquote></div><div><br></div><div><br></div><div>In the destination node :</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>nova-compute.log<br><b>2023-04-09 11:32:30.052 7 WARNING os_brick.initiator.connectors.nvmeof [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error in _get_host_uuid: Unexpected error while running command.<br>Command: blkid overlay -s UUID -o value<br>Exit code: 2<br>Stdout: ''<br>Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.</b><br><br>2023-04-09 11:33:05.354 7 INFO nova.compute.resource_tracker [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from migration f60eb077-8e1c-404a-aca5-64e5609ca02c<br>2023-04-09 11:33:05.392 7 WARNING nova.compute.resource_tracker [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] Instance f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 has been moved to another host computehci01(computehci01). There are allocations remaining against the source host that might need to be removed: {'resources': {'VCPU': 1, 'MEMORY_MB': 2048}}.<br>2023-04-09 11:33:30.086 7 ERROR nova.volume.cinder [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Create attachment failed for volume 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb. Error: Gateway Time-out (HTTP 504) Code: 504: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Exception during message handling: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 72, in wrapped<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server context, exc, binary)<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise()<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 63, in wrapped<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", line 1439, in decorated_function<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 213, in decorated_function<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise()<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 201, in decorated_function<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8367, in pre_live_migration<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server bdm.save()<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise()<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8304, in pre_live_migration<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server connector=connector, mountpoint=bdm.device_name)<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 395, in wrapper<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = method(self, ctx, *args, **kwargs)<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 433, in wrapper<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = method(self, ctx, volume_id, *args, **kwargs)<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 813, in attachment_create<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server instance_uuid=instance_id)<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise()<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 801, in attachment_create<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server volume_id, _connector, instance_id)<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", line 421, in substitution<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return method.func(obj, *args, **kwargs)<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 39, in create<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server retval = self._create('/attachments', body, 'attachment')<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", line 306, in _create<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server resp, body = <a href="http://self.api.client.post" target="_blank">self.api.client.post</a>(url, body=body)<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 227, in post<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return self._cs_request(url, 'POST', **kwargs)<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 215, in _cs_request<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return self.request(url, method, **kwargs)<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 201, in request<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, body)<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)<br>2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server <br>2023-04-09 11:33:30.238 7 INFO nova.virt.block_device [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Attempting to driver detach volume 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb from mountpoint /dev/vda<br>2023-04-09 11:33:30.239 7 WARNING nova.virt.block_device [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Detaching volume from unknown instance<br>2023-04-09 11:33:30.240 7 WARNING nova.virt.libvirt.driver [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] During detach_volume, instance disappeared.: nova.exception.InstanceNotFound: Instance f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 could not be found.<br>2023-04-09 11:33:30.678 7 INFO nova.virt.libvirt.driver [-] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Instance destroyed successfully.<br>2023-04-09 11:33:30.682 7 INFO os_vif [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Successfully unplugged vif VIFBridge(active=True,address=fa:16:3e:9b:92:46,bridge_name='qbr46651303-65',has_traffic_filtering=True,id=46651303-65e1-4809-95cc-7c2d97acf16b,network=Network(532470c1-dd2f-4505-8557-ba24e9d4062b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap46651303-65')</div></blockquote><div><br></div><div>All I can see is <br></div><div><b>b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution
error in _get_host_uuid: Unexpected error while running command.<br>Command: blkid overlay -s UUID -o value<br>Exit code: 2<br>Stdout: ''<br>Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.</b><br></div><div><br></div><div>Searching the web I didn't find anything useful about this error, and I don't know if it's the origin of the problem.</div><div><br></div><div>My default storage is ceph rbd, I have an HCI deployment, and all my VMs are for now created on Ceph only. But I have configured two other cinder backends, NFS and LVM, but they are node used for now.<br></div><div><br></div><div>Thanks in advance.<br></div><div><br></div><div>Regards.<br></div></div>
</blockquote></div></div>
</blockquote></div>