[Kolla-ansible][Yoga] Cannot live migrate VMs
Hi, I cannot live-migrate any VM I have. All my compute nodes are enabled. I have a few VMs, I am in the testing phase. I tried to live-migrate toward an empty node, then toward a node which already had a VM in it. It is the same problem. In the log files I have : Source node : Source
==> /var/log/kolla/nova/nova-compute.log <== 2023-04-09 11:32:49.419 7 INFO nova.compute.resource_tracker [req-4e91d562-d7ad-4585-a827-3db6277fd81e - - - - -] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from migration f60eb077-8e1c-404a-aca5-64e5609ca02c 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [-] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Pre live migration failed at computehci02: oslo_messaging.rpc.client.RemoteError: Remote error: ClientException Gateway Time-out (HTTP 504) ['Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 72, in wrapped\n context, exc, binary)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", line 1439, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 213, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 201, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8367, in pre_live_migration\n bdm.save()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8304, in pre_live_migration\n connector=connector, mountpoint=bdm.device_name)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 433, in wrapper\n res = method(self, ctx, volume_id, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 801, in attachment_create\n volume_id, _connector, instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", line 421, in substitution\n return method.func(obj, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 39, in create\n retval = self._create(\'/attachments\', body, \'attachment\')\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", line 306, in _create\n resp, body = self.api.client.post(url, body=body)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 215, in _cs_request\n return self.request(url, method, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 201, in request\n raise exceptions.from_response(resp, body)\n', 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n']. 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Traceback (most recent call last): 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8501, in _do_pre_live_migration_from_source 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] block_migration, disk, dest, migrate_data) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/rpcapi.py", line 948, in pre_live_migration 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] **msg_args) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 192, in call 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] retry=self.retry, transport_options=self.transport_options) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128, in _send 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] transport_options=transport_options) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 691, in send 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] transport_options=transport_options) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] raise result 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] oslo_messaging.rpc.client.RemoteError: Remote error: ClientException Gateway Time-out (HTTP 504) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] ['Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 72, in wrapped\n context, exc, binary)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", line 1439, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 213, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 201, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8367, in pre_live_migration\n bdm.save()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8304, in pre_live_migration\n connector=connector, mountpoint=bdm.device_name)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 433, in wrapper\n res = method(self, ctx, volume_id, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 801, in attachment_create\n volume_id, _connector, instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", line 421, in substitution\n return method.func(obj, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 39, in create\n retval = self._create(\'/attachments\', body, \'attachment\')\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", line 306, in _create\n resp, body = self.api.client.post(url, body=body)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 215, in _cs_request\n return self.request(url, method, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 201, in request\n raise exceptions.from_response(resp, body)\n', 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n']. 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] 2023-04-09 11:33:30.157 7 INFO nova.compute.manager [-] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Swapping old allocation on dict_keys(['2857ae62-3519-4b13-b700-858c827dd78d']) held by migration f60eb077-8e1c-404a-aca5-64e5609ca02c for instance
In the destination node :
nova-compute.log
*2023-04-09 11:32:30.052 7 WARNING os_brick.initiator.connectors.nvmeof [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error in _get_host_uuid: Unexpected error while running command.Command: blkid overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.*
2023-04-09 11:33:05.354 7 INFO nova.compute.resource_tracker [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from migration f60eb077-8e1c-404a-aca5-64e5609ca02c 2023-04-09 11:33:05.392 7 WARNING nova.compute.resource_tracker [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] Instance f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 has been moved to another host computehci01(computehci01). There are allocations remaining against the source host that might need to be removed: {'resources': {'VCPU': 1, 'MEMORY_MB': 2048}}. 2023-04-09 11:33:30.086 7 ERROR nova.volume.cinder [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Create attachment failed for volume 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb. Error: Gateway Time-out (HTTP 504) Code: 504: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Exception during message handling: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 72, in wrapped 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server context, exc, binary) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 63, in wrapped 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", line 1439, in decorated_function 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 213, in decorated_function 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 201, in decorated_function 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8367, in pre_live_migration 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server bdm.save() 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8304, in pre_live_migration 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server connector=connector, mountpoint=bdm.device_name) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 395, in wrapper 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = method(self, ctx, *args, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 433, in wrapper 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = method(self, ctx, volume_id, *args, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 813, in attachment_create 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server instance_uuid=instance_id) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 801, in attachment_create 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server volume_id, _connector, instance_id) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", line 421, in substitution 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return method.func(obj, *args, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 39, in create 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server retval = self._create('/attachments', body, 'attachment') 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", line 306, in _create 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server resp, body = self.api.client.post(url, body=body) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 227, in post 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return self._cs_request(url, 'POST', **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 215, in _cs_request 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return self.request(url, method, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 201, in request 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, body) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server 2023-04-09 11:33:30.238 7 INFO nova.virt.block_device [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Attempting to driver detach volume 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb from mountpoint /dev/vda 2023-04-09 11:33:30.239 7 WARNING nova.virt.block_device [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Detaching volume from unknown instance 2023-04-09 11:33:30.240 7 WARNING nova.virt.libvirt.driver [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] During detach_volume, instance disappeared.: nova.exception.InstanceNotFound: Instance f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 could not be found. 2023-04-09 11:33:30.678 7 INFO nova.virt.libvirt.driver [-] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Instance destroyed successfully. 2023-04-09 11:33:30.682 7 INFO os_vif [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Successfully unplugged vif VIFBridge(active=True,address=fa:16:3e:9b:92:46,bridge_name='qbr46651303-65',has_traffic_filtering=True,id=46651303-65e1-4809-95cc-7c2d97acf16b,network=Network(532470c1-dd2f-4505-8557-ba24e9d4062b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap46651303-65')
All I can see is *b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error in _get_host_uuid: Unexpected error while running command.Command: blkid overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.* Searching the web I didn't find anything useful about this error, and I don't know if it's the origin of the problem. My default storage is ceph rbd, I have an HCI deployment, and all my VMs are for now created on Ceph only. But I have configured two other cinder backends, NFS and LVM, but they are node used for now. Thanks in advance. Regards.
You should check the status of your Cinder services. On Sun, 9 Apr 2023 at 19:43, wodel youchi <wodel.youchi@gmail.com> wrote:
Hi,
I cannot live-migrate any VM I have. All my compute nodes are enabled. I have a few VMs, I am in the testing phase. I tried to live-migrate toward an empty node, then toward a node which already had a VM in it. It is the same problem.
In the log files I have :
Source node : Source
==> /var/log/kolla/nova/nova-compute.log <== 2023-04-09 11:32:49.419 7 INFO nova.compute.resource_tracker [req-4e91d562-d7ad-4585-a827-3db6277fd81e - - - - -] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from migration f60eb077-8e1c-404a-aca5-64e5609ca02c 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [-] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Pre live migration failed at computehci02: oslo_messaging.rpc.client.RemoteError: Remote error: ClientException Gateway Time-out (HTTP 504) ['Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 72, in wrapped\n context, exc, binary)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", line 1439, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 213, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 201, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8367, in pre_live_migration\n bdm.save()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8304, in pre_live_migration\n connector=connector, mountpoint=bdm.device_name)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 433, in wrapper\n res = method(self, ctx, volume_id, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 801, in attachment_create\n volume_id, _connector, instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", line 421, in substitution\n return method.func(obj, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 39, in create\n retval = self._create(\'/attachments\', body, \'attachment\')\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", line 306, in _create\n resp, body = self.api.client.post(url, body=body)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 215, in _cs_request\n return self.request(url, method, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 201, in request\n raise exceptions.from_response(resp, body)\n', 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n']. 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Traceback (most recent call last): 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8501, in _do_pre_live_migration_from_source 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] block_migration, disk, dest, migrate_data) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/rpcapi.py", line 948, in pre_live_migration 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] **msg_args) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 192, in call 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] retry=self.retry, transport_options=self.transport_options) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128, in _send 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] transport_options=transport_options) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 691, in send 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] transport_options=transport_options) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] raise result 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] oslo_messaging.rpc.client.RemoteError: Remote error: ClientException Gateway Time-out (HTTP 504) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] ['Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 72, in wrapped\n context, exc, binary)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", line 1439, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 213, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 201, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8367, in pre_live_migration\n bdm.save()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8304, in pre_live_migration\n connector=connector, mountpoint=bdm.device_name)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 433, in wrapper\n res = method(self, ctx, volume_id, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 801, in attachment_create\n volume_id, _connector, instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", line 421, in substitution\n return method.func(obj, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 39, in create\n retval = self._create(\'/attachments\', body, \'attachment\')\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", line 306, in _create\n resp, body = self.api.client.post(url, body=body)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 215, in _cs_request\n return self.request(url, method, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 201, in request\n raise exceptions.from_response(resp, body)\n', 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n']. 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] 2023-04-09 11:33:30.157 7 INFO nova.compute.manager [-] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Swapping old allocation on dict_keys(['2857ae62-3519-4b13-b700-858c827dd78d']) held by migration f60eb077-8e1c-404a-aca5-64e5609ca02c for instance
In the destination node :
nova-compute.log
*2023-04-09 11:32:30.052 7 WARNING os_brick.initiator.connectors.nvmeof [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error in _get_host_uuid: Unexpected error while running command.Command: blkid overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.*
2023-04-09 11:33:05.354 7 INFO nova.compute.resource_tracker [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from migration f60eb077-8e1c-404a-aca5-64e5609ca02c 2023-04-09 11:33:05.392 7 WARNING nova.compute.resource_tracker [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] Instance f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 has been moved to another host computehci01(computehci01). There are allocations remaining against the source host that might need to be removed: {'resources': {'VCPU': 1, 'MEMORY_MB': 2048}}. 2023-04-09 11:33:30.086 7 ERROR nova.volume.cinder [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Create attachment failed for volume 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb. Error: Gateway Time-out (HTTP 504) Code: 504: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Exception during message handling: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 72, in wrapped 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server context, exc, binary) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 63, in wrapped 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", line 1439, in decorated_function 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 213, in decorated_function 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 201, in decorated_function 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8367, in pre_live_migration 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server bdm.save() 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8304, in pre_live_migration 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server connector=connector, mountpoint=bdm.device_name) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 395, in wrapper 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = method(self, ctx, *args, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 433, in wrapper 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = method(self, ctx, volume_id, *args, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 813, in attachment_create 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server instance_uuid=instance_id) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 801, in attachment_create 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server volume_id, _connector, instance_id) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", line 421, in substitution 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return method.func(obj, *args, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 39, in create 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server retval = self._create('/attachments', body, 'attachment') 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", line 306, in _create 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server resp, body = self.api.client.post(url, body=body) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 227, in post 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return self._cs_request(url, 'POST', **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 215, in _cs_request 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return self.request(url, method, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 201, in request 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, body) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server 2023-04-09 11:33:30.238 7 INFO nova.virt.block_device [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Attempting to driver detach volume 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb from mountpoint /dev/vda 2023-04-09 11:33:30.239 7 WARNING nova.virt.block_device [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Detaching volume from unknown instance 2023-04-09 11:33:30.240 7 WARNING nova.virt.libvirt.driver [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] During detach_volume, instance disappeared.: nova.exception.InstanceNotFound: Instance f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 could not be found. 2023-04-09 11:33:30.678 7 INFO nova.virt.libvirt.driver [-] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Instance destroyed successfully. 2023-04-09 11:33:30.682 7 INFO os_vif [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Successfully unplugged vif VIFBridge(active=True,address=fa:16:3e:9b:92:46,bridge_name='qbr46651303-65',has_traffic_filtering=True,id=46651303-65e1-4809-95cc-7c2d97acf16b,network=Network(532470c1-dd2f-4505-8557-ba24e9d4062b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap46651303-65')
All I can see is
*b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error in _get_host_uuid: Unexpected error while running command.Command: blkid overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.*
Searching the web I didn't find anything useful about this error, and I don't know if it's the origin of the problem.
My default storage is ceph rbd, I have an HCI deployment, and all my VMs are for now created on Ceph only. But I have configured two other cinder backends, NFS and LVM, but they are node used for now.
Thanks in advance.
Regards.
Hi, Indeed, there was a mistake in the cinder.conf file. Thanks. Regards. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> Le mar. 11 avr. 2023 à 10:45, Pierre Riteau <pierre@stackhpc.com> a écrit :
You should check the status of your Cinder services.
On Sun, 9 Apr 2023 at 19:43, wodel youchi <wodel.youchi@gmail.com> wrote:
Hi,
I cannot live-migrate any VM I have. All my compute nodes are enabled. I have a few VMs, I am in the testing phase. I tried to live-migrate toward an empty node, then toward a node which already had a VM in it. It is the same problem.
In the log files I have :
Source node : Source
==> /var/log/kolla/nova/nova-compute.log <== 2023-04-09 11:32:49.419 7 INFO nova.compute.resource_tracker [req-4e91d562-d7ad-4585-a827-3db6277fd81e - - - - -] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from migration f60eb077-8e1c-404a-aca5-64e5609ca02c 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [-] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Pre live migration failed at computehci02: oslo_messaging.rpc.client.RemoteError: Remote error: ClientException Gateway Time-out (HTTP 504) ['Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 72, in wrapped\n context, exc, binary)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", line 1439, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 213, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 201, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8367, in pre_live_migration\n bdm.save()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8304, in pre_live_migration\n connector=connector, mountpoint=bdm.device_name)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 433, in wrapper\n res = method(self, ctx, volume_id, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 801, in attachment_create\n volume_id, _connector, instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", line 421, in substitution\n return method.func(obj, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 39, in create\n retval = self._create(\'/attachments\', body, \'attachment\')\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", line 306, in _create\n resp, body = self.api.client.post(url, body=body)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 215, in _cs_request\n return self.request(url, method, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 201, in request\n raise exceptions.from_response(resp, body)\n', 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n']. 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Traceback (most recent call last): 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8501, in _do_pre_live_migration_from_source 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] block_migration, disk, dest, migrate_data) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/rpcapi.py", line 948, in pre_live_migration 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] **msg_args) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 192, in call 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] retry=self.retry, transport_options=self.transport_options) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128, in _send 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] transport_options=transport_options) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 691, in send 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] transport_options=transport_options) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] raise result 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] oslo_messaging.rpc.client.RemoteError: Remote error: ClientException Gateway Time-out (HTTP 504) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] ['Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 72, in wrapped\n context, exc, binary)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", line 1439, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 213, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 201, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8367, in pre_live_migration\n bdm.save()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8304, in pre_live_migration\n connector=connector, mountpoint=bdm.device_name)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 433, in wrapper\n res = method(self, ctx, volume_id, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 801, in attachment_create\n volume_id, _connector, instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", line 421, in substitution\n return method.func(obj, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 39, in create\n retval = self._create(\'/attachments\', body, \'attachment\')\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", line 306, in _create\n resp, body = self.api.client.post(url, body=body)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 215, in _cs_request\n return self.request(url, method, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 201, in request\n raise exceptions.from_response(resp, body)\n', 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n']. 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] 2023-04-09 11:33:30.157 7 INFO nova.compute.manager [-] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Swapping old allocation on dict_keys(['2857ae62-3519-4b13-b700-858c827dd78d']) held by migration f60eb077-8e1c-404a-aca5-64e5609ca02c for instance
In the destination node :
nova-compute.log
*2023-04-09 11:32:30.052 7 WARNING os_brick.initiator.connectors.nvmeof [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error in _get_host_uuid: Unexpected error while running command.Command: blkid overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.*
2023-04-09 11:33:05.354 7 INFO nova.compute.resource_tracker [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from migration f60eb077-8e1c-404a-aca5-64e5609ca02c 2023-04-09 11:33:05.392 7 WARNING nova.compute.resource_tracker [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] Instance f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 has been moved to another host computehci01(computehci01). There are allocations remaining against the source host that might need to be removed: {'resources': {'VCPU': 1, 'MEMORY_MB': 2048}}. 2023-04-09 11:33:30.086 7 ERROR nova.volume.cinder [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Create attachment failed for volume 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb. Error: Gateway Time-out (HTTP 504) Code: 504: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Exception during message handling: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 72, in wrapped 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server context, exc, binary) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 63, in wrapped 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", line 1439, in decorated_function 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 213, in decorated_function 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 201, in decorated_function 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8367, in pre_live_migration 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server bdm.save() 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8304, in pre_live_migration 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server connector=connector, mountpoint=bdm.device_name) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 395, in wrapper 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = method(self, ctx, *args, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 433, in wrapper 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = method(self, ctx, volume_id, *args, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 813, in attachment_create 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server instance_uuid=instance_id) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 801, in attachment_create 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server volume_id, _connector, instance_id) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", line 421, in substitution 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return method.func(obj, *args, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 39, in create 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server retval = self._create('/attachments', body, 'attachment') 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", line 306, in _create 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server resp, body = self.api.client.post(url, body=body) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 227, in post 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return self._cs_request(url, 'POST', **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 215, in _cs_request 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return self.request(url, method, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 201, in request 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, body) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server 2023-04-09 11:33:30.238 7 INFO nova.virt.block_device [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Attempting to driver detach volume 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb from mountpoint /dev/vda 2023-04-09 11:33:30.239 7 WARNING nova.virt.block_device [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Detaching volume from unknown instance 2023-04-09 11:33:30.240 7 WARNING nova.virt.libvirt.driver [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] During detach_volume, instance disappeared.: nova.exception.InstanceNotFound: Instance f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 could not be found. 2023-04-09 11:33:30.678 7 INFO nova.virt.libvirt.driver [-] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Instance destroyed successfully. 2023-04-09 11:33:30.682 7 INFO os_vif [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Successfully unplugged vif VIFBridge(active=True,address=fa:16:3e:9b:92:46,bridge_name='qbr46651303-65',has_traffic_filtering=True,id=46651303-65e1-4809-95cc-7c2d97acf16b,network=Network(532470c1-dd2f-4505-8557-ba24e9d4062b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap46651303-65')
All I can see is
*b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error in _get_host_uuid: Unexpected error while running command.Command: blkid overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.*
Searching the web I didn't find anything useful about this error, and I don't know if it's the origin of the problem.
My default storage is ceph rbd, I have an HCI deployment, and all my VMs are for now created on Ceph only. But I have configured two other cinder backends, NFS and LVM, but they are node used for now.
Thanks in advance.
Regards.
Hi, this seems to be related to cinder, do you see anything in the cinder-volume.log? Zitat von wodel youchi <wodel.youchi@gmail.com>:
Hi,
I cannot live-migrate any VM I have. All my compute nodes are enabled. I have a few VMs, I am in the testing phase. I tried to live-migrate toward an empty node, then toward a node which already had a VM in it. It is the same problem.
In the log files I have :
Source node : Source
==> /var/log/kolla/nova/nova-compute.log <== 2023-04-09 11:32:49.419 7 INFO nova.compute.resource_tracker [req-4e91d562-d7ad-4585-a827-3db6277fd81e - - - - -] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from migration f60eb077-8e1c-404a-aca5-64e5609ca02c 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [-] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Pre live migration failed at computehci02: oslo_messaging.rpc.client.RemoteError: Remote error: ClientException Gateway Time-out (HTTP 504) ['Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 72, in wrapped\n context, exc, binary)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", line 1439, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 213, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 201, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8367, in pre_live_migration\n bdm.save()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8304, in pre_live_migration\n connector=connector, mountpoint=bdm.device_name)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 433, in wrapper\n res = method(self, ctx, volume_id, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 801, in attachment_create\n volume_id, _connector, instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", line 421, in substitution\n return method.func(obj, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 39, in create\n retval = self._create(\'/attachments\', body, \'attachment\')\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", line 306, in _create\n resp, body = self.api.client.post(url, body=body)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 215, in _cs_request\n return self.request(url, method, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 201, in request\n raise exceptions.from_response(resp, body)\n', 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n']. 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Traceback (most recent call last): 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8501, in _do_pre_live_migration_from_source 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] block_migration, disk, dest, migrate_data) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/rpcapi.py", line 948, in pre_live_migration 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] **msg_args) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 192, in call 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] retry=self.retry, transport_options=self.transport_options) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128, in _send 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] transport_options=transport_options) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 691, in send 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] transport_options=transport_options) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] raise result 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] oslo_messaging.rpc.client.RemoteError: Remote error: ClientException Gateway Time-out (HTTP 504) 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] ['Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 72, in wrapped\n context, exc, binary)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", line 1439, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 213, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 201, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8367, in pre_live_migration\n bdm.save()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8304, in pre_live_migration\n connector=connector, mountpoint=bdm.device_name)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 433, in wrapper\n res = method(self, ctx, volume_id, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 801, in attachment_create\n volume_id, _connector, instance_id)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", line 421, in substitution\n return method.func(obj, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 39, in create\n retval = self._create(\'/attachments\', body, \'attachment\')\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", line 306, in _create\n resp, body = self.api.client.post(url, body=body)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 215, in _cs_request\n return self.request(url, method, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 201, in request\n raise exceptions.from_response(resp, body)\n', 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n']. 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] 2023-04-09 11:33:30.157 7 INFO nova.compute.manager [-] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Swapping old allocation on dict_keys(['2857ae62-3519-4b13-b700-858c827dd78d']) held by migration f60eb077-8e1c-404a-aca5-64e5609ca02c for instance
In the destination node :
nova-compute.log
*2023-04-09 11:32:30.052 7 WARNING os_brick.initiator.connectors.nvmeof [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error in _get_host_uuid: Unexpected error while running command.Command: blkid overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.*
2023-04-09 11:33:05.354 7 INFO nova.compute.resource_tracker [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from migration f60eb077-8e1c-404a-aca5-64e5609ca02c 2023-04-09 11:33:05.392 7 WARNING nova.compute.resource_tracker [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] Instance f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 has been moved to another host computehci01(computehci01). There are allocations remaining against the source host that might need to be removed: {'resources': {'VCPU': 1, 'MEMORY_MB': 2048}}. 2023-04-09 11:33:30.086 7 ERROR nova.volume.cinder [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Create attachment failed for volume 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb. Error: Gateway Time-out (HTTP 504) Code: 504: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Exception during message handling: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 72, in wrapped 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server context, exc, binary) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 63, in wrapped 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", line 1439, in decorated_function 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 213, in decorated_function 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 201, in decorated_function 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8367, in pre_live_migration 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server bdm.save() 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 8304, in pre_live_migration 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server connector=connector, mountpoint=bdm.device_name) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 395, in wrapper 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = method(self, ctx, *args, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 433, in wrapper 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res = method(self, ctx, volume_id, *args, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 813, in attachment_create 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server instance_uuid=instance_id) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise self.value 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 801, in attachment_create 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server volume_id, _connector, instance_id) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py", line 421, in substitution 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return method.func(obj, *args, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 39, in create 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server retval = self._create('/attachments', body, 'attachment') 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py", line 306, in _create 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server resp, body = self.api.client.post(url, body=body) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 227, in post 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return self._cs_request(url, 'POST', **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 215, in _cs_request 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return self.request(url, method, **kwargs) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py", line 201, in request 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, body) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504) 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server 2023-04-09 11:33:30.238 7 INFO nova.virt.block_device [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Attempting to driver detach volume 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb from mountpoint /dev/vda 2023-04-09 11:33:30.239 7 WARNING nova.virt.block_device [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Detaching volume from unknown instance 2023-04-09 11:33:30.240 7 WARNING nova.virt.libvirt.driver [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] During detach_volume, instance disappeared.: nova.exception.InstanceNotFound: Instance f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 could not be found. 2023-04-09 11:33:30.678 7 INFO nova.virt.libvirt.driver [-] [instance: f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Instance destroyed successfully. 2023-04-09 11:33:30.682 7 INFO os_vif [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4 b0f76b5c6dcb457fa716762bbf954837 - default default] Successfully unplugged vif VIFBridge(active=True,address=fa:16:3e:9b:92:46,bridge_name='qbr46651303-65',has_traffic_filtering=True,id=46651303-65e1-4809-95cc-7c2d97acf16b,network=Network(532470c1-dd2f-4505-8557-ba24e9d4062b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap46651303-65')
All I can see is
*b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error in _get_host_uuid: Unexpected error while running command.Command: blkid overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.*
Searching the web I didn't find anything useful about this error, and I don't know if it's the origin of the problem.
My default storage is ceph rbd, I have an HCI deployment, and all my VMs are for now created on Ceph only. But I have configured two other cinder backends, NFS and LVM, but they are node used for now.
Thanks in advance.
Regards.
Thanks for the help, I had a configuration mistake, I configured cinder with multiple stores and when adding them I made a mistake, the name of the rbd store is rbd-1, when I created the cinder.conf I wrote rbd1, so cinder-volume was not seeing the rbd store correctly, after correcting the typo and deploying again, migration worked without issues. Regards.
participants (3)
-
Eugen Block
-
Pierre Riteau
-
wodel youchi