[Kolla-ansible][Yoga] Cannot live migrate VMs
Pierre Riteau
pierre at stackhpc.com
Tue Apr 11 09:45:19 UTC 2023
You should check the status of your Cinder services.
On Sun, 9 Apr 2023 at 19:43, wodel youchi <wodel.youchi at gmail.com> wrote:
> Hi,
>
> I cannot live-migrate any VM I have.
> All my compute nodes are enabled. I have a few VMs, I am in the testing
> phase.
> I tried to live-migrate toward an empty node, then toward a node which
> already had a VM in it. It is the same problem.
>
> In the log files I have :
>
> Source node :
> Source
>
>> ==> /var/log/kolla/nova/nova-compute.log <==
>> 2023-04-09 11:32:49.419 7 INFO nova.compute.resource_tracker
>> [req-4e91d562-d7ad-4585-a827-3db6277fd81e - - - - -] [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from
>> migration f60eb077-8e1c-404a-aca5-64e5609ca02c
>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [-] [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Pre live migration failed at
>> computehci02: oslo_messaging.rpc.client.RemoteError: Remote error:
>> ClientException Gateway Time-out (HTTP 504)
>> ['Traceback (most recent call last):\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py",
>> line 165, in _process_incoming\n res =
>> self.dispatcher.dispatch(message)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py",
>> line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt,
>> args)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py",
>> line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py",
>> line 72, in wrapped\n context, exc, binary)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 227, in __exit__\n self.force_reraise()\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 200, in force_reraise\n raise self.value\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py",
>> line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py",
>> line 1439, in decorated_function\n return function(self, context, *args,
>> **kwargs)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>> line 213, in decorated_function\n kwargs[\'instance\'], e,
>> sys.exc_info())\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 227, in __exit__\n self.force_reraise()\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 200, in force_reraise\n raise self.value\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>> line 201, in decorated_function\n return function(self, context, *args,
>> **kwargs)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>> line 8367, in pre_live_migration\n bdm.save()\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 227, in __exit__\n self.force_reraise()\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 200, in force_reraise\n raise self.value\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>> line 8304, in pre_live_migration\n connector=connector,
>> mountpoint=bdm.device_name)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>> line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', '
>> File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>> line 433, in wrapper\n res = method(self, ctx, volume_id, *args,
>> **kwargs)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>> line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 227, in __exit__\n self.force_reraise()\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 200, in force_reraise\n raise self.value\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>> line 801, in attachment_create\n volume_id, _connector, instance_id)\n',
>> ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py",
>> line 421, in substitution\n return method.func(obj, *args, **kwargs)\n',
>> ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py",
>> line 39, in create\n retval = self._create(\'/attachments\', body,
>> \'attachment\')\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py",
>> line 306, in _create\n resp, body = self.api.client.post(url,
>> body=body)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
>> line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n',
>> ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
>> line 215, in _cs_request\n return self.request(url, method,
>> **kwargs)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
>> line 201, in request\n raise exceptions.from_response(resp, body)\n',
>> 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n'].
>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Traceback (most recent call last):
>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>> line 8501, in _do_pre_live_migration_from_source
>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] block_migration, disk, dest,
>> migrate_data)
>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/rpcapi.py",
>> line 948, in pre_live_migration
>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] **msg_args)
>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py",
>> line 192, in call
>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] retry=self.retry,
>> transport_options=self.transport_options)
>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py",
>> line 128, in _send
>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7]
>> transport_options=transport_options)
>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py",
>> line 691, in send
>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7]
>> transport_options=transport_options)
>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py",
>> line 681, in _send
>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] raise result
>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7]
>> oslo_messaging.rpc.client.RemoteError: Remote error: ClientException
>> Gateway Time-out (HTTP 504)
>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] ['Traceback (most recent call
>> last):\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py",
>> line 165, in _process_incoming\n res =
>> self.dispatcher.dispatch(message)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py",
>> line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt,
>> args)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py",
>> line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py",
>> line 72, in wrapped\n context, exc, binary)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 227, in __exit__\n self.force_reraise()\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 200, in force_reraise\n raise self.value\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py",
>> line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py",
>> line 1439, in decorated_function\n return function(self, context, *args,
>> **kwargs)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>> line 213, in decorated_function\n kwargs[\'instance\'], e,
>> sys.exc_info())\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 227, in __exit__\n self.force_reraise()\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 200, in force_reraise\n raise self.value\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>> line 201, in decorated_function\n return function(self, context, *args,
>> **kwargs)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>> line 8367, in pre_live_migration\n bdm.save()\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 227, in __exit__\n self.force_reraise()\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 200, in force_reraise\n raise self.value\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>> line 8304, in pre_live_migration\n connector=connector,
>> mountpoint=bdm.device_name)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>> line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', '
>> File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>> line 433, in wrapper\n res = method(self, ctx, volume_id, *args,
>> **kwargs)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>> line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 227, in __exit__\n self.force_reraise()\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 200, in force_reraise\n raise self.value\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>> line 801, in attachment_create\n volume_id, _connector, instance_id)\n',
>> ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py",
>> line 421, in substitution\n return method.func(obj, *args, **kwargs)\n',
>> ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py",
>> line 39, in create\n retval = self._create(\'/attachments\', body,
>> \'attachment\')\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py",
>> line 306, in _create\n resp, body = self.api.client.post(url,
>> body=body)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
>> line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n',
>> ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
>> line 215, in _cs_request\n return self.request(url, method,
>> **kwargs)\n', ' File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
>> line 201, in request\n raise exceptions.from_response(resp, body)\n',
>> 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n'].
>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7]
>> 2023-04-09 11:33:30.157 7 INFO nova.compute.manager [-] [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Swapping old allocation on
>> dict_keys(['2857ae62-3519-4b13-b700-858c827dd78d']) held by migration
>> f60eb077-8e1c-404a-aca5-64e5609ca02c for instance
>
>
>
> In the destination node :
>
>> nova-compute.log
>>
>>
>>
>>
>> *2023-04-09 11:32:30.052 7 WARNING os_brick.initiator.connectors.nvmeof
>> [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4
>> b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error
>> in _get_host_uuid: Unexpected error while running command.Command: blkid
>> overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '':
>> oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while
>> running command.*
>>
>> 2023-04-09 11:33:05.354 7 INFO nova.compute.resource_tracker
>> [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from
>> migration f60eb077-8e1c-404a-aca5-64e5609ca02c
>> 2023-04-09 11:33:05.392 7 WARNING nova.compute.resource_tracker
>> [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] Instance
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 has been moved to another host
>> computehci01(computehci01). There are allocations remaining against the
>> source host that might need to be removed: {'resources': {'VCPU': 1,
>> 'MEMORY_MB': 2048}}.
>> 2023-04-09 11:33:30.086 7 ERROR nova.volume.cinder
>> [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4
>> b0f76b5c6dcb457fa716762bbf954837 - default default] [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Create attachment failed for volume
>> 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb. Error: Gateway Time-out (HTTP 504)
>> Code: 504: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP
>> 504)
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
>> [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4
>> b0f76b5c6dcb457fa716762bbf954837 - default default] Exception during
>> message handling: cinderclient.exceptions.ClientException: Gateway Time-out
>> (HTTP 504)
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server Traceback (most
>> recent call last):
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py",
>> line 165, in _process_incoming
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res =
>> self.dispatcher.dispatch(message)
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py",
>> line 309, in dispatch
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return
>> self._do_dispatch(endpoint, method, ctxt, args)
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py",
>> line 229, in _do_dispatch
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server result =
>> func(ctxt, **new_args)
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py",
>> line 72, in wrapped
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server context,
>> exc, binary)
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 227, in __exit__
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
>> self.force_reraise()
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 200, in force_reraise
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise
>> self.value
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py",
>> line 63, in wrapped
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return
>> f(self, context, *args, **kw)
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py",
>> line 1439, in decorated_function
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return
>> function(self, context, *args, **kwargs)
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>> line 213, in decorated_function
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
>> kwargs['instance'], e, sys.exc_info())
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 227, in __exit__
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
>> self.force_reraise()
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 200, in force_reraise
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise
>> self.value
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>> line 201, in decorated_function
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return
>> function(self, context, *args, **kwargs)
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>> line 8367, in pre_live_migration
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server bdm.save()
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 227, in __exit__
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
>> self.force_reraise()
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 200, in force_reraise
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise
>> self.value
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>> line 8304, in pre_live_migration
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
>> connector=connector, mountpoint=bdm.device_name)
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>> line 395, in wrapper
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res =
>> method(self, ctx, *args, **kwargs)
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>> line 433, in wrapper
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res =
>> method(self, ctx, volume_id, *args, **kwargs)
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>> line 813, in attachment_create
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
>> instance_uuid=instance_id)
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 227, in __exit__
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
>> self.force_reraise()
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>> line 200, in force_reraise
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise
>> self.value
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>> line 801, in attachment_create
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server volume_id,
>> _connector, instance_id)
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py",
>> line 421, in substitution
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return
>> method.func(obj, *args, **kwargs)
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py",
>> line 39, in create
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server retval =
>> self._create('/attachments', body, 'attachment')
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py",
>> line 306, in _create
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server resp, body
>> = self.api.client.post(url, body=body)
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
>> line 227, in post
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return
>> self._cs_request(url, 'POST', **kwargs)
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
>> line 215, in _cs_request
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return
>> self.request(url, method, **kwargs)
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
>> line 201, in request
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise
>> exceptions.from_response(resp, body)
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
>> cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)
>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
>> 2023-04-09 11:33:30.238 7 INFO nova.virt.block_device
>> [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4
>> b0f76b5c6dcb457fa716762bbf954837 - default default] [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Attempting to driver detach volume
>> 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb from mountpoint /dev/vda
>> 2023-04-09 11:33:30.239 7 WARNING nova.virt.block_device
>> [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4
>> b0f76b5c6dcb457fa716762bbf954837 - default default] [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Detaching volume from unknown instance
>> 2023-04-09 11:33:30.240 7 WARNING nova.virt.libvirt.driver
>> [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4
>> b0f76b5c6dcb457fa716762bbf954837 - default default] [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] During detach_volume, instance
>> disappeared.: nova.exception.InstanceNotFound: Instance
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 could not be found.
>> 2023-04-09 11:33:30.678 7 INFO nova.virt.libvirt.driver [-] [instance:
>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Instance destroyed successfully.
>> 2023-04-09 11:33:30.682 7 INFO os_vif
>> [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4
>> b0f76b5c6dcb457fa716762bbf954837 - default default] Successfully unplugged
>> vif
>> VIFBridge(active=True,address=fa:16:3e:9b:92:46,bridge_name='qbr46651303-65',has_traffic_filtering=True,id=46651303-65e1-4809-95cc-7c2d97acf16b,network=Network(532470c1-dd2f-4505-8557-ba24e9d4062b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap46651303-65')
>>
>
> All I can see is
>
>
>
>
> *b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution
> error in _get_host_uuid: Unexpected error while running command.Command:
> blkid overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '':
> oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while
> running command.*
>
> Searching the web I didn't find anything useful about this error, and I
> don't know if it's the origin of the problem.
>
> My default storage is ceph rbd, I have an HCI deployment, and all my VMs
> are for now created on Ceph only. But I have configured two other cinder
> backends, NFS and LVM, but they are node used for now.
>
> Thanks in advance.
>
> Regards.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230411/8dd018f0/attachment-0001.htm>
More information about the openstack-discuss
mailing list