[Kolla-ansible][Yoga] Cannot live migrate VMs
wodel youchi
wodel.youchi at gmail.com
Tue Apr 11 11:57:47 UTC 2023
Hi,
Indeed, there was a mistake in the cinder.conf file.
Thanks.
Regards.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Virus-free.www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Le mar. 11 avr. 2023 à 10:45, Pierre Riteau <pierre at stackhpc.com> a écrit :
> You should check the status of your Cinder services.
>
> On Sun, 9 Apr 2023 at 19:43, wodel youchi <wodel.youchi at gmail.com> wrote:
>
>> Hi,
>>
>> I cannot live-migrate any VM I have.
>> All my compute nodes are enabled. I have a few VMs, I am in the testing
>> phase.
>> I tried to live-migrate toward an empty node, then toward a node which
>> already had a VM in it. It is the same problem.
>>
>> In the log files I have :
>>
>> Source node :
>> Source
>>
>>> ==> /var/log/kolla/nova/nova-compute.log <==
>>> 2023-04-09 11:32:49.419 7 INFO nova.compute.resource_tracker
>>> [req-4e91d562-d7ad-4585-a827-3db6277fd81e - - - - -] [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from
>>> migration f60eb077-8e1c-404a-aca5-64e5609ca02c
>>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [-] [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Pre live migration failed at
>>> computehci02: oslo_messaging.rpc.client.RemoteError: Remote error:
>>> ClientException Gateway Time-out (HTTP 504)
>>> ['Traceback (most recent call last):\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py",
>>> line 165, in _process_incoming\n res =
>>> self.dispatcher.dispatch(message)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py",
>>> line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt,
>>> args)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py",
>>> line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py",
>>> line 72, in wrapped\n context, exc, binary)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 227, in __exit__\n self.force_reraise()\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 200, in force_reraise\n raise self.value\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py",
>>> line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py",
>>> line 1439, in decorated_function\n return function(self, context, *args,
>>> **kwargs)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>>> line 213, in decorated_function\n kwargs[\'instance\'], e,
>>> sys.exc_info())\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 227, in __exit__\n self.force_reraise()\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 200, in force_reraise\n raise self.value\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>>> line 201, in decorated_function\n return function(self, context, *args,
>>> **kwargs)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>>> line 8367, in pre_live_migration\n bdm.save()\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 227, in __exit__\n self.force_reraise()\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 200, in force_reraise\n raise self.value\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>>> line 8304, in pre_live_migration\n connector=connector,
>>> mountpoint=bdm.device_name)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>>> line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', '
>>> File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>>> line 433, in wrapper\n res = method(self, ctx, volume_id, *args,
>>> **kwargs)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>>> line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 227, in __exit__\n self.force_reraise()\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 200, in force_reraise\n raise self.value\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>>> line 801, in attachment_create\n volume_id, _connector, instance_id)\n',
>>> ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py",
>>> line 421, in substitution\n return method.func(obj, *args, **kwargs)\n',
>>> ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py",
>>> line 39, in create\n retval = self._create(\'/attachments\', body,
>>> \'attachment\')\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py",
>>> line 306, in _create\n resp, body = self.api.client.post(url,
>>> body=body)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
>>> line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n',
>>> ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
>>> line 215, in _cs_request\n return self.request(url, method,
>>> **kwargs)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
>>> line 201, in request\n raise exceptions.from_response(resp, body)\n',
>>> 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n'].
>>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Traceback (most recent call last):
>>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>>> line 8501, in _do_pre_live_migration_from_source
>>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] block_migration, disk, dest,
>>> migrate_data)
>>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/rpcapi.py",
>>> line 948, in pre_live_migration
>>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] **msg_args)
>>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py",
>>> line 192, in call
>>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] retry=self.retry,
>>> transport_options=self.transport_options)
>>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py",
>>> line 128, in _send
>>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7]
>>> transport_options=transport_options)
>>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py",
>>> line 691, in send
>>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7]
>>> transport_options=transport_options)
>>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py",
>>> line 681, in _send
>>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] raise result
>>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7]
>>> oslo_messaging.rpc.client.RemoteError: Remote error: ClientException
>>> Gateway Time-out (HTTP 504)
>>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] ['Traceback (most recent call
>>> last):\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py",
>>> line 165, in _process_incoming\n res =
>>> self.dispatcher.dispatch(message)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py",
>>> line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt,
>>> args)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py",
>>> line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py",
>>> line 72, in wrapped\n context, exc, binary)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 227, in __exit__\n self.force_reraise()\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 200, in force_reraise\n raise self.value\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py",
>>> line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py",
>>> line 1439, in decorated_function\n return function(self, context, *args,
>>> **kwargs)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>>> line 213, in decorated_function\n kwargs[\'instance\'], e,
>>> sys.exc_info())\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 227, in __exit__\n self.force_reraise()\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 200, in force_reraise\n raise self.value\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>>> line 201, in decorated_function\n return function(self, context, *args,
>>> **kwargs)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>>> line 8367, in pre_live_migration\n bdm.save()\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 227, in __exit__\n self.force_reraise()\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 200, in force_reraise\n raise self.value\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>>> line 8304, in pre_live_migration\n connector=connector,
>>> mountpoint=bdm.device_name)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>>> line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', '
>>> File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>>> line 433, in wrapper\n res = method(self, ctx, volume_id, *args,
>>> **kwargs)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>>> line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 227, in __exit__\n self.force_reraise()\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 200, in force_reraise\n raise self.value\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>>> line 801, in attachment_create\n volume_id, _connector, instance_id)\n',
>>> ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py",
>>> line 421, in substitution\n return method.func(obj, *args, **kwargs)\n',
>>> ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py",
>>> line 39, in create\n retval = self._create(\'/attachments\', body,
>>> \'attachment\')\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py",
>>> line 306, in _create\n resp, body = self.api.client.post(url,
>>> body=body)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
>>> line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n',
>>> ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
>>> line 215, in _cs_request\n return self.request(url, method,
>>> **kwargs)\n', ' File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
>>> line 201, in request\n raise exceptions.from_response(resp, body)\n',
>>> 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n'].
>>> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7]
>>> 2023-04-09 11:33:30.157 7 INFO nova.compute.manager [-] [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Swapping old allocation on
>>> dict_keys(['2857ae62-3519-4b13-b700-858c827dd78d']) held by migration
>>> f60eb077-8e1c-404a-aca5-64e5609ca02c for instance
>>
>>
>>
>> In the destination node :
>>
>>> nova-compute.log
>>>
>>>
>>>
>>>
>>> *2023-04-09 11:32:30.052 7 WARNING os_brick.initiator.connectors.nvmeof
>>> [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4
>>> b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error
>>> in _get_host_uuid: Unexpected error while running command.Command: blkid
>>> overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '':
>>> oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while
>>> running command.*
>>>
>>> 2023-04-09 11:33:05.354 7 INFO nova.compute.resource_tracker
>>> [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from
>>> migration f60eb077-8e1c-404a-aca5-64e5609ca02c
>>> 2023-04-09 11:33:05.392 7 WARNING nova.compute.resource_tracker
>>> [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] Instance
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 has been moved to another host
>>> computehci01(computehci01). There are allocations remaining against the
>>> source host that might need to be removed: {'resources': {'VCPU': 1,
>>> 'MEMORY_MB': 2048}}.
>>> 2023-04-09 11:33:30.086 7 ERROR nova.volume.cinder
>>> [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4
>>> b0f76b5c6dcb457fa716762bbf954837 - default default] [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Create attachment failed for volume
>>> 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb. Error: Gateway Time-out (HTTP 504)
>>> Code: 504: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP
>>> 504)
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
>>> [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4
>>> b0f76b5c6dcb457fa716762bbf954837 - default default] Exception during
>>> message handling: cinderclient.exceptions.ClientException: Gateway Time-out
>>> (HTTP 504)
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server Traceback
>>> (most recent call last):
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py",
>>> line 165, in _process_incoming
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res =
>>> self.dispatcher.dispatch(message)
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py",
>>> line 309, in dispatch
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return
>>> self._do_dispatch(endpoint, method, ctxt, args)
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py",
>>> line 229, in _do_dispatch
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server result =
>>> func(ctxt, **new_args)
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py",
>>> line 72, in wrapped
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server context,
>>> exc, binary)
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 227, in __exit__
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
>>> self.force_reraise()
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 200, in force_reraise
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise
>>> self.value
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py",
>>> line 63, in wrapped
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return
>>> f(self, context, *args, **kw)
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py",
>>> line 1439, in decorated_function
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return
>>> function(self, context, *args, **kwargs)
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>>> line 213, in decorated_function
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
>>> kwargs['instance'], e, sys.exc_info())
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 227, in __exit__
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
>>> self.force_reraise()
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 200, in force_reraise
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise
>>> self.value
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>>> line 201, in decorated_function
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return
>>> function(self, context, *args, **kwargs)
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>>> line 8367, in pre_live_migration
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server bdm.save()
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 227, in __exit__
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
>>> self.force_reraise()
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 200, in force_reraise
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise
>>> self.value
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
>>> line 8304, in pre_live_migration
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
>>> connector=connector, mountpoint=bdm.device_name)
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>>> line 395, in wrapper
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res =
>>> method(self, ctx, *args, **kwargs)
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>>> line 433, in wrapper
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res =
>>> method(self, ctx, volume_id, *args, **kwargs)
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>>> line 813, in attachment_create
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
>>> instance_uuid=instance_id)
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 227, in __exit__
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
>>> self.force_reraise()
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
>>> line 200, in force_reraise
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise
>>> self.value
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
>>> line 801, in attachment_create
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server volume_id,
>>> _connector, instance_id)
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py",
>>> line 421, in substitution
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return
>>> method.func(obj, *args, **kwargs)
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py",
>>> line 39, in create
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server retval =
>>> self._create('/attachments', body, 'attachment')
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py",
>>> line 306, in _create
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server resp, body
>>> = self.api.client.post(url, body=body)
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
>>> line 227, in post
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return
>>> self._cs_request(url, 'POST', **kwargs)
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
>>> line 215, in _cs_request
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return
>>> self.request(url, method, **kwargs)
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
>>> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
>>> line 201, in request
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise
>>> exceptions.from_response(resp, body)
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
>>> cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)
>>> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
>>> 2023-04-09 11:33:30.238 7 INFO nova.virt.block_device
>>> [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4
>>> b0f76b5c6dcb457fa716762bbf954837 - default default] [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Attempting to driver detach volume
>>> 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb from mountpoint /dev/vda
>>> 2023-04-09 11:33:30.239 7 WARNING nova.virt.block_device
>>> [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4
>>> b0f76b5c6dcb457fa716762bbf954837 - default default] [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Detaching volume from unknown instance
>>> 2023-04-09 11:33:30.240 7 WARNING nova.virt.libvirt.driver
>>> [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4
>>> b0f76b5c6dcb457fa716762bbf954837 - default default] [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] During detach_volume, instance
>>> disappeared.: nova.exception.InstanceNotFound: Instance
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 could not be found.
>>> 2023-04-09 11:33:30.678 7 INFO nova.virt.libvirt.driver [-] [instance:
>>> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Instance destroyed successfully.
>>> 2023-04-09 11:33:30.682 7 INFO os_vif
>>> [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4
>>> b0f76b5c6dcb457fa716762bbf954837 - default default] Successfully unplugged
>>> vif
>>> VIFBridge(active=True,address=fa:16:3e:9b:92:46,bridge_name='qbr46651303-65',has_traffic_filtering=True,id=46651303-65e1-4809-95cc-7c2d97acf16b,network=Network(532470c1-dd2f-4505-8557-ba24e9d4062b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap46651303-65')
>>>
>>
>> All I can see is
>>
>>
>>
>>
>> *b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution
>> error in _get_host_uuid: Unexpected error while running command.Command:
>> blkid overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '':
>> oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while
>> running command.*
>>
>> Searching the web I didn't find anything useful about this error, and I
>> don't know if it's the origin of the problem.
>>
>> My default storage is ceph rbd, I have an HCI deployment, and all my VMs
>> are for now created on Ceph only. But I have configured two other cinder
>> backends, NFS and LVM, but they are node used for now.
>>
>> Thanks in advance.
>>
>> Regards.
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230411/e775a16b/attachment-0001.htm>
More information about the openstack-discuss
mailing list