[Kolla-ansible][Yoga] Cannot live migrate VMs
wodel youchi
wodel.youchi at gmail.com
Sun Apr 9 10:47:41 UTC 2023
Hi,
I cannot live-migrate any VM I have.
All my compute nodes are enabled. I have a few VMs, I am in the testing
phase.
I tried to live-migrate toward an empty node, then toward a node which
already had a VM in it. It is the same problem.
In the log files I have :
Source node :
Source
> ==> /var/log/kolla/nova/nova-compute.log <==
> 2023-04-09 11:32:49.419 7 INFO nova.compute.resource_tracker
> [req-4e91d562-d7ad-4585-a827-3db6277fd81e - - - - -] [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from
> migration f60eb077-8e1c-404a-aca5-64e5609ca02c
> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [-] [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Pre live migration failed at
> computehci02: oslo_messaging.rpc.client.RemoteError: Remote error:
> ClientException Gateway Time-out (HTTP 504)
> ['Traceback (most recent call last):\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py",
> line 165, in _process_incoming\n res =
> self.dispatcher.dispatch(message)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py",
> line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt,
> args)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py",
> line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py",
> line 72, in wrapped\n context, exc, binary)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 227, in __exit__\n self.force_reraise()\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 200, in force_reraise\n raise self.value\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py",
> line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py",
> line 1439, in decorated_function\n return function(self, context, *args,
> **kwargs)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
> line 213, in decorated_function\n kwargs[\'instance\'], e,
> sys.exc_info())\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 227, in __exit__\n self.force_reraise()\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 200, in force_reraise\n raise self.value\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
> line 201, in decorated_function\n return function(self, context, *args,
> **kwargs)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
> line 8367, in pre_live_migration\n bdm.save()\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 227, in __exit__\n self.force_reraise()\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 200, in force_reraise\n raise self.value\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
> line 8304, in pre_live_migration\n connector=connector,
> mountpoint=bdm.device_name)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
> line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', '
> File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
> line 433, in wrapper\n res = method(self, ctx, volume_id, *args,
> **kwargs)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
> line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 227, in __exit__\n self.force_reraise()\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 200, in force_reraise\n raise self.value\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
> line 801, in attachment_create\n volume_id, _connector, instance_id)\n',
> ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py",
> line 421, in substitution\n return method.func(obj, *args, **kwargs)\n',
> ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py",
> line 39, in create\n retval = self._create(\'/attachments\', body,
> \'attachment\')\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py",
> line 306, in _create\n resp, body = self.api.client.post(url,
> body=body)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
> line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n',
> ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
> line 215, in _cs_request\n return self.request(url, method,
> **kwargs)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
> line 201, in request\n raise exceptions.from_response(resp, body)\n',
> 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n'].
> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Traceback (most recent call last):
> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
> line 8501, in _do_pre_live_migration_from_source
> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] block_migration, disk, dest,
> migrate_data)
> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/rpcapi.py",
> line 948, in pre_live_migration
> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] **msg_args)
> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py",
> line 192, in call
> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] retry=self.retry,
> transport_options=self.transport_options)
> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py",
> line 128, in _send
> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7]
> transport_options=transport_options)
> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 691, in send
> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7]
> transport_options=transport_options)
> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 681, in _send
> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] raise result
> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7]
> oslo_messaging.rpc.client.RemoteError: Remote error: ClientException
> Gateway Time-out (HTTP 504)
> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] ['Traceback (most recent call
> last):\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py",
> line 165, in _process_incoming\n res =
> self.dispatcher.dispatch(message)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py",
> line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt,
> args)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py",
> line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py",
> line 72, in wrapped\n context, exc, binary)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 227, in __exit__\n self.force_reraise()\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 200, in force_reraise\n raise self.value\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py",
> line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py",
> line 1439, in decorated_function\n return function(self, context, *args,
> **kwargs)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
> line 213, in decorated_function\n kwargs[\'instance\'], e,
> sys.exc_info())\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 227, in __exit__\n self.force_reraise()\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 200, in force_reraise\n raise self.value\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
> line 201, in decorated_function\n return function(self, context, *args,
> **kwargs)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
> line 8367, in pre_live_migration\n bdm.save()\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 227, in __exit__\n self.force_reraise()\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 200, in force_reraise\n raise self.value\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
> line 8304, in pre_live_migration\n connector=connector,
> mountpoint=bdm.device_name)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
> line 395, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', '
> File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
> line 433, in wrapper\n res = method(self, ctx, volume_id, *args,
> **kwargs)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
> line 813, in attachment_create\n instance_uuid=instance_id)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 227, in __exit__\n self.force_reraise()\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 200, in force_reraise\n raise self.value\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
> line 801, in attachment_create\n volume_id, _connector, instance_id)\n',
> ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py",
> line 421, in substitution\n return method.func(obj, *args, **kwargs)\n',
> ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py",
> line 39, in create\n retval = self._create(\'/attachments\', body,
> \'attachment\')\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py",
> line 306, in _create\n resp, body = self.api.client.post(url,
> body=body)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
> line 227, in post\n return self._cs_request(url, \'POST\', **kwargs)\n',
> ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
> line 215, in _cs_request\n return self.request(url, method,
> **kwargs)\n', ' File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
> line 201, in request\n raise exceptions.from_response(resp, body)\n',
> 'cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)\n'].
> 2023-04-09 11:33:30.118 7 ERROR nova.compute.manager [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7]
> 2023-04-09 11:33:30.157 7 INFO nova.compute.manager [-] [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Swapping old allocation on
> dict_keys(['2857ae62-3519-4b13-b700-858c827dd78d']) held by migration
> f60eb077-8e1c-404a-aca5-64e5609ca02c for instance
In the destination node :
> nova-compute.log
>
>
>
>
> *2023-04-09 11:32:30.052 7 WARNING os_brick.initiator.connectors.nvmeof
> [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4
> b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution error
> in _get_host_uuid: Unexpected error while running command.Command: blkid
> overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '':
> oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while
> running command.*
>
> 2023-04-09 11:33:05.354 7 INFO nova.compute.resource_tracker
> [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Updating resource usage from
> migration f60eb077-8e1c-404a-aca5-64e5609ca02c
> 2023-04-09 11:33:05.392 7 WARNING nova.compute.resource_tracker
> [req-9d480442-87f7-4d4e-a866-c46d645ac7f2 - - - - -] Instance
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 has been moved to another host
> computehci01(computehci01). There are allocations remaining against the
> source host that might need to be removed: {'resources': {'VCPU': 1,
> 'MEMORY_MB': 2048}}.
> 2023-04-09 11:33:30.086 7 ERROR nova.volume.cinder
> [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4
> b0f76b5c6dcb457fa716762bbf954837 - default default] [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Create attachment failed for volume
> 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb. Error: Gateway Time-out (HTTP 504)
> Code: 504: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP
> 504)
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
> [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4
> b0f76b5c6dcb457fa716762bbf954837 - default default] Exception during
> message handling: cinderclient.exceptions.ClientException: Gateway Time-out
> (HTTP 504)
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server Traceback (most
> recent call last):
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py",
> line 165, in _process_incoming
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res =
> self.dispatcher.dispatch(message)
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py",
> line 309, in dispatch
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return
> self._do_dispatch(endpoint, method, ctxt, args)
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py",
> line 229, in _do_dispatch
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server result =
> func(ctxt, **new_args)
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py",
> line 72, in wrapped
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server context,
> exc, binary)
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 227, in __exit__
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
> self.force_reraise()
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 200, in force_reraise
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise
> self.value
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py",
> line 63, in wrapped
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return
> f(self, context, *args, **kw)
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py",
> line 1439, in decorated_function
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return
> function(self, context, *args, **kwargs)
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
> line 213, in decorated_function
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
> kwargs['instance'], e, sys.exc_info())
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 227, in __exit__
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
> self.force_reraise()
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 200, in force_reraise
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise
> self.value
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
> line 201, in decorated_function
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return
> function(self, context, *args, **kwargs)
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
> line 8367, in pre_live_migration
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server bdm.save()
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 227, in __exit__
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
> self.force_reraise()
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 200, in force_reraise
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise
> self.value
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py",
> line 8304, in pre_live_migration
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
> connector=connector, mountpoint=bdm.device_name)
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
> line 395, in wrapper
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res =
> method(self, ctx, *args, **kwargs)
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
> line 433, in wrapper
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server res =
> method(self, ctx, volume_id, *args, **kwargs)
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
> line 813, in attachment_create
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
> instance_uuid=instance_id)
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 227, in __exit__
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
> self.force_reraise()
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py",
> line 200, in force_reraise
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise
> self.value
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py",
> line 801, in attachment_create
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server volume_id,
> _connector, instance_id)
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/api_versions.py",
> line 421, in substitution
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return
> method.func(obj, *args, **kwargs)
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/v3/attachments.py",
> line 39, in create
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server retval =
> self._create('/attachments', body, 'attachment')
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/base.py",
> line 306, in _create
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server resp, body =
> self.api.client.post(url, body=body)
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
> line 227, in post
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return
> self._cs_request(url, 'POST', **kwargs)
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
> line 215, in _cs_request
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server return
> self.request(url, method, **kwargs)
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server File
> "/var/lib/kolla/venv/lib/python3.6/site-packages/cinderclient/client.py",
> line 201, in request
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server raise
> exceptions.from_response(resp, body)
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
> cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)
> 2023-04-09 11:33:30.115 7 ERROR oslo_messaging.rpc.server
> 2023-04-09 11:33:30.238 7 INFO nova.virt.block_device
> [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4
> b0f76b5c6dcb457fa716762bbf954837 - default default] [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Attempting to driver detach volume
> 3d7bd0af-5e4c-48fc-a027-f6c9dd6280bb from mountpoint /dev/vda
> 2023-04-09 11:33:30.239 7 WARNING nova.virt.block_device
> [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4
> b0f76b5c6dcb457fa716762bbf954837 - default default] [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Detaching volume from unknown instance
> 2023-04-09 11:33:30.240 7 WARNING nova.virt.libvirt.driver
> [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4
> b0f76b5c6dcb457fa716762bbf954837 - default default] [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] During detach_volume, instance
> disappeared.: nova.exception.InstanceNotFound: Instance
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7 could not be found.
> 2023-04-09 11:33:30.678 7 INFO nova.virt.libvirt.driver [-] [instance:
> f4c3ee82-d5cf-4b79-8608-76be7faeb2c7] Instance destroyed successfully.
> 2023-04-09 11:33:30.682 7 INFO os_vif
> [req-69822759-d982-4d01-b448-232796077002 0439953e7cfe4a13a1b4bb118b5dc3c4
> b0f76b5c6dcb457fa716762bbf954837 - default default] Successfully unplugged
> vif
> VIFBridge(active=True,address=fa:16:3e:9b:92:46,bridge_name='qbr46651303-65',has_traffic_filtering=True,id=46651303-65e1-4809-95cc-7c2d97acf16b,network=Network(532470c1-dd2f-4505-8557-ba24e9d4062b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap46651303-65')
>
All I can see is
*b0f76b5c6dcb457fa716762bbf954837 - default default] Process execution
error in _get_host_uuid: Unexpected error while running command.Command:
blkid overlay -s UUID -o valueExit code: 2Stdout: ''Stderr: '':
oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while
running command.*
Searching the web I didn't find anything useful about this error, and I
don't know if it's the origin of the problem.
My default storage is ceph rbd, I have an HCI deployment, and all my VMs
are for now created on Ceph only. But I have configured two other cinder
backends, NFS and LVM, but they are node used for now.
Thanks in advance.
Regards.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230409/e0dad245/attachment-0001.htm>
More information about the openstack-discuss
mailing list