[glance][openstack-ansible] Snapshots disappear during saving
Oliver Wenz
oliver.wenz at dhbw-mannheim.de
Thu Mar 25 09:13:24 UTC 2021
I manually set the fallocate_reserve values to 1% on the swift host and the
services don't show errors anymore (I also tried with higher values than 1%).
However, when trying to take the snapshot it still disappears and I still get
the following for the swift-proxy service:
Mar 25 08:55:53 infra1-swift-proxy-container-27169fa7 proxy-server[84]: Client
disconnected without sending last chunk (txn:
tx5590a6d463f645caaaf67-00605c4fe4) (client_ip: 192.168.110.
106)
192.168.110.106 is the IP of eth1 on my glance container so maybe that indicates
that this is a glance problem after all?
Because of the following log from the glance-api service:
Mar 25 08:54:59 infra1-glance-container-99614ac2 glance-wsgi-api[6177]:
2021-03-25 08:54:59.046 6177 INFO glance.api.v2.image_data
[req-42a0c48b-895c-4776-9e9c-586eb596b540 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to create trust: no
such option collect_timing in group [keystone_authtoken] Use the existing user
token.
Mar 25 08:55:54 infra1-glance-container-99614ac2 uwsgi[6177]: Thu Mar 25
08:55:54 2021 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client
disconnected) on request /v2/images/16f50adc-ebec-4812-96bd-2bbf6d2014b5/file
(ip 192.168.110.214) !!!
I checked the logs on the compute host (192.168.110.214) where the instance I'm
trying to take a snapshot of is running and I found this in the logs:
Mar 25 08:55:15 bc1bl14 nova-compute[2252]: 2021-03-25 08:55:15.050 2252 INFO
nova.virt.libvirt.imagecache [req-24323bc9-65bc-47b6-8ef1-81fc88b2b267 - - - -
-] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at
(/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478):
checking
Mar 25 08:55:15 bc1bl14 nova-compute[2252]: 2021-03-25 08:55:15.052 2252 INFO
nova.virt.libvirt.imagecache [req-24323bc9-65bc-47b6-8ef1-81fc88b2b267 - - - -
-] image e73512b8-7099-4691-aa48-966aa35b59ff at
(/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3):
checking
Mar 25 08:55:15 bc1bl14 nova-compute[2252]: 2021-03-25 08:55:15.759 2252 INFO
nova.virt.libvirt.imagecache [req-24323bc9-65bc-47b6-8ef1-81fc88b2b267 - - - -
-] Active base files:
/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478
/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
Mar 25 08:55:57 bc1bl14 nova-compute[2252]: 2021-03-25 08:55:57.102 2252 INFO
nova.compute.manager [req-42a0c48b-895c-4776-9e9c-586eb596b540
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Successfully reverted
task state from image_uploading on failure for instance.
Mar 25 08:55:57 bc1bl14 nova-compute[2252]: 2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server [req-42a0c48b-895c-4776-9e9c-586eb596b540
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] Exception during message handling:
glanceclient.exc.HTTPInternalServerError: HTTP 500 Internal Server Error: The
server has either erred or is incapable of performing the requested operation.
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server Traceback (most recent call last):
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
line 2478, in snapshot
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server metadata['location'] = root_disk.direct_snapshot(
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/imagebackend.py",
line 452, in direct_snapshot
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server raise NotImplementedError(_('direct_snapshot() is
not implemented'))
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server NotImplementedError: direct_snapshot() is not
implemented
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server During handling of the above exception, another
exception occurred:
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server Traceback (most recent call last):
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/server.py",
line 165, in _process_incoming
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py",
line 309, in dispatch
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt,
args)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py",
line 229, in _do_dispatch
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/exception_wrapper.py",
line 76, in wrapped
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server _emit_exception_notification(
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server self.force_reraise()
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
reraise
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server raise value
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/exception_wrapper.py",
line 69, in wrapped
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server return f(self, context, *args, **kw)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 188, in decorated_function
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server LOG.warning("Failed to revert task state for
instance. "
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server self.force_reraise()
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
reraise
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server raise value
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 159, in decorated_function
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/utils.py",
line 1434, in decorated_function
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 216, in decorated_function
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context,
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server self.force_reraise()
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
reraise
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server raise value
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 205, in decorated_function
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 236, in decorated_function
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server compute_utils.delete_image(
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server self.force_reraise()
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
reraise
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server raise value
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 232, in decorated_function
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server return function(self, context, image_id, instance,
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 3908, in snapshot_instance
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server self._snapshot_instance(context, image_id,
instance,
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 3941, in _snapshot_instance
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server self.driver.snapshot(context, instance, image_id,
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
line 2549, in snapshot
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server self._image_api.update(context,
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 1247, in update
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server return session.update(context, image_id,
image_info, data=data,
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 696, in update
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server _reraise_translated_image_exception(image_id)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 1035, in _reraise_translated_image_exception
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server raise new_exc.with_traceback(exc_trace)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 694, in update
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server image = self._update_v2(context,
sent_service_image_meta, data)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 713, in _update_v2
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server image = self._upload_data(context, image_id, data)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 589, in _upload_data
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server utils.tpool_execute(self._client.call,
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/utils.py", line
694, in tpool_execute
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server eventlet.tpool.execute(func, *args, **kwargs)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/eventlet/tpool.py",
line 129, in execute
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server six.reraise(c, e, tb)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
reraise
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server raise value
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/eventlet/tpool.py",
line 83, in tworker
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server rv = meth(*args, **kwargs)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 192, in call
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server result = getattr(controller, method)(*args,
**kwargs)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/utils.py",
line 600, in inner
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server return RequestIdProxy(wrapped(*args, **kwargs))
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/v2/images.py",
line 289, in upload
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server resp, body = self.http_client.put(url,
headers=hdrs, data=body)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/keystoneauth1/adapter.py",
line 404, in put
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server return self.request(url, 'PUT', **kwargs)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/http.py",
line 380, in request
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server return self._handle_response(resp)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/http.py",
line 120, in _handle_response
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server raise exc.from_response(resp, resp.content)
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server glanceclient.exc.HTTPInternalServerError: HTTP 500
Internal Server Error: The server has either erred or is incapable of performing
the requested operation.
2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server
Any hints where the error might be coming from are much appreciated!
Kind regards,
Oliver
More information about the openstack-discuss
mailing list