Re: [glance][openstack-ansible] Snapshots disappear during saving

25 Mar 2021

      I manually set the fallocate_reserve values to 1% on the swift host and the
services don't show errors anymore (I also tried with higher values than 1%).

However, when trying to take the snapshot it still disappears and I still get
the following for the swift-proxy service:

Mar 25 08:55:53 infra1-swift-proxy-container-27169fa7 proxy-server[84]: Client
disconnected without sending last chunk (txn:
tx5590a6d463f645caaaf67-00605c4fe4) (client_ip: 192.168.110.
106)

192.168.110.106 is the IP of eth1 on my glance container so maybe that indicates
that this is a glance problem after all?

Because of the following log from the glance-api service:

Mar 25 08:54:59 infra1-glance-container-99614ac2 glance-wsgi-api[6177]:
2021-03-25 08:54:59.046 6177 INFO glance.api.v2.image_data
[req-42a0c48b-895c-4776-9e9c-586eb596b540 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to create trust: no
such option collect_timing in group [keystone_authtoken] Use the existing user
token.
Mar 25 08:55:54 infra1-glance-container-99614ac2 uwsgi[6177]: Thu Mar 25
08:55:54 2021 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client
disconnected) on request /v2/images/16f50adc-ebec-4812-96bd-2bbf6d2014b5/file
(ip 192.168.110.214) !!!

I checked the logs on the compute host (192.168.110.214) where the instance I'm
trying to take a snapshot of is running and I found this in the logs:

Mar 25 08:55:15 bc1bl14 nova-compute[2252]: 2021-03-25 08:55:15.050 2252 INFO
nova.virt.libvirt.imagecache [req-24323bc9-65bc-47b6-8ef1-81fc88b2b267 - - - -
-] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at
(/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478):
checking
Mar 25 08:55:15 bc1bl14 nova-compute[2252]: 2021-03-25 08:55:15.052 2252 INFO
nova.virt.libvirt.imagecache [req-24323bc9-65bc-47b6-8ef1-81fc88b2b267 - - - -
-] image e73512b8-7099-4691-aa48-966aa35b59ff at
(/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3):
checking
Mar 25 08:55:15 bc1bl14 nova-compute[2252]: 2021-03-25 08:55:15.759 2252 INFO
nova.virt.libvirt.imagecache [req-24323bc9-65bc-47b6-8ef1-81fc88b2b267 - - - -
-] Active base files:
/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478
/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
Mar 25 08:55:57 bc1bl14 nova-compute[2252]: 2021-03-25 08:55:57.102 2252 INFO
nova.compute.manager [req-42a0c48b-895c-4776-9e9c-586eb596b540
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Successfully reverted
task state from image_uploading on failure for instance.
Mar 25 08:55:57 bc1bl14 nova-compute[2252]: 2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server [req-42a0c48b-895c-4776-9e9c-586eb596b540
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] Exception during message handling:
glanceclient.exc.HTTPInternalServerError: HTTP 500 Internal Server Error: The
server has either erred or is incapable of performing the requested operation.
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server Traceback (most recent call last):
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
line 2478, in snapshot
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     metadata['location'] = root_disk.direct_snapshot(
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/imagebackend.py",
line 452, in direct_snapshot
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     raise NotImplementedError(_('direct_snapshot() is
not implemented'))
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server NotImplementedError: direct_snapshot() is not
implemented
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server 
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server During handling of the above exception, another
exception occurred:
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server 
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server Traceback (most recent call last):
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/server.py",
line 165, in _process_incoming
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py",
line 309, in dispatch
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt,
args)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py",
line 229, in _do_dispatch
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     result = func(ctxt, **new_args)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/exception_wrapper.py",
line 76, in wrapped
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     _emit_exception_notification(
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     self.force_reraise()
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
reraise
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     raise value
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/exception_wrapper.py",
line 69, in wrapped
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     return f(self, context, *args, **kw)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 188, in decorated_function
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     LOG.warning("Failed to revert task state for
instance. "
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     self.force_reraise()
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
reraise
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     raise value
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 159, in decorated_function
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/utils.py",
line 1434, in decorated_function
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 216, in decorated_function
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     compute_utils.add_instance_fault_from_exc(context,
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     self.force_reraise()
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
reraise
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     raise value
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 205, in decorated_function
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 236, in decorated_function
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     compute_utils.delete_image(
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     self.force_reraise()
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
reraise
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     raise value
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 232, in decorated_function
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     return function(self, context, image_id, instance,
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 3908, in snapshot_instance
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     self._snapshot_instance(context, image_id,
instance,
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
line 3941, in _snapshot_instance
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     self.driver.snapshot(context, instance, image_id,
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
line 2549, in snapshot
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     self._image_api.update(context,
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 1247, in update
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     return session.update(context, image_id,
image_info, data=data,
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 696, in update
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     _reraise_translated_image_exception(image_id)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 1035, in _reraise_translated_image_exception
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     raise new_exc.with_traceback(exc_trace)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 694, in update
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     image = self._update_v2(context,
sent_service_image_meta, data)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 713, in _update_v2
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     image = self._upload_data(context, image_id, data)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 589, in _upload_data
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     utils.tpool_execute(self._client.call,
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/utils.py", line
694, in tpool_execute
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     eventlet.tpool.execute(func, *args, **kwargs)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/eventlet/tpool.py",
line 129, in execute
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     six.reraise(c, e, tb)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
reraise
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     raise value
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/eventlet/tpool.py",
line 83, in tworker
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
line 192, in call
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     result = getattr(controller, method)(*args,
**kwargs)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/utils.py",
line 600, in inner
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     return RequestIdProxy(wrapped(*args, **kwargs))
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/v2/images.py",
line 289, in upload
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     resp, body = self.http_client.put(url,
headers=hdrs, data=body)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/keystoneauth1/adapter.py",
line 404, in put
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     return self.request(url, 'PUT', **kwargs)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/http.py",
line 380, in request
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     return self._handle_response(resp)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server   File
"/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/http.py",
line 120, in _handle_response
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server     raise exc.from_response(resp, resp.content)
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server glanceclient.exc.HTTPInternalServerError: HTTP 500
Internal Server Error: The server has either erred or is incapable of performing
the requested operation.
                                            2021-03-25 08:55:57.106 2252 ERROR
oslo_messaging.rpc.server 

Any hints where the error might be coming from are much appreciated!

Kind regards,
Oliver

Re: [glance][openstack-ansible] Snapshots disappear during saving

Oliver Wenz