[glance][openstack-ansible] Snapshots disappear during saving
Dmitriy Rabotyagov
noonedeadpunk at ya.ru
Thu Mar 25 10:01:54 UTC 2021
Oh, I have a guess what this might actually be. During snapshot upload process user token that is used for the upload might get expired. If that's the case, following changes in user_variables might help to resolve the issue:
glance_glance_api_conf_overrides:
keystone_authtoken:
service_token_roles_required: True
service_token_roles: service
25.03.2021, 11:20, "Oliver Wenz" <oliver.wenz at dhbw-mannheim.de>:
> I manually set the fallocate_reserve values to 1% on the swift host and the
> services don't show errors anymore (I also tried with higher values than 1%).
>
> However, when trying to take the snapshot it still disappears and I still get
> the following for the swift-proxy service:
>
> Mar 25 08:55:53 infra1-swift-proxy-container-27169fa7 proxy-server[84]: Client
> disconnected without sending last chunk (txn:
> tx5590a6d463f645caaaf67-00605c4fe4) (client_ip: 192.168.110.
> 106)
>
> 192.168.110.106 is the IP of eth1 on my glance container so maybe that indicates
> that this is a glance problem after all?
>
> Because of the following log from the glance-api service:
>
> Mar 25 08:54:59 infra1-glance-container-99614ac2 glance-wsgi-api[6177]:
> 2021-03-25 08:54:59.046 6177 INFO glance.api.v2.image_data
> [req-42a0c48b-895c-4776-9e9c-586eb596b540 956806468e9f43dbaad1807a5208de52
> ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to create trust: no
> such option collect_timing in group [keystone_authtoken] Use the existing user
> token.
> Mar 25 08:55:54 infra1-glance-container-99614ac2 uwsgi[6177]: Thu Mar 25
> 08:55:54 2021 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client
> disconnected) on request /v2/images/16f50adc-ebec-4812-96bd-2bbf6d2014b5/file
> (ip 192.168.110.214) !!!
>
> I checked the logs on the compute host (192.168.110.214) where the instance I'm
> trying to take a snapshot of is running and I found this in the logs:
>
> Mar 25 08:55:15 bc1bl14 nova-compute[2252]: 2021-03-25 08:55:15.050 2252 INFO
> nova.virt.libvirt.imagecache [req-24323bc9-65bc-47b6-8ef1-81fc88b2b267 - - - -
> -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at
> (/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478):
> checking
> Mar 25 08:55:15 bc1bl14 nova-compute[2252]: 2021-03-25 08:55:15.052 2252 INFO
> nova.virt.libvirt.imagecache [req-24323bc9-65bc-47b6-8ef1-81fc88b2b267 - - - -
> -] image e73512b8-7099-4691-aa48-966aa35b59ff at
> (/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3):
> checking
> Mar 25 08:55:15 bc1bl14 nova-compute[2252]: 2021-03-25 08:55:15.759 2252 INFO
> nova.virt.libvirt.imagecache [req-24323bc9-65bc-47b6-8ef1-81fc88b2b267 - - - -
> -] Active base files:
> /var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478
> /var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
> Mar 25 08:55:57 bc1bl14 nova-compute[2252]: 2021-03-25 08:55:57.102 2252 INFO
> nova.compute.manager [req-42a0c48b-895c-4776-9e9c-586eb596b540
> 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
> default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Successfully reverted
> task state from image_uploading on failure for instance.
> Mar 25 08:55:57 bc1bl14 nova-compute[2252]: 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server [req-42a0c48b-895c-4776-9e9c-586eb596b540
> 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
> default] Exception during message handling:
> glanceclient.exc.HTTPInternalServerError: HTTP 500 Internal Server Error: The
> server has either erred or is incapable of performing the requested operation.
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server Traceback (most recent call last):
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
> line 2478, in snapshot
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server metadata['location'] = root_disk.direct_snapshot(
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/imagebackend.py",
> line 452, in direct_snapshot
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server raise NotImplementedError(_('direct_snapshot() is
> not implemented'))
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server NotImplementedError: direct_snapshot() is not
> implemented
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server During handling of the above exception, another
> exception occurred:
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server Traceback (most recent call last):
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/server.py",
> line 165, in _process_incoming
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py",
> line 309, in dispatch
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt,
> args)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py",
> line 229, in _do_dispatch
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server result = func(ctxt, **new_args)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/exception_wrapper.py",
> line 76, in wrapped
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server _emit_exception_notification(
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
> line 220, in __exit__
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server self.force_reraise()
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
> line 196, in force_reraise
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
> reraise
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server raise value
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/exception_wrapper.py",
> line 69, in wrapped
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server return f(self, context, *args, **kw)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
> line 188, in decorated_function
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server LOG.warning("Failed to revert task state for
> instance. "
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
> line 220, in __exit__
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server self.force_reraise()
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
> line 196, in force_reraise
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
> reraise
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server raise value
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
> line 159, in decorated_function
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/utils.py",
> line 1434, in decorated_function
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
> line 216, in decorated_function
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context,
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
> line 220, in __exit__
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server self.force_reraise()
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
> line 196, in force_reraise
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
> reraise
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server raise value
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
> line 205, in decorated_function
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
> line 236, in decorated_function
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server compute_utils.delete_image(
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
> line 220, in __exit__
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server self.force_reraise()
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
> line 196, in force_reraise
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
> reraise
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server raise value
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
> line 232, in decorated_function
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server return function(self, context, image_id, instance,
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
> line 3908, in snapshot_instance
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server self._snapshot_instance(context, image_id,
> instance,
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py",
> line 3941, in _snapshot_instance
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server self.driver.snapshot(context, instance, image_id,
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
> line 2549, in snapshot
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server self._image_api.update(context,
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
> line 1247, in update
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server return session.update(context, image_id,
> image_info, data=data,
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
> line 696, in update
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server _reraise_translated_image_exception(image_id)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
> line 1035, in _reraise_translated_image_exception
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server raise new_exc.with_traceback(exc_trace)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
> line 694, in update
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server image = self._update_v2(context,
> sent_service_image_meta, data)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
> line 713, in _update_v2
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server image = self._upload_data(context, image_id, data)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
> line 589, in _upload_data
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server utils.tpool_execute(self._client.call,
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/utils.py", line
> 694, in tpool_execute
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server eventlet.tpool.execute(func, *args, **kwargs)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/eventlet/tpool.py",
> line 129, in execute
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server six.reraise(c, e, tb)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in
> reraise
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server raise value
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/eventlet/tpool.py",
> line 83, in tworker
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server rv = meth(*args, **kwargs)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py",
> line 192, in call
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server result = getattr(controller, method)(*args,
> **kwargs)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/utils.py",
> line 600, in inner
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server return RequestIdProxy(wrapped(*args, **kwargs))
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/v2/images.py",
> line 289, in upload
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server resp, body = self.http_client.put(url,
> headers=hdrs, data=body)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/keystoneauth1/adapter.py",
> line 404, in put
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server return self.request(url, 'PUT', **kwargs)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/http.py",
> line 380, in request
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server return self._handle_response(resp)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server File
> "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/http.py",
> line 120, in _handle_response
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server raise exc.from_response(resp, resp.content)
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server glanceclient.exc.HTTPInternalServerError: HTTP 500
> Internal Server Error: The server has either erred or is incapable of performing
> the requested operation.
> 2021-03-25 08:55:57.106 2252 ERROR
> oslo_messaging.rpc.server
>
> Any hints where the error might be coming from are much appreciated!
>
> Kind regards,
> Oliver
--
Kind Regards,
Dmitriy Rabotyagov
More information about the openstack-discuss
mailing list