[glance][openstack-ansible] Snapshots disappear during saving
Oliver Wenz
oliver.wenz at dhbw-mannheim.de
Wed Mar 17 11:34:50 UTC 2021
Hi!
We are currently experiencing problems with our OpenStack Ansible Victoria cloud
when trying to create snapshots from instances. In some cases, everything works
but often the pending snapshots just disappear.
When this happens, the following glance-api.service errors show up:
Mar 17 08:33:12 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
08:33:11.975 85 INFO glance.api.v2.image_data
[req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to create trust: no
such option collect_timing in group [keystone_authtoken] Use the existing user
token.
Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
08:35:15.283 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] REQ: curl -i
http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/1b9af05a-f7c9-4315-9354-e09f1df66321-00001
-X PUT -H "X-Auth-Token: gAAAAABgUb7IBvZ_..."
Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
08:35:15.285 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] RESP STATUS: 504 Gateway Time-out
Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
08:35:15.285 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] RESP HEADERS: {'content-length': '92', 'cache-control': 'no-cache',
'content-type': 'text/html', 'connection': 'close'}
Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
08:35:15.286 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] RESP BODY: b"<html><body><h1>504 Gateway Time-out</h1>\nThe server
didn't respond in time.\n</body></html>\n"
Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
08:35:16.306 85 ERROR glance_store._drivers.swift.store
[req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Error during chunked upload
to backend, deleting stale chunks.: swiftclient.exceptions.ClientException:
put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...)
failure and no ability to reset contents for reupload.
Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
08:35:16.320 85 ERROR glance_store._drivers.swift.store
[req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to add object to
Swift.
Got error
from Swift: put_object('glance_images',
'1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to
reset contents for reupload..: swiftclient.exceptions.ClientException:
put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...)
failure and no ability to reset contents for reupload.
Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
08:35:16.327 85 ERROR glance.api.v2.image_data
[req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to upload image data
due to internal error: glance_store.exceptions.BackendException: Failed to add
object to Swift.
Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi
[req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Caught error: Failed to add
object to Swift.
Got error
from Swift: put_object('glance_images',
'1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to
reset contents for reupload..: glance_store.exceptions.BackendException: Failed
to add object to Swift.
Got error
from Swift: put_object('glance_images',
'1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to
reset contents for reupload..
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi Traceback (most recent call last):
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py",
line 1014, in add
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi self._delete_stale_chunks(
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise()
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value,
self.tb)
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703,
in reraise
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi raise value
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py",
line 1003, in add
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi
manager.get_connection().put_object(
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py",
line 1960, in put_object
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi return self._retry(reset_func,
put_object, container, obj, contents,
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py",
line 1843, in _retry
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi reset_func(func, *args, **kwargs)
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py",
line 1940, in _default_reset
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi raise
ClientException('put_object(%r, %r, ...) failure and no '
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi swiftclient.exceptions.ClientException:
put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...)
failure and no ability to reset contents for reupload.
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi During handling of the above exception,
another exception occurred:
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi Traceback (most recent call last):
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py",
line 1347, in __call__
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi action_result =
self.dispatch(self.controller, action,
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py",
line 1391, in dispatch
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi return method(*args, **kwargs)
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/utils.py",
line 416, in wrapped
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi return func(self, req, *args,
**kwargs)
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py",
line 298, in upload
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi self._restore(image_repo, image)
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise()
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value,
self.tb)
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703,
in reraise
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi raise value
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py",
line 163, in upload
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi image.set_data(data, size,
backend=backend)
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/domain/proxy.py",
line 208, in set_data
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi self.base.set_data(data, size,
backend=backend, set_active=set_active)
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py",
line 501, in set_data
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi _send_notification(notify_error,
'image.upload', msg)
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise()
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value,
self.tb)
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703,
in reraise
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi raise value
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py",
line 447, in set_data
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi self.repo.set_data(data, size,
backend=backend,
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/policy.py",
line 198, in set_data
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi return self.image.set_data(*args,
**kwargs)
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/quota/__init__.py",
line 318, in set_data
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi self.image.set_data(data,
size=size, backend=backend,
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py",
line 567, in set_data
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi self._upload_to_store(data,
verifier, backend, size)
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py",
line 458, in _upload_to_store
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi multihash, loc_meta) =
self.store_api.add_with_multihash(
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py",
line 398, in add_with_multihash
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi return
store_add_to_backend_with_multihash(
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py",
line 480, in store_add_to_backend_with_multihash
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi (location, size, checksum,
multihash, metadata) = store.add(
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/driver.py",
line 279, in add_adapter
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi metadata_dict) =
store_add_fun(*args, **kwargs)
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/capabilities.py",
line 176, in op_checker
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi return store_op_fun(store, *args,
**kwargs)
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py",
line 1082, in add
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi raise
glance_store.BackendException(msg)
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi
glance_store.exceptions.BackendException: Failed to add object to Swift.
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi Got error from Swift:
put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...)
failure and no ability to reset contents for reupload..
2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi
Mar 17 08:35:16 infra1-glance-container-99614ac2 uwsgi[85]: Wed Mar 17 08:35:16
2021 - uwsgi_response_writev_headers_and_body_do(): Connection reset by peer
[core/writer.c line 306] during PUT
/v2/images/1b9af05a-f7c9-4315-9354-e09f1df66321/file (192.168.110.215)
Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
08:35:16.441 85 CRITICAL glance [req-32345bbf-a88f-450e-90b2-69c6a5804a7a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] Unhandled error: OSError: write error
2021-03-17
08:35:16.441 85 ERROR glance OSError: write error
2021-03-17
08:35:16.441 85 ERROR glance
It seems as if the problems occur more often with large instances, i.e. there
are fewer problems with new Ubuntu 20.04 instances but after an 'apt-upgrade'
on the instance, the error occurs every time a snapshot is taken. Any help is
much appreciated!
Kind regards,
Oliver
More information about the openstack-discuss
mailing list