[glance][openstack-ansible] Snapshots disappear during saving

Oliver Wenz oliver.wenz at dhbw-mannheim.de
Wed Mar 17 11:34:50 UTC 2021


Hi!
We are currently experiencing problems with our OpenStack Ansible Victoria cloud
when trying to create snapshots from instances. In some cases, everything works
but often the pending snapshots just disappear.
 
When this happens, the following glance-api.service errors show up:
 
Mar 17 08:33:12 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
08:33:11.975 85 INFO glance.api.v2.image_data
[req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to create trust: no
such option collect_timing in group [keystone_authtoken] Use the existing user
token.
Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
08:35:15.283 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] REQ: curl -i
http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/1b9af05a-f7c9-4315-9354-e09f1df66321-00001
-X PUT -H "X-Auth-Token: gAAAAABgUb7IBvZ_..."
Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
08:35:15.285 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] RESP STATUS: 504 Gateway Time-out
Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
08:35:15.285 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] RESP HEADERS: {'content-length': '92', 'cache-control': 'no-cache',
'content-type': 'text/html', 'connection': 'close'}
Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
08:35:15.286 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] RESP BODY: b"<html><body><h1>504 Gateway Time-out</h1>\nThe server
didn't respond in time.\n</body></html>\n"
Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
08:35:16.306 85 ERROR glance_store._drivers.swift.store
[req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Error during chunked upload
to backend, deleting stale chunks.: swiftclient.exceptions.ClientException:
put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...)
failure and no ability to reset contents for reupload.
Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
08:35:16.320 85 ERROR glance_store._drivers.swift.store
[req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to add object to
Swift.
                                                                      Got error
from Swift: put_object('glance_images',
'1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to
reset contents for reupload..: swiftclient.exceptions.ClientException:
put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...)
failure and no ability to reset contents for reupload.
Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
08:35:16.327 85 ERROR glance.api.v2.image_data
[req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to upload image data
due to internal error: glance_store.exceptions.BackendException: Failed to add
object to Swift.
Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi
[req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52
ebe0fe5f3893495e82598c07716f5d45 - default default] Caught error: Failed to add
object to Swift.
                                                                      Got error
from Swift: put_object('glance_images',
'1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to
reset contents for reupload..: glance_store.exceptions.BackendException: Failed
to add object to Swift.
                                                                      Got error
from Swift: put_object('glance_images',
'1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to
reset contents for reupload..
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi Traceback (most recent call last):
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py",
line 1014, in add
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     self._delete_stale_chunks(
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     self.force_reraise()
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     six.reraise(self.type_, self.value,
self.tb)
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703,
in reraise
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     raise value
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py",
line 1003, in add
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi
    manager.get_connection().put_object(
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py",
line 1960, in put_object
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     return self._retry(reset_func,
put_object, container, obj, contents,
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py",
line 1843, in _retry
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     reset_func(func, *args, **kwargs)
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py",
line 1940, in _default_reset
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     raise
ClientException('put_object(%r, %r, ...) failure and no '
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi swiftclient.exceptions.ClientException:
put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...)
failure and no ability to reset contents for reupload.
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi During handling of the above exception,
another exception occurred:
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi Traceback (most recent call last):
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py",
line 1347, in __call__
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     action_result =
self.dispatch(self.controller, action,
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py",
line 1391, in dispatch
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     return method(*args, **kwargs)
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/utils.py",
line 416, in wrapped
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     return func(self, req, *args,
**kwargs)
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py",
line 298, in upload
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     self._restore(image_repo, image)
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     self.force_reraise()
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     six.reraise(self.type_, self.value,
self.tb)
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703,
in reraise
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     raise value
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py",
line 163, in upload
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     image.set_data(data, size,
backend=backend)
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/domain/proxy.py",
line 208, in set_data
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     self.base.set_data(data, size,
backend=backend, set_active=set_active)
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py",
line 501, in set_data
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     _send_notification(notify_error,
'image.upload', msg)
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     self.force_reraise()
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     six.reraise(self.type_, self.value,
self.tb)
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703,
in reraise
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     raise value
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py",
line 447, in set_data
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     self.repo.set_data(data, size,
backend=backend,
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/policy.py",
line 198, in set_data
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     return self.image.set_data(*args,
**kwargs)
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/quota/__init__.py",
line 318, in set_data
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     self.image.set_data(data,
size=size, backend=backend,
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py",
line 567, in set_data
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     self._upload_to_store(data,
verifier, backend, size)
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py",
line 458, in _upload_to_store
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     multihash, loc_meta) =
self.store_api.add_with_multihash(
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py",
line 398, in add_with_multihash
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     return
store_add_to_backend_with_multihash(
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py",
line 480, in store_add_to_backend_with_multihash
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     (location, size, checksum,
multihash, metadata) = store.add(
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/driver.py",
line 279, in add_adapter
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     metadata_dict) =
store_add_fun(*args, **kwargs)
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/capabilities.py",
line 176, in op_checker
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     return store_op_fun(store, *args,
**kwargs)
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi   File
"/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py",
line 1082, in add
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi     raise
glance_store.BackendException(msg)
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi
glance_store.exceptions.BackendException: Failed to add object to Swift.
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi Got error from Swift:
put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...)
failure and no ability to reset contents for reupload..
                                                                      2021-03-17
08:35:16.425 85 ERROR glance.common.wsgi
Mar 17 08:35:16 infra1-glance-container-99614ac2 uwsgi[85]: Wed Mar 17 08:35:16
2021 - uwsgi_response_writev_headers_and_body_do(): Connection reset by peer
[core/writer.c line 306] during PUT
/v2/images/1b9af05a-f7c9-4315-9354-e09f1df66321/file (192.168.110.215)
Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
08:35:16.441 85 CRITICAL glance [req-32345bbf-a88f-450e-90b2-69c6a5804a7a
956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
default] Unhandled error: OSError: write error
                                                                      2021-03-17
08:35:16.441 85 ERROR glance OSError: write error
                                                                      2021-03-17
08:35:16.441 85 ERROR glance

 
It seems as if the problems occur more often with large instances, i.e. there
are fewer problems with new  Ubuntu 20.04 instances but after an 'apt-upgrade'
on the instance, the error occurs every time a snapshot is taken. Any help is
much appreciated!
 
Kind regards,
Oliver



More information about the openstack-discuss mailing list