[glance][openstack-ansible] Snapshots disappear during saving
Dmitriy Rabotyagov
noonedeadpunk at ya.ru
Thu Mar 18 10:44:47 UTC 2021
Hi Olver,
Am I right that you're also using OpenStack Swift and it's intentional to store images there?
Since the issue is related to the upload process into the Swift. So also checking Swift logs
be usefull as well.
17.03.2021, 13:42, "Oliver Wenz" <oliver.wenz at dhbw-mannheim.de>:
> Hi!
> We are currently experiencing problems with our OpenStack Ansible Victoria cloud
> when trying to create snapshots from instances. In some cases, everything works
> but often the pending snapshots just disappear.
>
> When this happens, the following glance-api.service errors show up:
>
> Mar 17 08:33:12 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
> 08:33:11.975 85 INFO glance.api.v2.image_data
> [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52
> ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to create trust: no
> such option collect_timing in group [keystone_authtoken] Use the existing user
> token.
> Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
> 08:35:15.283 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a
> 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
> default] REQ: curl -i
> http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/1b9af05a-f7c9-4315-9354-e09f1df66321-00001
> -X PUT -H "X-Auth-Token: gAAAAABgUb7IBvZ_..."
> Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
> 08:35:15.285 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a
> 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
> default] RESP STATUS: 504 Gateway Time-out
> Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
> 08:35:15.285 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a
> 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
> default] RESP HEADERS: {'content-length': '92', 'cache-control': 'no-cache',
> 'content-type': 'text/html', 'connection': 'close'}
> Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
> 08:35:15.286 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a
> 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
> default] RESP BODY: b"<html><body><h1>504 Gateway Time-out</h1>\nThe server
> didn't respond in time.\n</body></html>\n"
> Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
> 08:35:16.306 85 ERROR glance_store._drivers.swift.store
> [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52
> ebe0fe5f3893495e82598c07716f5d45 - default default] Error during chunked upload
> to backend, deleting stale chunks.: swiftclient.exceptions.ClientException:
> put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...)
> failure and no ability to reset contents for reupload.
> Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
> 08:35:16.320 85 ERROR glance_store._drivers.swift.store
> [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52
> ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to add object to
> Swift.
> Got error
> from Swift: put_object('glance_images',
> '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to
> reset contents for reupload..: swiftclient.exceptions.ClientException:
> put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...)
> failure and no ability to reset contents for reupload.
> Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
> 08:35:16.327 85 ERROR glance.api.v2.image_data
> [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52
> ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to upload image data
> due to internal error: glance_store.exceptions.BackendException: Failed to add
> object to Swift.
> Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi
> [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52
> ebe0fe5f3893495e82598c07716f5d45 - default default] Caught error: Failed to add
> object to Swift.
> Got error
> from Swift: put_object('glance_images',
> '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to
> reset contents for reupload..: glance_store.exceptions.BackendException: Failed
> to add object to Swift.
> Got error
> from Swift: put_object('glance_images',
> '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to
> reset contents for reupload..
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi Traceback (most recent call last):
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py",
> line 1014, in add
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi self._delete_stale_chunks(
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
> line 220, in __exit__
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise()
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
> line 196, in force_reraise
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value,
> self.tb)
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703,
> in reraise
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi raise value
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py",
> line 1003, in add
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi
> manager.get_connection().put_object(
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py",
> line 1960, in put_object
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi return self._retry(reset_func,
> put_object, container, obj, contents,
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py",
> line 1843, in _retry
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi reset_func(func, *args, **kwargs)
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py",
> line 1940, in _default_reset
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi raise
> ClientException('put_object(%r, %r, ...) failure and no '
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi swiftclient.exceptions.ClientException:
> put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...)
> failure and no ability to reset contents for reupload.
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi During handling of the above exception,
> another exception occurred:
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi Traceback (most recent call last):
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py",
> line 1347, in __call__
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi action_result =
> self.dispatch(self.controller, action,
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py",
> line 1391, in dispatch
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi return method(*args, **kwargs)
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/utils.py",
> line 416, in wrapped
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi return func(self, req, *args,
> **kwargs)
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py",
> line 298, in upload
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi self._restore(image_repo, image)
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
> line 220, in __exit__
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise()
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
> line 196, in force_reraise
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value,
> self.tb)
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703,
> in reraise
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi raise value
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py",
> line 163, in upload
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi image.set_data(data, size,
> backend=backend)
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/domain/proxy.py",
> line 208, in set_data
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi self.base.set_data(data, size,
> backend=backend, set_active=set_active)
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py",
> line 501, in set_data
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi _send_notification(notify_error,
> 'image.upload', msg)
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
> line 220, in __exit__
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise()
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py",
> line 196, in force_reraise
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value,
> self.tb)
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703,
> in reraise
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi raise value
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py",
> line 447, in set_data
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi self.repo.set_data(data, size,
> backend=backend,
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/policy.py",
> line 198, in set_data
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi return self.image.set_data(*args,
> **kwargs)
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/quota/__init__.py",
> line 318, in set_data
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi self.image.set_data(data,
> size=size, backend=backend,
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py",
> line 567, in set_data
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi self._upload_to_store(data,
> verifier, backend, size)
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py",
> line 458, in _upload_to_store
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi multihash, loc_meta) =
> self.store_api.add_with_multihash(
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py",
> line 398, in add_with_multihash
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi return
> store_add_to_backend_with_multihash(
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py",
> line 480, in store_add_to_backend_with_multihash
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi (location, size, checksum,
> multihash, metadata) = store.add(
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/driver.py",
> line 279, in add_adapter
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi metadata_dict) =
> store_add_fun(*args, **kwargs)
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/capabilities.py",
> line 176, in op_checker
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi return store_op_fun(store, *args,
> **kwargs)
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi File
> "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py",
> line 1082, in add
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi raise
> glance_store.BackendException(msg)
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi
> glance_store.exceptions.BackendException: Failed to add object to Swift.
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi Got error from Swift:
> put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...)
> failure and no ability to reset contents for reupload..
> 2021-03-17
> 08:35:16.425 85 ERROR glance.common.wsgi
> Mar 17 08:35:16 infra1-glance-container-99614ac2 uwsgi[85]: Wed Mar 17 08:35:16
> 2021 - uwsgi_response_writev_headers_and_body_do(): Connection reset by peer
> [core/writer.c line 306] during PUT
> /v2/images/1b9af05a-f7c9-4315-9354-e09f1df66321/file (192.168.110.215)
> Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17
> 08:35:16.441 85 CRITICAL glance [req-32345bbf-a88f-450e-90b2-69c6a5804a7a
> 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default
> default] Unhandled error: OSError: write error
> 2021-03-17
> 08:35:16.441 85 ERROR glance OSError: write error
> 2021-03-17
> 08:35:16.441 85 ERROR glance
>
> It seems as if the problems occur more often with large instances, i.e. there
> are fewer problems with new Ubuntu 20.04 instances but after an 'apt-upgrade'
> on the instance, the error occurs every time a snapshot is taken. Any help is
> much appreciated!
>
> Kind regards,
> Oliver
--
Kind Regards,
Dmitriy Rabotyagov
More information about the openstack-discuss
mailing list