[glance][openstack-ansible] Snapshots disappear during saving
Hi! We are currently experiencing problems with our OpenStack Ansible Victoria cloud when trying to create snapshots from instances. In some cases, everything works but often the pending snapshots just disappear. When this happens, the following glance-api.service errors show up: Mar 17 08:33:12 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:33:11.975 85 INFO glance.api.v2.image_data [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token. Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.283 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] REQ: curl -i http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_... -X PUT -H "X-Auth-Token: gAAAAABgUb7IBvZ_..." Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.285 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP STATUS: 504 Gateway Time-out Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.285 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP HEADERS: {'content-length': '92', 'cache-control': 'no-cache', 'content-type': 'text/html', 'connection': 'close'} Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.286 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP BODY: b"<html><body><h1>504 Gateway Time-out</h1>\nThe server didn't respond in time.\n</body></html>\n" Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.306 85 ERROR glance_store._drivers.swift.store [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Error during chunked upload to backend, deleting stale chunks.: swiftclient.exceptions.ClientException: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload. Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.320 85 ERROR glance_store._drivers.swift.store [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to add object to Swift. Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload..: swiftclient.exceptions.ClientException: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload. Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.327 85 ERROR glance.api.v2.image_data [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to upload image data due to internal error: glance_store.exceptions.BackendException: Failed to add object to Swift. Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Caught error: Failed to add object to Swift. Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload..: glance_store.exceptions.BackendException: Failed to add object to Swift. Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload.. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1014, in add 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self._delete_stale_chunks( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise() 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise value 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1003, in add 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi manager.get_connection().put_object( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1960, in put_object 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return self._retry(reset_func, put_object, container, obj, contents, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1843, in _retry 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi reset_func(func, *args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1940, in _default_reset 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise ClientException('put_object(%r, %r, ...) failure and no ' 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi swiftclient.exceptions.ClientException: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi During handling of the above exception, another exception occurred: 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1347, in __call__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1391, in dispatch 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return method(*args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/utils.py", line 416, in wrapped 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 298, in upload 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self._restore(image_repo, image) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise() 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise value 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 163, in upload 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/domain/proxy.py", line 208, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 501, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise() 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise value 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 447, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/policy.py", line 198, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/quota/__init__.py", line 318, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 567, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 458, in _upload_to_store 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 398, in add_with_multihash 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return store_add_to_backend_with_multihash( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 480, in store_add_to_backend_with_multihash 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/driver.py", line 279, in add_adapter 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/capabilities.py", line 176, in op_checker 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1082, in add 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise glance_store.BackendException(msg) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi glance_store.exceptions.BackendException: Failed to add object to Swift. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload.. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Mar 17 08:35:16 infra1-glance-container-99614ac2 uwsgi[85]: Wed Mar 17 08:35:16 2021 - uwsgi_response_writev_headers_and_body_do(): Connection reset by peer [core/writer.c line 306] during PUT /v2/images/1b9af05a-f7c9-4315-9354-e09f1df66321/file (192.168.110.215) Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.441 85 CRITICAL glance [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unhandled error: OSError: write error 2021-03-17 08:35:16.441 85 ERROR glance OSError: write error 2021-03-17 08:35:16.441 85 ERROR glance It seems as if the problems occur more often with large instances, i.e. there are fewer problems with new Ubuntu 20.04 instances but after an 'apt-upgrade' on the instance, the error occurs every time a snapshot is taken. Any help is much appreciated! Kind regards, Oliver
Hi Olver, Am I right that you're also using OpenStack Swift and it's intentional to store images there? Since the issue is related to the upload process into the Swift. So also checking Swift logs be usefull as well. 17.03.2021, 13:42, "Oliver Wenz" <oliver.wenz@dhbw-mannheim.de>:
Hi! We are currently experiencing problems with our OpenStack Ansible Victoria cloud when trying to create snapshots from instances. In some cases, everything works but often the pending snapshots just disappear.
When this happens, the following glance-api.service errors show up:
Mar 17 08:33:12 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:33:11.975 85 INFO glance.api.v2.image_data [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token. Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.283 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] REQ: curl -i http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_... -X PUT -H "X-Auth-Token: gAAAAABgUb7IBvZ_..." Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.285 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP STATUS: 504 Gateway Time-out Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.285 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP HEADERS: {'content-length': '92', 'cache-control': 'no-cache', 'content-type': 'text/html', 'connection': 'close'} Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.286 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP BODY: b"<html><body><h1>504 Gateway Time-out</h1>\nThe server didn't respond in time.\n</body></html>\n" Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.306 85 ERROR glance_store._drivers.swift.store [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Error during chunked upload to backend, deleting stale chunks.: swiftclient.exceptions.ClientException: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload. Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.320 85 ERROR glance_store._drivers.swift.store [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to add object to Swift. Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload..: swiftclient.exceptions.ClientException: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload. Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.327 85 ERROR glance.api.v2.image_data [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to upload image data due to internal error: glance_store.exceptions.BackendException: Failed to add object to Swift. Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Caught error: Failed to add object to Swift. Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload..: glance_store.exceptions.BackendException: Failed to add object to Swift. Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload.. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1014, in add 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self._delete_stale_chunks( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise() 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise value 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1003, in add 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi manager.get_connection().put_object( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1960, in put_object 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return self._retry(reset_func, put_object, container, obj, contents, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1843, in _retry 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi reset_func(func, *args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1940, in _default_reset 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise ClientException('put_object(%r, %r, ...) failure and no ' 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi swiftclient.exceptions.ClientException: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi During handling of the above exception, another exception occurred: 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1347, in __call__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1391, in dispatch 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return method(*args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/utils.py", line 416, in wrapped 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 298, in upload 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self._restore(image_repo, image) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise() 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise value 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 163, in upload 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/domain/proxy.py", line 208, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 501, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise() 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise value 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 447, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/policy.py", line 198, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/quota/__init__.py", line 318, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 567, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 458, in _upload_to_store 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 398, in add_with_multihash 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return store_add_to_backend_with_multihash( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 480, in store_add_to_backend_with_multihash 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/driver.py", line 279, in add_adapter 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/capabilities.py", line 176, in op_checker 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1082, in add 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise glance_store.BackendException(msg) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi glance_store.exceptions.BackendException: Failed to add object to Swift. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload.. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Mar 17 08:35:16 infra1-glance-container-99614ac2 uwsgi[85]: Wed Mar 17 08:35:16 2021 - uwsgi_response_writev_headers_and_body_do(): Connection reset by peer [core/writer.c line 306] during PUT /v2/images/1b9af05a-f7c9-4315-9354-e09f1df66321/file (192.168.110.215) Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.441 85 CRITICAL glance [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unhandled error: OSError: write error 2021-03-17 08:35:16.441 85 ERROR glance OSError: write error 2021-03-17 08:35:16.441 85 ERROR glance
It seems as if the problems occur more often with large instances, i.e. there are fewer problems with new Ubuntu 20.04 instances but after an 'apt-upgrade' on the instance, the error occurs every time a snapshot is taken. Any help is much appreciated!
Kind regards, Oliver
-- Kind Regards, Dmitriy Rabotyagov
participants (2)
-
Dmitriy Rabotyagov
-
Oliver Wenz