March 2021 - openstack-discuss - lists.openstack.org

[neutron] Drivers meeting agenda - 19.03.2021
by Slawek Kaplonski 19 Mar '21

19 Mar '21

Hi, First of all sorry for sending it so late - I simply forgot to send this email yesterday :) For today's drivers meeting we have 1 RFE to discuss: * https://bugs.launchpad.net/neutron/+bug/1904559 - it is continuation of the discussion from last week. Michael Johnson from Designate team provided his feedack on it so I think that based on that we can make final decision about that RFE on our today's meeting. -- Slawek Kaplonski Principal Software Engineer Red Hat

1 1

[release] Release countdown for week R-3 Mar 22 - Mar 26
by Herve Beraud 19 Mar '21

19 Mar '21

Development Focus ----------------- The Release Candidate (RC) deadline is next Thursday, 25 March, 2021. Work should be focused on fixing any release-critical bugs. General Information ------------------- All deliverables released under a cycle-with-rc model should have a first release candidate by the end of the week, from which a stable/wallaby branch will be cut. This branch will track the Wallaby release. Once stable/wallaby has been created, the master branch will be ready to switch to Xena development. While the master branch will no longer be feature-frozen, please prioritize any work necessary for completing Wallaby plans. Release-critical bugfixes will need to be merged in the master branch first, then backported to the stable/wallaby branch before a new release candidate can be proposed. Actions ------- Early in the week, the release team will be proposing RC1 patches for all cycle-with-rc projects, using the latest commit from the master branch. If your team is ready to go for cutting RC1, please let us know by leaving a +1 on these patches. If there are still a few more patches needed before RC1, you can -1 the patch and update it later in the week with the new commit hash you would like to use. Remember, stable/wallaby branches will be created with this, so you will want to make sure you have what you need included to avoid needing to backport changes from the master branch (which will technically then be Xena) to this stable branch for any additional RCs before the final release. The release team will also be proposing releases for any deliverable following a cycle-with-intermediary model that has not produced any Wallaby release so far. Finally, now is a good time to finalize release highlights. Release highlights help shape the messaging around the release and make sure that your work is properly represented. Upcoming Deadlines & Dates -------------------------- RC1 deadline: 25 March, 2021 (R-3 week) Final RC deadline: 08 April, 2021 (R-1 week) Final Wallaby release: 14 April, 2021 Xena virtual PTG: 19 - 23 April, 2021 Thanks for your reading -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

1 0

OSP13 - Queens - Inconsistent data between in DB between ipamallocation and ipallocation
by Laurent Dumont 19 Mar '21

19 Mar '21

Hey everyone, We just troubleshooted (and fixed) a weird issue with ipamallocation. It looked like ipallocations we're not properly removed from the ipamallocation table. This caused port creations to fail with an error about the IP being already allocated (when there wasn't an actual port using the IP). We found this old issue https://bugs.launchpad.net/neutron/+bug/1884532 but not a whole lot of details in there. Has anyone seen this before? Thanks!

2 2

[neutron] Unit test jobs broken
by Slawek Kaplonski 18 Mar '21

18 Mar '21

Hi, Our UT gate jobs are now broken due to new neutron-lib release. Fix for that is already proposed in [1]. So if Your UT jobs are failing like e.g. in [2] please don't recheck Your patch before [1] will be merged. [1] https://review.opendev.org/c/openstack/neutron/+/780802 [2] https://8ee33ec1a424812a1857-16390f05fde22eb723929856dcc38fcd.ssl.cf5.rackc… -- Slawek Kaplonski Principal Software Engineer Red Hat

1 1

[Neutron] [Designate] Private / Internal DNS Zones with custom records for i.e. service discovery
by Christian Rohmann 18 Mar '21

18 Mar '21

Hey Openstack-Discuss, apart from the standardized and auto-created records for ports / floating-ips and instances (https://docs.openstack.org/neutron/latest/admin/config-dns-int.html) - is there any way to allow users to add their own records which then only resolve internally? The Neutron API (https://docs.openstack.org/api-ref/network/v2/#id52) seems to be all about the resources it manages, so no additional or custom records there. Looking at the Designate API https://docs.openstack.org/api-ref/dns/?expanded=create-zone-detail#create-… is does not seem to be an option to mark a zone as "internal" or "private". But maybe there is another way to add records to the internal zone? I am thinking of an only internally resolvable / valid DNS zone carrying records for i.e. service discovery / cluster forming. There are more and more tools just looking up a DNS records to find cluster members ... * ElasticSearch: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-dis… * Hazelcast: https://github.com/hazelcast/hazelcast-kubernetes#understanding-discovery-m… * HiveMQ: https://github.com/hivemq/hivemq-dns-cluster-discovery-extension/blob/maste… * RabbitMQ: https://www.rabbitmq.com/cluster-formation.html#peer-discovery-dns [...] and with Kubernetes and the headless service concept there are more tools (ab)using DNS for this every week. So having internal dns zones which only resolve within the project would be really helpful. The hyperscalers call this feature * AWS "Private hosted zones" (https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-priv…) * Azure "Private DNS" (https://medium.com/azure-architects/exploring-azure-private-dns-be65de08f780) * GCP "Private zone" (https://cloud.google.com/blog/products/networking/introducing-private-dns-z…) * Alibaba Cloud "DNS PrivateZone" (https://www.alibabacloud.com/product/private-zone) Regards Christian

3 2

[tripleo] launchpad wallaby-3 milestone closed use wallaby-rc1
by Marios Andreou 18 Mar '21

18 Mar '21

Hi tripleo o/ as just mentioned on irc #tripleo, I closed out the wallaby-3 launchpad milestone yesterday. If you want to set a milestone in bugs please use wallaby-rc1 (wallaby-3 is no longer in the "assign milestone" list). I moved the wallaby-3 unclosed bugs to wallaby-rc1 with our helper script [1] results at [2]. Please try and keep bug status updated. Things like moving it to fix-released once we're done merging fixes - it is easy to forget about that. We have 155 bugs triaged at https://launchpad.net/tripleo/+milestone/wallaby-rc1 at least some of those can be moved to fixed/closed nofix etc. Yes I know, #cool story bro thank you for reading ;) and for your help on bug status regards, marios [1] https://opendev.org/openstack/tripleo-ci/src/branch/master/scripts/move_bug… [2] https://gist.github.com/marios/b3155fe3b1318cc26bfa4bc15c764a26#gistcomment…

1 0

Re: [nova] Train: how to use newer API microversion
by Radosław Piliszek 18 Mar '21

18 Mar '21

On Thu, Mar 18, 2021 at 3:50 PM Eugen Block <eblock(a)nde.ag> wrote: > I also found that already, but it doesn't describe how I can move from > v2 to v2.1 in order to use only the newer API. Do you have any > information about that? If you (or your end users) rely on the legacy APIs, you can have both like this (192.0.2.40 is the example IP address of the API host; the /compute could be :some_port if you prefer that): +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ | xxx | RegionOne | nova | compute | True | public | http://192.0.2.40/compute/v2.1 | | xxx | RegionOne | nova_legacy | compute_legacy | True | public | http://192.0.2.40/compute/v2/$(project_id)s | +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ If you don't need the legacy APIs, you can delete their endpoints. -yoctozepto

1 0

[ironic] [infra] Making Glean work with IPA for static IP assignment
by Dmitry Tantsur 18 Mar '21

18 Mar '21

Hi Ian, Given our timezone difference, I've decided to send you an email, adding openstack-discuss in CC for greater exposure. We're trying to make IPA (our agent ramdisk) to work without relying on DHCP for cases like Edge deployments. We've settled on the network_data.json format for the API side and wanted to use Glean on the ramdisk to apply it. You can read more details in [1]. The problem is, I cannot make Glean work with any ramdisk I build. The crux of the problem seems to be that NetworkManager (used by default in RHEL, CentOS, Fedora and Debian at least) starts very early, creates the default connection and ignores whatever files Glean happens to write afterwards. On Debian running `systemctl restart networking` actually helped to pick the new configuration, but I'm not sure we want to do that in Glean. I haven't been able to make NetworkManager pick up the changes on RH systems so far. I build ramdisks using IPA-builder [2] by adding the simple-init element. I've tried removing dhcp-all-interfaces (which we depend on by default) to no effect. I've tried disabling the DHCP server, ended up with no IP connectivity at all. I haven't tried to shutdown and restart a connection as recommended in [3] since it's not trivial to do via SSH. Do you maybe have any hints how to proceed? I'd be curious to know how static IP assignment works in the infra setup. Do you have images with NetworkManager there? Do you use the simple-init element? Any help is very appreciated. Dmitry [1] https://specs.openstack.org/openstack/ironic-specs/specs/not-implemented/L3… [2] https://opendev.org/openstack/ironic-python-agent-builder [3] https://mail.gnome.org/archives/networkmanager-list/2014-January/msg00032.h… -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

4 9

[nova] Train: how to use newer API microversion
by Eugen Block 18 Mar '21

18 Mar '21

Hi *, I'm a little confused and could use some guidance regarding endpoints and microversions. Just recently I did a lot of maintenance in our old cloud, I believe it was created with Kilo and is now Train. Anyway, I wanted to use the "server create" option "--nic none" but couldn't because of the v2 endpoint for the nova service, right? According to the cli help this option is available from microversion 2.37+. This was the first attempt before adding new endpoints: controller:~ # openstack server create --image <UUID> --nic none --flavor 3 test-nic-none nics must be a list or a tuple, not <class 'str'> These were the endpoints before I added new ones: controller:~ # openstack endpoint list | grep nova | <UUID> | RegionOne | nova | compute | True | public | http://controller:<PORT>/v2/%(tenant_id)s | | <UUID> | RegionOne | nova | compute | True | internal | http://controller:<PORT>/v2/%(tenant_id)s | | <UUID> | RegionOne | nova | compute | True | admin | http://controller:<PORT>/v2/%(tenant_id)s | I checked with the Train install guide and created new endpoints for nova: controller:~ # openstack endpoint create --region RegionOne compute public http://controller:<PORT>/v2.1 controller:~ # openstack endpoint create --region RegionOne compute internal http://controller:<PORT>/v2.1 controller:~ # openstack endpoint create --region RegionOne compute admin http://controller:<PORT>/v2.1 I should mention that I haven't restarted any services yet, I wanted to check each step carefully since people are working in that cloud. In the debug output from creating a new instance I saw that it's still trying to use v2 api, so I disabled it, resulting in error messages that there's no compute service available. So I reenabled the old endpoints and now have both versions active. Then I tried this: controller:~ # openstack --os-compute-api-version 2.37 server create --image <UUID> --nic none --flavor 3 test-nic-none Invalid input for field/attribute networks. Value: none. 'none' is not of type 'array' (HTTP 400) (Request-ID: req-b3c53ab2-8040-42cf-bf07-d03e8f46cfd5) Now I'm running out of ideas how to upgrade our api endpoints so the services will actually be able to use them properly. I read Train release notes (we're planning to upgrade to Ussuri soon) and multiple docs about microversions but nothing seems to cover my issue. Could anyone shed some light? I must admit, during the upgrade cycles in the past years I just hoped everything would keep working and it did, I didn't put too much attention in the api versions yet. Any help is appreciated! Thanks and best regards, Eugen

2 2

[glance][openstack-ansible] Snapshots disappear during saving
by Oliver Wenz 18 Mar '21

18 Mar '21

Hi! We are currently experiencing problems with our OpenStack Ansible Victoria cloud when trying to create snapshots from instances. In some cases, everything works but often the pending snapshots just disappear. When this happens, the following glance-api.service errors show up: Mar 17 08:33:12 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:33:11.975 85 INFO glance.api.v2.image_data [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token. Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.283 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] REQ: curl -i http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance… -X PUT -H "X-Auth-Token: gAAAAABgUb7IBvZ_..." Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.285 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP STATUS: 504 Gateway Time-out Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.285 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP HEADERS: {'content-length': '92', 'cache-control': 'no-cache', 'content-type': 'text/html', 'connection': 'close'} Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.286 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP BODY: b"<html><body><h1>504 Gateway Time-out</h1>\nThe server didn't respond in time.\n</body></html>\n" Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.306 85 ERROR glance_store._drivers.swift.store [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Error during chunked upload to backend, deleting stale chunks.: swiftclient.exceptions.ClientException: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload. Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.320 85 ERROR glance_store._drivers.swift.store [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to add object to Swift. Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload..: swiftclient.exceptions.ClientException: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload. Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.327 85 ERROR glance.api.v2.image_data [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to upload image data due to internal error: glance_store.exceptions.BackendException: Failed to add object to Swift. Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Caught error: Failed to add object to Swift. Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload..: glance_store.exceptions.BackendException: Failed to add object to Swift. Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload.. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1014, in add 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self._delete_stale_chunks( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise() 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise value 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1003, in add 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi manager.get_connection().put_object( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1960, in put_object 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return self._retry(reset_func, put_object, container, obj, contents, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1843, in _retry 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi reset_func(func, *args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1940, in _default_reset 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise ClientException('put_object(%r, %r, ...) failure and no ' 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi swiftclient.exceptions.ClientException: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi During handling of the above exception, another exception occurred: 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1347, in __call__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1391, in dispatch 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return method(*args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/utils.py", line 416, in wrapped 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 298, in upload 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self._restore(image_repo, image) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise() 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise value 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 163, in upload 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/domain/proxy.py", line 208, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 501, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise() 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise value 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 447, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/policy.py", line 198, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/quota/__init__.py", line 318, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 567, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 458, in _upload_to_store 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 398, in add_with_multihash 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return store_add_to_backend_with_multihash( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 480, in store_add_to_backend_with_multihash 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/driver.py", line 279, in add_adapter 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/capabilities.py", line 176, in op_checker 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1082, in add 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise glance_store.BackendException(msg) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi glance_store.exceptions.BackendException: Failed to add object to Swift. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload.. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Mar 17 08:35:16 infra1-glance-container-99614ac2 uwsgi[85]: Wed Mar 17 08:35:16 2021 - uwsgi_response_writev_headers_and_body_do(): Connection reset by peer [core/writer.c line 306] during PUT /v2/images/1b9af05a-f7c9-4315-9354-e09f1df66321/file (192.168.110.215) Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.441 85 CRITICAL glance [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unhandled error: OSError: write error 2021-03-17 08:35:16.441 85 ERROR glance OSError: write error 2021-03-17 08:35:16.441 85 ERROR glance It seems as if the problems occur more often with large instances, i.e. there are fewer problems with new Ubuntu 20.04 instances but after an 'apt-upgrade' on the instance, the error occurs every time a snapshot is taken. Any help is much appreciated! Kind regards, Oliver

2 1