March 2021 - openstack-discuss - lists.openstack.org

Re: [glance][openstack-ansible] Snapshots disappear during saving
by Oliver Wenz 22 Mar '21

22 Mar '21

Hi Dmitriy, thanks for your answer! Yes, we do use swift and its use as glance backend is intentional. I got the following from the swift-proxy-server logs in the swift container on the infra host after taking a snapshot: Mar 22 08:43:43 infra1-swift-proxy-container-27169fa7 proxy-server[87]: Client disconnected without sending last chunk (txn: txa7c64547baf0450eb0034-006058588b) (client_ip: 192.168.110. 106) Mar 22 08:43:43 infra1-swift-proxy-container-27169fa7 proxy-server[87]: 192.168.110.106 192.168.110.211 22/Mar/2021/08/43/43 PUT /v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/3ec63ec2-aa3b-4c3b-a904-b55d1a6ec878 -00001 HTTP/1.0 499 - python-swiftclient-3.10.1 gAAAAABgWFiKMz9R... 204800000 89 - txa7c64547baf0450eb0034-006058588b - 52.2991 - - 1616402571.623875856 1616402623.922997952 0 On the swift host some services logs contain errors. E.g. the swift-container-updater service: Mar 22 08:42:19 bc1bl12 systemd[1]: swift-container-updater.service: Main process exited, code=exited, status=1/FAILURE Mar 22 08:42:19 bc1bl12 systemd[1]: swift-container-updater.service: Failed with result 'exit-code'. Mar 22 08:42:21 bc1bl12 systemd[1]: swift-container-updater.service: Scheduled restart job, restart counter is at 162982. Mar 22 08:42:21 bc1bl12 systemd[1]: Stopped swift-container-updater service. Mar 22 08:42:21 bc1bl12 systemd[1]: Started swift-container-updater service. Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: Traceback (most recent call last): Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: File "/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/swift/common/utils.py", line 803, in config_fallocate_value Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: reserve_value = float(reserve_value[:-1]) Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: ValueError: could not convert string to float: '1%' Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: During handling of the above exception, another exception occurred: Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: Traceback (most recent call last): Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: File "/openstack/venvs/swift-22.1.0/bin/swift-container-updater", line 23, in <module> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: run_daemon(ContainerUpdater, conf_file, **options) Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: File "/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/swift/common/daemon.py", line 304, in run_daemon Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: utils.config_fallocate_value(conf.get('fallocate_reserve', '1%')) Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: File "/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/swift/common/utils.py", line 809, in config_fallocate_value Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: raise ValueError('Error: %s is an invalid value for fallocate' Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: ValueError: Error: 1%% is an invalid value for fallocate_reserve. The same exit-code and traceback shows in the logs of swift-container-auditor, swift-account-auditor and swift-account-reaper services. Does this tell you anything useful? We didn't experience any problems when uploading files to containers, only when taking snapshots of instances. Kind regards, Oliver > ------------------------------ > > Message: 4 > Date: Thu, 18 Mar 2021 12:44:47 +0200 > From: Dmitriy Rabotyagov <noonedeadpunk(a)ya.ru> > To: "openstack-discuss(a)lists.openstack.org" > <openstack-discuss(a)lists.openstack.org> > Subject: Re: [glance][openstack-ansible] Snapshots disappear during > saving > Message-ID: <374941616064157(a)mail.yandex.ru> > Content-Type: text/plain; charset=utf-8 > > Hi Olver, > > Am I right that you're also using OpenStack Swift and it's intentional to > store images there? > Since the issue is related to the upload process into the Swift. So also > checking Swift logs > be usefull as well. >

2 2

[oslo] Cancel meeting today
by Herve Beraud 22 Mar '21

22 Mar '21

Hello Osloers, I won't be around to run the meeting today. If someone else from the Oslo team wants to run it, please feel free. Thanks for your understanding -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

1 0

[release] Xena PTG
by Herve Beraud 22 Mar '21

22 Mar '21

Oï releasers, The PTG is fast approaching (Apr 19 - 23). To help to organize the gathering: 1) please fill the doodle[1] with the time slots that fit well for you; 2) please add your PTG topics in our etherpad[2]. Voting will be closed on the 21st of March. Thanks for your reading, [1] https://doodle.com/poll/8d8n2picqnhchhsv [2] https://etherpad.opendev.org/p/xena-ptg-os-relmgt -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

1 2

Authenticating to devstack and listing nova flavors + servers
by Joe Wolfe 22 Mar '21

22 Mar '21

Hi All, I am working on an api and would like to get some more context behinds authenticating into the api - so I am trying to get a list out of available flavors 1 and servers to start off with. I am able to get a token that I can interact with against keystone which shows up as https://mydevstackurl/identity/ <https://mydevstackurl/identity/> however when I try and run a simple curl against my compute as follows: curl -H "X-Auth-Token: $OS_TOKEN" -s https://http://mydevstackurl/compute/api/v2.1/servers <https://http//mydevstackurl/compute/api/v2.1/servers> I got no response back from curl - no 200 ,no 30x no 4xx nothing. Can anyone help to get me started ? The documents seem to be pretty detailed - but the methods provided don’t seem to work either in the tutorial: for example: curl -s -H "X-Auth-Token: $OS_TOKEN" \ $OS_COMPUTE_API/flavors \ | python -m json.tool Only returns: zsh: no matches found: http://mydevstackurl/identity/v3/auth/tokens?nocatalog <http://mydevstackurl/identity/v3/auth/tokens?nocatalog> Expecting value: line 1 column 1 (char 0) Thanks

2 1

Re: [openstack-community] Authenticating to devstack and listing nova flavors + servers
by Jeremy Stanley 22 Mar '21

22 Mar '21

On 2021-03-21 16:39:42 -0400 (-0400), Joe Wolfe wrote: > I am working on an api and would like to get some more context > behinds authenticating into the api - so I am trying to get a list > out of available flavors 1 and servers to start off with. I am > able to get a token that I can interact with against keystone > which shows up as https://mydevstackurl/identity/ however when I > try and run a simple curl against my compute as follows: > > curl -H "X-Auth-Token: $OS_TOKEN" -s https://http://mydevstackurl/compute/api/v2.1/servers > > I got no response back from curl - no 200 ,no 30x no 4xx nothing. > > Can anyone help to get me started ? The documents seem to be > pretty detailed - but the methods provided don’t seem to work > either in the tutorial: for example: > > curl -s -H "X-Auth-Token: $OS_TOKEN" \ > $OS_COMPUTE_API/flavors \ > | python -m json.tool > > Only returns: > > zsh: no matches found: http://mydevstackurl/identity/v3/auth/tokens?nocatalog > Expecting value: line 1 column 1 (char 0) As the description of the community mailing list states, "This list does not provide support for OpenStack software." I have Cc'd the openstack-discuss mailing list which is intended for "Discussion of OpenStack use and development." Please follow up there with your question. I'm keeping you in the To header in case you're not subscribed there, so that people who reply to this will hopefully include you directly. -- Jeremy Stanley

1 0

Fw: Aw: Re: OpenStack Victoria - Kolla-Ansible - Cinder - iSCSI-backend - Update driver status failed: config is uninitialized.
by Stefan Kelber 22 Mar '21

22 Mar '21

1 1

OpenStack Victoria - Kolla-Ansible - Cinder - iSCSI-backend - Update driver status failed: config is uninitialized.
by Stefan Kelber 21 Mar '21

21 Mar '21

3 2

iPXE update/configuration for the undercloud.
by Kendrick 21 Mar '21

21 Mar '21

I have a melanox connectx3 and a few broadcom systems that do not properly pxe boot, using maas the broadcoms work fine. I was looking around and one of the solutions to similar problems from 2017 and 19 were to use a different version of the ipxe files. I was hoping to know what settings/changes/scripts openstack undercloud used so i can make sure the connectx3 support is built in to the ipxe file and what files needed to be copied where from the source buld to the containers since the containers have about 20 sets of the ipxe files from what i can see when i use find to locate them from the host. Regards Kendrick

1 0

[all][tc] Dropping lower-constraints testing from all projects
by Ghanshyam Mann 20 Mar '21

20 Mar '21

Hello Everyone, You might have seen the discussion around dropping the lower constraints testing as it becomes more challenging than the current value of doing it. Few of the ML thread around this discussion: - http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019521… - http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019390… As Oslo and many other project dropping or already dropped it, we should decide it for all other projects also otherwise it can be more changing than it is currently. We have not defined it in PTI or testing runtime so it is always up to projects if they still want to keep it but we should decide a general recommendation here. -gmann

12 46

[tripleo][ci] ovb featureset01/35 and other jobs are down
by Wesley Hayutin 20 Mar '21

20 Mar '21

Greetings, ATM, the RDO team is working w/ Vexxhost to diagnose an outage in vexxhost that has caused various problems w/ heat stacks and now overcloud node registration and introspection. The issue is tracked w/ vexx and https://bugs.launchpad.net/tripleo/+bug/1920101. The third party check jobs are not expected to pass at this time. We'll follow up when we have more information. Thank you

1 0

[tripleo] Nominate David J. Peacock (dpeacock) for Validation Framework Core
by Gaël Chamoulaud 20 Mar '21

20 Mar '21

Hi TripleO Devs, David is already a key member of our team since a long time now, he provided all the needed ansible roles for the Validation Framework into tripleo-ansible-operator. He continuously provides excellent code reviews and he is a source of great ideas for the future of the Validation Framework. That's why we would highly benefit from his addition to the core reviewer team. Assuming that there are no objections, we will add David to the core team next week. Thanks, David, for your excellent work! -- Gaël Chamoulaud - (He/Him/His) .::. Red Hat .::. OpenStack .::. .::. DFG:DF Squad:VF .::.

9 12

[Cinder] Review pretty please
by Sam Morrison 19 Mar '21

19 Mar '21

Hi Cinder cores, I've had a simple review [1] waiting for several months, could I please get some feedback on this. Thanks! Sam [1] https://review.opendev.org/c/openstack/cinder/+/764875

3 2

[ops] Bandwidth problem on computes
by Jahson Babel 19 Mar '21

19 Mar '21

Hello everyone, I have a bandwidth problem between the computes nodes of an openstack cluster. This cluster runs on Rocky version with OpenVSwitch. To simplify I'll just pick 3 servers, one controller and two computes nodes all connected to the same switch. Every server is configured with two 10G links. Those links are configured in LACP /teaming. From what I understand of teaming and this configuration I should be able to get 10Gbps between all three nodes. But if I iperf we are way below this : compute1 # sudo iperf3 -c compute2 -p 5201 Connecting to host compute2, port 5201 [ 4] local X.X.X.X port 44946 connected to X.X.X.X port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 342 MBytes 2.87 Gbits/sec 137 683 KBytes [ 4] 1.00-2.00 sec 335 MBytes 2.81 Gbits/sec 8 501 KBytes Plus the problem seems to be only present with incoming traffic. Which mean I can almost get the full 10gbps if I iperf from a compute to the controller. compute1 # sudo iperf3 -c controller -p 5201 Connecting to host controller, port 5201 [ 4] local X.X.X.X port 39008 connected to X.X.X.X port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.10 GBytes 9.41 Gbits/sec 0 691 KBytes [ 4] 1.00-2.00 sec 1.09 GBytes 9.38 Gbits/sec 0 803 KBytes If I do the opposite I get the same results I was getting between the 2 computes. From the tests we've done it seems related to the openstack's services, specifically neutron or OpenVSwitch. From the time those services are running we can't get the full bandwidth. Stopping the services won't fix the issue, in our case removing the packages and rebooting is the only way to obtain the full bandwidth between computes. I voluntarily didn't mention VMs to simplify the question but of course this behavior can also be observed in VMs Knowing that we can achieve 10Gbps it doesn't seems related to the hardware nor the OS. That why we suspect OpenStack's services. But I couldn't find any evidence or misconfiguration that could confirm that. So if anyone got some hints about that kind of setup and/or how mitigate bandwidth decrease I would appreciate. Let know if you need more info. Thanks in advance, Jahson

2 2

Re: [nova] Train: how to use newer API microversion
by Eugen Block 19 Mar '21

19 Mar '21

Alright, I made it work it seems. After I added the new endpoints (v2.1) and disabled the old ones (v2) I restarted nova-api service. This allowed me to use microversion 2.37 to use the option "--nic none", so that works as expected now. I just don't know yet if there are any clients that require v2 so for now I let them in "disabled" state. Thanks, Eugen Zitat von Eugen Block <eblock(a)nde.ag>: >> If you (or your end users) rely on the legacy APIs, you can have both >> like this (192.0.2.40 is the example IP address of the API host; the >> /compute could be :some_port if you prefer that): > > I started with disabling the old endpoints but then I couldn't use > compute at all to create a new instance. There must be something > still relying on the old endpoint but I can't find it right now. > I'll give it another shot tomorrow. Thanks for your input. > > > Zitat von Radosław Piliszek <radoslaw.piliszek(a)gmail.com>: > >> On Thu, Mar 18, 2021 at 3:50 PM Eugen Block <eblock(a)nde.ag> wrote: >>> I also found that already, but it doesn't describe how I can move from >>> v2 to v2.1 in order to use only the newer API. Do you have any >>> information about that? >> >> If you (or your end users) rely on the legacy APIs, you can have both >> like this (192.0.2.40 is the example IP address of the API host; the >> /compute could be :some_port if you prefer that): >> >> +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ >> | ID | Region | Service Name | Service Type | Enabled | >> Interface | URL | >> +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ >> | xxx | RegionOne | nova | compute | True | public >> | http://192.0.2.40/compute/v2.1 | >> | xxx | RegionOne | nova_legacy | compute_legacy | True | public >> | http://192.0.2.40/compute/v2/$(project_id)s | >> +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ >> >> If you don't need the legacy APIs, you can delete their endpoints. >> >> -yoctozepto

1 0

[keystone][kolla][horizon] versioned or unversioned keystone endpoints?
by Mark Goddard 19 Mar '21

19 Mar '21

Hi, There seems to be some inconsistency around whether Keystone endpoints in the catalog should be versioned or not. Specifically, should they have a /v3 suffix? I did a survey of code using keystone-manage bootstrap via codesearch: * Kolla Ansible: unversioned * DevStack: unversioned * puppet-keystone: unversioned * Keystone install guides: versioned * OSA: unversioned I imagine that when v2 was around, the endpoint needed to be unversioned. But was there a point when it became recommended to add a version? I'm asking because of an issue in horizon when downloading a clouds.yaml file for application credentials. If the auth_url field is unversioned, then some older client libraries will fail with a Not Found error during e.g. openstack server list. I tested using Stein and Train U-C. The former fails, the latter works. If I use Stein U-C, then update keystoneauth1 to 3.17.3 (from Train U-C), it works. Thanks, Mark

1 0

[neutron] Drivers meeting agenda - 19.03.2021
by Slawek Kaplonski 19 Mar '21

19 Mar '21

Hi, First of all sorry for sending it so late - I simply forgot to send this email yesterday :) For today's drivers meeting we have 1 RFE to discuss: * https://bugs.launchpad.net/neutron/+bug/1904559 - it is continuation of the discussion from last week. Michael Johnson from Designate team provided his feedack on it so I think that based on that we can make final decision about that RFE on our today's meeting. -- Slawek Kaplonski Principal Software Engineer Red Hat

1 1

[release] Release countdown for week R-3 Mar 22 - Mar 26
by Herve Beraud 19 Mar '21

19 Mar '21

Development Focus ----------------- The Release Candidate (RC) deadline is next Thursday, 25 March, 2021. Work should be focused on fixing any release-critical bugs. General Information ------------------- All deliverables released under a cycle-with-rc model should have a first release candidate by the end of the week, from which a stable/wallaby branch will be cut. This branch will track the Wallaby release. Once stable/wallaby has been created, the master branch will be ready to switch to Xena development. While the master branch will no longer be feature-frozen, please prioritize any work necessary for completing Wallaby plans. Release-critical bugfixes will need to be merged in the master branch first, then backported to the stable/wallaby branch before a new release candidate can be proposed. Actions ------- Early in the week, the release team will be proposing RC1 patches for all cycle-with-rc projects, using the latest commit from the master branch. If your team is ready to go for cutting RC1, please let us know by leaving a +1 on these patches. If there are still a few more patches needed before RC1, you can -1 the patch and update it later in the week with the new commit hash you would like to use. Remember, stable/wallaby branches will be created with this, so you will want to make sure you have what you need included to avoid needing to backport changes from the master branch (which will technically then be Xena) to this stable branch for any additional RCs before the final release. The release team will also be proposing releases for any deliverable following a cycle-with-intermediary model that has not produced any Wallaby release so far. Finally, now is a good time to finalize release highlights. Release highlights help shape the messaging around the release and make sure that your work is properly represented. Upcoming Deadlines & Dates -------------------------- RC1 deadline: 25 March, 2021 (R-3 week) Final RC deadline: 08 April, 2021 (R-1 week) Final Wallaby release: 14 April, 2021 Xena virtual PTG: 19 - 23 April, 2021 Thanks for your reading -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

1 0

OSP13 - Queens - Inconsistent data between in DB between ipamallocation and ipallocation
by Laurent Dumont 19 Mar '21

19 Mar '21

Hey everyone, We just troubleshooted (and fixed) a weird issue with ipamallocation. It looked like ipallocations we're not properly removed from the ipamallocation table. This caused port creations to fail with an error about the IP being already allocated (when there wasn't an actual port using the IP). We found this old issue https://bugs.launchpad.net/neutron/+bug/1884532 but not a whole lot of details in there. Has anyone seen this before? Thanks!

2 2

[neutron] Unit test jobs broken
by Slawek Kaplonski 19 Mar '21

19 Mar '21

Hi, Our UT gate jobs are now broken due to new neutron-lib release. Fix for that is already proposed in [1]. So if Your UT jobs are failing like e.g. in [2] please don't recheck Your patch before [1] will be merged. [1] https://review.opendev.org/c/openstack/neutron/+/780802 [2] https://8ee33ec1a424812a1857-16390f05fde22eb723929856dcc38fcd.ssl.cf5.rackc… -- Slawek Kaplonski Principal Software Engineer Red Hat

1 1

[Neutron] [Designate] Private / Internal DNS Zones with custom records for i.e. service discovery
by Christian Rohmann 19 Mar '21

19 Mar '21

Hey Openstack-Discuss, apart from the standardized and auto-created records for ports / floating-ips and instances (https://docs.openstack.org/neutron/latest/admin/config-dns-int.html) - is there any way to allow users to add their own records which then only resolve internally? The Neutron API (https://docs.openstack.org/api-ref/network/v2/#id52) seems to be all about the resources it manages, so no additional or custom records there. Looking at the Designate API https://docs.openstack.org/api-ref/dns/?expanded=create-zone-detail#create-… is does not seem to be an option to mark a zone as "internal" or "private". But maybe there is another way to add records to the internal zone? I am thinking of an only internally resolvable / valid DNS zone carrying records for i.e. service discovery / cluster forming. There are more and more tools just looking up a DNS records to find cluster members ... * ElasticSearch: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-dis… * Hazelcast: https://github.com/hazelcast/hazelcast-kubernetes#understanding-discovery-m… * HiveMQ: https://github.com/hivemq/hivemq-dns-cluster-discovery-extension/blob/maste… * RabbitMQ: https://www.rabbitmq.com/cluster-formation.html#peer-discovery-dns [...] and with Kubernetes and the headless service concept there are more tools (ab)using DNS for this every week. So having internal dns zones which only resolve within the project would be really helpful. The hyperscalers call this feature * AWS "Private hosted zones" (https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-priv…) * Azure "Private DNS" (https://medium.com/azure-architects/exploring-azure-private-dns-be65de08f780) * GCP "Private zone" (https://cloud.google.com/blog/products/networking/introducing-private-dns-z…) * Alibaba Cloud "DNS PrivateZone" (https://www.alibabacloud.com/product/private-zone) Regards Christian

3 2

[tripleo] launchpad wallaby-3 milestone closed use wallaby-rc1
by Marios Andreou 19 Mar '21

19 Mar '21

Hi tripleo o/ as just mentioned on irc #tripleo, I closed out the wallaby-3 launchpad milestone yesterday. If you want to set a milestone in bugs please use wallaby-rc1 (wallaby-3 is no longer in the "assign milestone" list). I moved the wallaby-3 unclosed bugs to wallaby-rc1 with our helper script [1] results at [2]. Please try and keep bug status updated. Things like moving it to fix-released once we're done merging fixes - it is easy to forget about that. We have 155 bugs triaged at https://launchpad.net/tripleo/+milestone/wallaby-rc1 at least some of those can be moved to fixed/closed nofix etc. Yes I know, #cool story bro thank you for reading ;) and for your help on bug status regards, marios [1] https://opendev.org/openstack/tripleo-ci/src/branch/master/scripts/move_bug… [2] https://gist.github.com/marios/b3155fe3b1318cc26bfa4bc15c764a26#gistcomment…

1 0

Re: [nova] Train: how to use newer API microversion
by Radosław Piliszek 19 Mar '21

19 Mar '21

On Thu, Mar 18, 2021 at 3:50 PM Eugen Block <eblock(a)nde.ag> wrote: > I also found that already, but it doesn't describe how I can move from > v2 to v2.1 in order to use only the newer API. Do you have any > information about that? If you (or your end users) rely on the legacy APIs, you can have both like this (192.0.2.40 is the example IP address of the API host; the /compute could be :some_port if you prefer that): +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ | xxx | RegionOne | nova | compute | True | public | http://192.0.2.40/compute/v2.1 | | xxx | RegionOne | nova_legacy | compute_legacy | True | public | http://192.0.2.40/compute/v2/$(project_id)s | +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ If you don't need the legacy APIs, you can delete their endpoints. -yoctozepto

1 0

[ironic] [infra] Making Glean work with IPA for static IP assignment
by Dmitry Tantsur 18 Mar '21

18 Mar '21

Hi Ian, Given our timezone difference, I've decided to send you an email, adding openstack-discuss in CC for greater exposure. We're trying to make IPA (our agent ramdisk) to work without relying on DHCP for cases like Edge deployments. We've settled on the network_data.json format for the API side and wanted to use Glean on the ramdisk to apply it. You can read more details in [1]. The problem is, I cannot make Glean work with any ramdisk I build. The crux of the problem seems to be that NetworkManager (used by default in RHEL, CentOS, Fedora and Debian at least) starts very early, creates the default connection and ignores whatever files Glean happens to write afterwards. On Debian running `systemctl restart networking` actually helped to pick the new configuration, but I'm not sure we want to do that in Glean. I haven't been able to make NetworkManager pick up the changes on RH systems so far. I build ramdisks using IPA-builder [2] by adding the simple-init element. I've tried removing dhcp-all-interfaces (which we depend on by default) to no effect. I've tried disabling the DHCP server, ended up with no IP connectivity at all. I haven't tried to shutdown and restart a connection as recommended in [3] since it's not trivial to do via SSH. Do you maybe have any hints how to proceed? I'd be curious to know how static IP assignment works in the infra setup. Do you have images with NetworkManager there? Do you use the simple-init element? Any help is very appreciated. Dmitry [1] https://specs.openstack.org/openstack/ironic-specs/specs/not-implemented/L3… [2] https://opendev.org/openstack/ironic-python-agent-builder [3] https://mail.gnome.org/archives/networkmanager-list/2014-January/msg00032.h… -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

4 9

[nova] Train: how to use newer API microversion
by Eugen Block 18 Mar '21

18 Mar '21

Hi *, I'm a little confused and could use some guidance regarding endpoints and microversions. Just recently I did a lot of maintenance in our old cloud, I believe it was created with Kilo and is now Train. Anyway, I wanted to use the "server create" option "--nic none" but couldn't because of the v2 endpoint for the nova service, right? According to the cli help this option is available from microversion 2.37+. This was the first attempt before adding new endpoints: controller:~ # openstack server create --image <UUID> --nic none --flavor 3 test-nic-none nics must be a list or a tuple, not <class 'str'> These were the endpoints before I added new ones: controller:~ # openstack endpoint list | grep nova | <UUID> | RegionOne | nova | compute | True | public | http://controller:<PORT>/v2/%(tenant_id)s | | <UUID> | RegionOne | nova | compute | True | internal | http://controller:<PORT>/v2/%(tenant_id)s | | <UUID> | RegionOne | nova | compute | True | admin | http://controller:<PORT>/v2/%(tenant_id)s | I checked with the Train install guide and created new endpoints for nova: controller:~ # openstack endpoint create --region RegionOne compute public http://controller:<PORT>/v2.1 controller:~ # openstack endpoint create --region RegionOne compute internal http://controller:<PORT>/v2.1 controller:~ # openstack endpoint create --region RegionOne compute admin http://controller:<PORT>/v2.1 I should mention that I haven't restarted any services yet, I wanted to check each step carefully since people are working in that cloud. In the debug output from creating a new instance I saw that it's still trying to use v2 api, so I disabled it, resulting in error messages that there's no compute service available. So I reenabled the old endpoints and now have both versions active. Then I tried this: controller:~ # openstack --os-compute-api-version 2.37 server create --image <UUID> --nic none --flavor 3 test-nic-none Invalid input for field/attribute networks. Value: none. 'none' is not of type 'array' (HTTP 400) (Request-ID: req-b3c53ab2-8040-42cf-bf07-d03e8f46cfd5) Now I'm running out of ideas how to upgrade our api endpoints so the services will actually be able to use them properly. I read Train release notes (we're planning to upgrade to Ussuri soon) and multiple docs about microversions but nothing seems to cover my issue. Could anyone shed some light? I must admit, during the upgrade cycles in the past years I just hoped everything would keep working and it did, I didn't put too much attention in the api versions yet. Any help is appreciated! Thanks and best regards, Eugen

2 2

[glance][openstack-ansible] Snapshots disappear during saving
by Oliver Wenz 18 Mar '21

18 Mar '21

Hi! We are currently experiencing problems with our OpenStack Ansible Victoria cloud when trying to create snapshots from instances. In some cases, everything works but often the pending snapshots just disappear. When this happens, the following glance-api.service errors show up: Mar 17 08:33:12 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:33:11.975 85 INFO glance.api.v2.image_data [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token. Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.283 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] REQ: curl -i http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance… -X PUT -H "X-Auth-Token: gAAAAABgUb7IBvZ_..." Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.285 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP STATUS: 504 Gateway Time-out Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.285 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP HEADERS: {'content-length': '92', 'cache-control': 'no-cache', 'content-type': 'text/html', 'connection': 'close'} Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.286 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP BODY: b"<html><body><h1>504 Gateway Time-out</h1>\nThe server didn't respond in time.\n</body></html>\n" Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.306 85 ERROR glance_store._drivers.swift.store [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Error during chunked upload to backend, deleting stale chunks.: swiftclient.exceptions.ClientException: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload. Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.320 85 ERROR glance_store._drivers.swift.store [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to add object to Swift. Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload..: swiftclient.exceptions.ClientException: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload. Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.327 85 ERROR glance.api.v2.image_data [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to upload image data due to internal error: glance_store.exceptions.BackendException: Failed to add object to Swift. Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Caught error: Failed to add object to Swift. Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload..: glance_store.exceptions.BackendException: Failed to add object to Swift. Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload.. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1014, in add 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self._delete_stale_chunks( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise() 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise value 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1003, in add 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi manager.get_connection().put_object( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1960, in put_object 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return self._retry(reset_func, put_object, container, obj, contents, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1843, in _retry 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi reset_func(func, *args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1940, in _default_reset 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise ClientException('put_object(%r, %r, ...) failure and no ' 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi swiftclient.exceptions.ClientException: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi During handling of the above exception, another exception occurred: 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1347, in __call__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1391, in dispatch 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return method(*args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/utils.py", line 416, in wrapped 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 298, in upload 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self._restore(image_repo, image) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise() 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise value 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 163, in upload 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/domain/proxy.py", line 208, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 501, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise() 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise value 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 447, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/policy.py", line 198, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/quota/__init__.py", line 318, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 567, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 458, in _upload_to_store 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 398, in add_with_multihash 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return store_add_to_backend_with_multihash( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 480, in store_add_to_backend_with_multihash 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/driver.py", line 279, in add_adapter 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/capabilities.py", line 176, in op_checker 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1082, in add 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise glance_store.BackendException(msg) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi glance_store.exceptions.BackendException: Failed to add object to Swift. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload.. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Mar 17 08:35:16 infra1-glance-container-99614ac2 uwsgi[85]: Wed Mar 17 08:35:16 2021 - uwsgi_response_writev_headers_and_body_do(): Connection reset by peer [core/writer.c line 306] during PUT /v2/images/1b9af05a-f7c9-4315-9354-e09f1df66321/file (192.168.110.215) Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.441 85 CRITICAL glance [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unhandled error: OSError: write error 2021-03-17 08:35:16.441 85 ERROR glance OSError: write error 2021-03-17 08:35:16.441 85 ERROR glance It seems as if the problems occur more often with large instances, i.e. there are fewer problems with new Ubuntu 20.04 instances but after an 'apt-upgrade' on the instance, the error occurs every time a snapshot is taken. Any help is much appreciated! Kind regards, Oliver

2 1