March 2021 - openstack-discuss - lists.openstack.org

[VOLUNTEER NEEDED] Wallaby Release Demo
by helena＠openstack.org 25 Mar '21

25 Mar '21

Hi everyone! We are closely approaching the Wallaby release (Yay!) and I would like to put together a demo video for the OpenInfra YouTube channel. The demo videos we have for Havana [1] and Juno [2] are our top-performing videos of all time. I am wondering if there is anyone in the community who would like to film a Wallaby demo for the channel. I would need the video by April 9th. In addition to the demo video, we will also have a community meeting. I will be reaching out to ptls soon on content for this. [1] [ https://www.youtube.com/watch?v=vm7aHJtQMQE ]( https://www.youtube.com/watch?v=vm7aHJtQMQE ) [2] [ https://www.youtube.com/watch?v=TgPTjrf1y0A ]( https://www.youtube.com/watch?v=TgPTjrf1y0A ) Cheers, Helena Open Infrastructure Foundation helens(a)openstack.org

1 0

[nova] Cannot use --fields in one of our headnodes
by Mauricio Tavares 25 Mar '21

25 Mar '21

Easy question of the day: One of our headnodes is barking when I use --fields with nova. nova list --all --fields=host,instance_name,status ERROR (TypeError): '<' not supported between instances of 'str' and 'NoneType' This works fine with the other headnodes, and all 3 are supposed to be the same (hardware, software). Would that indicate a bad installation or something else? Where should I be looking for answers?

2 1

[tripleo][ci] quay.io is down
by Wesley Hayutin 24 Mar '21

24 Mar '21

0/ We can't have nice things. quay.io is currently down. quay.io is utilized for non-tripleo containers like grafana, etc. This has caused jobs to fail. We'll keep you posted. Thanks!

4 5

[vpnaas-neutron][Victoria] - could not load neutron_vpnaas
by Luke Camilleri 24 Mar '21

24 Mar '21

We are configuring Neutron's vpnaas extension and although the guide seems to be quite short, it looks like that there are some dependencies that we are not aware of since we are receiving this error when we start the neutron server: WARNING stevedore.named [req-dca2e9a8-1799-45c7-9380-f247510af7e5 - - - - -] Could not load neutron_vpnaas.services.vpn.device_drivers.libreswan_ipsec.LibreSwanDriver We have followed this guide https://docs.openstack.org/neutron/latest/admin/vpnaas-scenario.html and before that we have also installed the below packages from the repos: openstack-neutron-vpnaas-17.0.0-2.el8.noarch libreswan-3.32-7.el8_3.x86_64 our config files are the below: l3_agent.ini [DEFAULT] interface_driver = linuxbridge debug = true [AGENT] extensions = vpnaas [vpnagent] vpn_device_driver = neutron_vpnaas.services.vpn.device_drivers.libreswan_ipsec.LibreSwanDriver neutron_vpnaas.conf [service_providers] service_provider = VPN:openswan:neutron_vpnaas.services.vpn.service_drivers.ipsec.IPsecVPNDriver:default vpn_agent.ini is empty since the guide documents the changes in the l3_agent.ini file in neutron.conf we have also added the below: service_plugins = router,vpnaas At this point, my main question would be, is VPNaaS supported with linux bridge or just with OpenvSwitch? Thanks in advance

1 0

[cyborg] No meeting tomorrow March 25th
by Wang, Xin-ran 24 Mar '21

24 Mar '21

Hi guys, The Cyborg IRC meeting tomorrow has been cancelled. We will meet again at the regular time next week. Please feel free to contact the team on IRC channel, or send out mail here. Thanks for your understanding. Thanks, Xin-Ran

1 0

[kolla] Feedback request: removing OracleLinux support
by Stefan Kelber 24 Mar '21

24 Mar '21

Hello, i just found this thread, in the situation to apparently have to move to ORACLE Linux, away from CentOS, as long as potential licening implications are up in the air, if i got all that right so far. My questions actually are now: - Would i have trouble today, if using ORACLE Linux for implementing Victoria via Kolla-Ansible? - And/or would it be feasible for the mid term to reactivate support for ORACLE LINUX, until the dust has settled? - On the long: will streaming change how OpenStack will be deployed via Kolla-Ansible? Unfortunately i have to implement a project these days, not allowing me to wait until then. Unfortunate timing... Best Stefan

2 1

[rabbitmq][kolla-ansible] - RabbitMQ disconnects - 60s timeout
by Adam Tomas 24 Mar '21

24 Mar '21

Hi, I have a problem with rabbitmq heartbeats timeout (described also here: https://bugzilla.redhat.com/show_bug.cgi?id=1711794 <https://bugzilla.redhat.com/show_bug.cgi?id=1711794>). Although I have threads=1 the problem still persists, generating a lot of messages in logs: 2021-03-18 15:17:17.482 [error] <0.122.51> closing AMQP connection <0.122.51> (x.x.x.100:60456 -> x.x.x.100:5672 - mod_wsgi:699:9a813fcb-c29f-4886-82bc-00bf478b6b64): missed heartbeats from client, timeout: 60s 2021-03-18 15:17:17.484 [info] <0.846.51> Closing all channels from connection '<<x.x.x.100:60456 -> x.x.x.100:5672">>' because it has been closed 2021-03-18 15:18:15.918 [error] <0.150.51> closing AMQP connection <0.150.51> (x.x.x.111:41934 -> x.x.x.100:5672 - mod_wsgi:697:b608c7b8-9644-434e-93af-c00222c0a700): missed heartbeats from client, timeout: 60s 2021-03-18 15:18:15.920 [error] <0.153.51> closing AMQP connection <0.153.51> (x.x.x.111:41936 -> x.x.x.100:5672 - mod_wsgi:697:77348197-148b-41a6-928f-c5eddfab57c9): missed heartbeats from client, timeout: 60s 2021-03-18 15:18:15.920 [info] <0.1527.51> Closing all channels from connection '<<x.x.x.111:41934 -> x.x.x.100:5672">>' because it has been closed 2021-03-18 15:18:15.922 [info] <0.1531.51> Closing all channels from connection '<<x.x.x.111:41936 -> x.x.x.100:5672">>' because it has been closed 2021-03-18 15:20:16.080 [info] <0.2196.51> accepting AMQP connection <0.2196.51> (x.x.x.111:34826 -> x.x.x.100:5672) 2021-03-18 15:20:16.080 [info] <0.2199.51> accepting AMQP connection <0.2199.51> (x.x.x.111:34828 -> x.x.x.100:5672) I’ve set heartbeat = 600 in rabbitmq.conf and still get disconnections after 60s timeout… How to set proper timeout to avoid disconnections? Best regards Adam

2 4

[cinder] Bug deputy report for week of 2021-03-15
by Sofia Enriquez 24 Mar '21

24 Mar '21

Hello, This is a bug report from 2021-03-16 to 2021-03-24. You're welcome to join the next Cinder Bug Meeting tomorrow. - Weekly on Wednesday at 1500 UTC in #openstack-cinder - Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting Critical:- High: - https://bugs.launchpad.net/oslo.db/+bug/1814199 :" soft_delete is wrong". Assigned to Gorka Eguileor. Medium: - https://bugs.launchpad.net/cinder/+bug/1298135 :" Cinder should handle token expiration for long ops". Unassigned. - https://bugs.launchpad.net/cinder/+bug/1920870 :" [Storwize] Fix issue while extending a volume with replication enabled". Assigned to Venkata krishna Thumu (venkatakt). - https://bugs.launchpad.net/cinder/+bug/1920237 :" race in backup manager remove_export call". Assigned to Eric Harney. Low:- Incomplete: - https://bugs.launchpad.net/cinder/+bug/1920912. " Add volume to a group which was created from a group snapshot of another consistency group fails and the group goes into error state". Unassigned. - https://bugs.launchpad.net/cinder/+bug/1920890: " Retype of in use Hyperswap volume failing". Unassigned. Regards, Sofi -- L. Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[neutron] vPTG meetings
by Slawek Kaplonski 24 Mar '21

24 Mar '21

Hi, Based on infomation from doodle, I just booked for us 4 slots, 3h each: * Monday 1300 - 1600 UTC * Tuesday 1300 - 1600 UTC * Thursday 1300 - 1600 UTC * Friday 1300 - 1600 UTC I think that this should be enough. If not we can try to set some ad-hoc meeting during the PTG week if that will be needed. -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[cinder][nova] Running parallel iSCSI/LVM c-vol backends is causing random failures in CI
by Lee Yarwood 24 Mar '21

24 Mar '21

Hello all, I reported the following bug last week but I've yet to get any real feedback after asking a few times in irc. Running parallel iSCSI/LVM c-vol backends is causing random failures in CI https://bugs.launchpad.net/cinder/+bug/1917750 AFAICT tgtadm is causing this behaviour. As I've stated in the bug with Fedora 32 and lioadm I don't see the WWN conflict between the two backends. Does anyone know if using lioadm is an option on Focal? Thanks in advance, Lee

6 11

[largescale-sig] Video meeting Mar 24: "Scaling RabbitMQ Clusters"
by Thierry Carrez 24 Mar '21

24 Mar '21

Hi everyone, The Large Scale SIG organizes video meetings around specific scaling topics, just one hour of discussion bootstrapped by a short presentation from operators of large scale deployments of OpenStack. Last month we did one on "Regions vs. Cells" which was pretty popular. Our next video meeting will be next Wednesday, March 24, at 1500utc on Zoom (link/password will be provided on this thread the day before the event). The theme will be: "Scaling RabbitMQ Clusters" quickstarted by a presentation from Gene Kuo (LINE). To see how 1500UTC will translate into in your corner of the world, you can doublecheck: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20210324T15 If you're interested in sharing your experience on this topic, or just want to learn more about that, please mark your calendar and join us next Wednesday ! -- Thierry Carrez (ttx)

1 1

[cinder] review priorities right now
by Brian Rosmaita 23 Mar '21

23 Mar '21

Please focus on the following two items in preparing for RC-1 on Thursday. 1. There are a few follow-up patches fixing bugs in driver features listed at the top of this etherpad: https://etherpad.opendev.org/p/cinder-wallaby-features 2. There's also a series of patches fixing bugs in Cinder quotas, which as you all know, have been a perennial pain point for operators. Most of these already have at least one +2. Start here: https://review.opendev.org/c/openstack/cinder/+/778182 and work your way up the "Relation chain" displayed in gerrit. Here's the complete list: https://review.opendev.org/q/topic:%22unused-quota-classes%22+(status:open%… Happy Reviewing! brian

1 0

Re: [Release-job-failures] Release of openstack/monasca-grafana-datasource for ref refs/tags/1.3.0 failed
by Thierry Carrez 23 Mar '21

23 Mar '21

We had a release job failure during the processing of the tag event when 1.3.0 was (successfully) pushed to openstack/monasca-grafana-datasource. Tags on this repository trigger the release-openstack-javascript job, which failed during pre playbook when trying to run yarn --version with the following error: /usr/share/yarn/lib/cli.js:46100 let { ^ SyntaxError: Unexpected token { at exports.runInThisContext (vm.js:53:16) at Module._compile (module.js:373:25) at Object.Module._extensions..js (module.js:416:10) at Module.load (module.js:343:32) at Function.Module._load (module.js:300:12) at Module.require (module.js:353:17) at require (internal/module.js:12:17) at Object.<anonymous> (/usr/share/yarn/bin/yarn.js:24:13) at Module._compile (module.js:409:26) at Object.Module._extensions..js (module.js:416:10) See https://zuul.opendev.org/t/openstack/build /cdffd2a26a0d4a5b8137edb392fa5971 This prevented the job from running (likely resulting in nothing being uploaded to NPM? Not a JS job specialist), which in turn prevented announce-release job from announcing it. -- Thierry Carrez (ttx)

5 7

[CloudKitty] vPTG meeting dates
by Rafael Weingärtner 23 Mar '21

23 Mar '21

Hello guys, We need to pick a date for our vPTG meeting. The suggested date and time is Monday April 19 13UTC - 14UTC (https://ethercalc.net/oz7q0gds9zfi) which would fit exactly in one of our Monday meetings. Does everybody else agree with this date and time? Let me know if that works for you. If it does, I will book the room. -- Rafael Weingärtner

2 2

[nova][placement] Xena PTG
by Balazs Gibizer 23 Mar '21

23 Mar '21

Hi, As you probably already know that the next PTG will be held between Apr 19 - 23. To organize the gathering I need your help: 1) Please fill the doodle[1] with timeslots when you have time to join to our sessions. Please do this before end of 21st of March. 2) Please add your PTG topic in the etherpad[2]. If you feel your topic needs a cross project cooperation please note that in the etherpad which other teams are needed. Cheers, gibi [1] https://doodle.com/poll/ib2eu3c4346iqii3 [2] https://etherpad.opendev.org/p/nova-xena-ptg

1 1

[openstack-helm] No meeting tomorrow March 23rd
by Gage Hugo 23 Mar '21

23 Mar '21

Hey team, Since there are no agenda items [0] for the IRC meeting tomorrow, March 23rd, we will cancel it. Our next IRC meeting will be March 30th. Thanks [0] https://etherpad.opendev.org/p/openstack-helm-weekly-meeting

1 0

Re: OpenStack Victoria - Kolla-Ansible - Cinder - iSCSI-backend - Update driver status failed: config is uninitialized.
by Stefan Kelber 23 Mar '21

23 Mar '21

1 0

[heat-translator] Ask new release 2.2.1 to fix Zuul jobs
by Yoshito Ito 23 Mar '21

23 Mar '21

Hi heat-translator core members, I'd like you to review the following patches [1][2], and to ask if we can release 2.2.1 with these commits to fix our Zuul jobs. In release 2.2.0, our Zuul jobs are broken [3] because of new release of tosca-parser 2.3.0, which provides strict validation of required attributes in [4]. The patch [1] fix this issue by updating our wrong test samples. The other [2] is now blocked by this issue and was better to be merged in 2.2.0. I missed the 2.2.0 release patch [5] because none of us was added as reviewers. So after merging [1] and [2], I will submit a patch to make new 2.2.1 release. [1] https://review.opendev.org/c/openstack/heat-translator/+/779642 [2] https://review.opendev.org/c/openstack/heat-translator/+/778612 [3] https://bugs.launchpad.net/heat-translator/+bug/1918360 [4] https://opendev.org/openstack/tosca-parser/commit/00d3a394d5a3bc13ed7d2f1d7… [5] https://review.opendev.org/c/openstack/releases/+/777964 Best regards, Yoshito Ito

2 2

Re: [infra][tact-sig] Extracting commits data for the first and the last code review revisions
by Chouchen, Moataz 23 Mar '21

23 Mar '21

Hi Jermey, Thank you for your quick answer! That's what I am looking for and I am very grateful to you and to all the community. Best regards, Moataz. ________________________________ From: Jeremy Stanley Sent: Monday, March 22, 2021 8:41 AM To: openstack-discuss(a)lists.openstack.org Cc: Chouchen, Moataz Subject: Re: [infra][tact-sig] Extracting commits data for the first and the last code review revisions [I'm keeping you in Cc since you don't appear to be subscribed to this mailing list.] On 2021-03-22 15:00:03 +0000 (+0000), Chouchen, Moataz wrote: [...] > I extracted the revision IDs for the first and the last revisions > (supposed to be first and last commit hashes) for each code review > in puppet projects. However, when we have more than one revision, > we were unable to find the commits for the first revision in the > git for all the projects and we only have commits data for the > last revision. I would like to know if the commit of the first > revision does exist in the CVS, and it's possible to extract? [...] Conveniently, Gerrit stores all of this in the repository and we replicate it to our Git server farm. The complication is that, because of the amend/rebase-oriented nature of Gerrit's code review workflow, those earlier revisions don't appear in the branch history. Instead, they are tracked with named references which look like changes/XX/YYYY/Z where YYYY is the numeric index identifier for the change, XX is the last two digits of YYYY (for sharding purposes), and Z is the revision number starting from 1. Let's take this puppet-nova change as an example: https://opendev.org/openstack/puppet-nova/commit/d385ca8 There you can see Gitea is displaying an indicator that there is a named ref of changes/63/764763/5 associated with the commit. That means it was revision 5 of change 764763 in Gerrit. You can fetch that ref like so: git fetch origin changes/63/764763/5 That will fill FETCH_HEAD which you can then use with checkout, show, or any other relevant subcommands. Conveniently, if you instead want the first revision of 764763, all you have to do differently is adjust the revision number: git fetch origin changes/63/764763/1 Hopefully that's what you're looking for? -- Jeremy Stanley

1 0

Extracting commits data for the first and the last code review revisions
by Chouchen, Moataz 23 Mar '21

23 Mar '21

Dear Openstack community, Hope you are doing well. I am Moataz Chouchen a Ph.D. student from ETS Montreal. I am interested in studying the impact of code review on the quality of puppet files. For this reason, I am interested in studying the code reviews containing puppet files (We investigate puppet projects.) and comparing puppet files in the first revision and the last revision for each code review. To achieve this, I extracted the revision IDs for the first and the last revisions (supposed to be first and last commit hashes) for each code review in puppet projects. However, when we have more than one revision, we were unable to find the commits for the first revision in the git for all the projects and we only have commits data for the last revision. I would like to know if the commit of the first revision does exist in the CVS, and it's possible to extract? In case these commits do not exist, how could, we retrieve the files in the first version of a review? Thank you very much, Best regards, Moataz

2 1

[oslo] Wallaby branches are now cut for oslo world
by Herve Beraud 23 Mar '21

23 Mar '21

Hello Osloers, As $subject (FYI) Thanks for your attention -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

2 1

Re: [glance][openstack-ansible] Snapshots disappear during saving
by Oliver Wenz 23 Mar '21

23 Mar '21

Hi Dmitriy, I tried the fix anyways, i.e. I replaced the value of swift_fallocate_reserve as shown here: https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/782117/… Now I'm getting the same error with '%%%' instead of '%%': Mar 22 15:04:49 bc1bl12 systemd[1]: swift-account-reaper.service: Main process exited, code=exited, status=1/FAILURE Mar 22 15:04:49 bc1bl12 systemd[1]: swift-account-reaper.service: Failed with result 'exit-code'. Mar 22 15:04:51 bc1bl12 systemd[1]: swift-account-reaper.service: Scheduled restart job, restart counter is at 171887. Mar 22 15:04:51 bc1bl12 systemd[1]: Stopped swift-account-reaper service. Mar 22 15:04:51 bc1bl12 systemd[1]: Started swift-account-reaper service. Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]: Traceback (most recent call last): Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]: File "/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/swift/common/utils.py", line 803, in config_fallocate_val> Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]: reserve_value = float(reserve_value[:-1]) Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]: ValueError: could not convert string to float: '1%%' Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]: During handling of the above exception, another exception occurred: Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]: Traceback (most recent call last): Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]: File "/openstack/venvs/swift-22.1.0/bin/swift-account-reaper", line 23, in <module> Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]: run_daemon(AccountReaper, conf_file, **options) Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]: File "/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/swift/common/daemon.py", line 304, in run_daemon Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]: utils.config_fallocate_value(conf.get('fallocate_reserve', '1%')) Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]: File "/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/swift/common/utils.py", line 809, in config_fallocate_val> Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]: raise ValueError('Error: %s is an invalid value for fallocate' Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]: ValueError: Error: 1%%% is an invalid value for fallocate_reserve. Kind regards, Oliver > Message: 3 > Date: Mon, 22 Mar 2021 12:06:19 +0200 > From: Dmitriy Rabotyagov <noonedeadpunk(a)ya.ru> > To: "openstack-discuss(a)lists.openstack.org" > <openstack-discuss(a)lists.openstack.org> > Subject: Re: [glance][openstack-ansible] Snapshots disappear during > saving > Message-ID: <984511616407178(a)mail.yandex.ru> > Content-Type: text/plain; charset=utf-8 > > Well, looking into the fix I suggested, I'm not sure if it's valid one. > There's really be a mess in patches, and according to the log provided, config > needs just `1%` instead of the `1%%` you currently have. > > And it feels like that's what default behaviour should do with [1]. But I'm > pretty sure that this error was making swift fail and thus having weird issues > while operating. > > So I'm not sure what specifily wrong with value of swift_fallocate_reserve - > maybe we've missed some config to define it or it has been overriden > somewhere, but it feels like current default should cover issue you see in > swift... > > [1] > https://opendev.org/openstack/openstack-ansible-os_swift/src/branch/stable/… > > 22.03.2021, 11:56, "Dmitriy Rabotyagov" <noonedeadpunk(a)ya.ru>: > > Yes, 1%% is smth we're fighting for years, as this setting changes on the > > swift side from time to time, and I really lost account which one is valid > > at the moment. > > > > Here's related SWIFT bug: > > > > https://bugs.launchpad.net/swift/+bug/1844368 > > > > I've just pushed > > https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/782117 to > > cover this issue. Can you try applying this change manually to see if this > > works? > > > > 22.03.2021, 11:25, "Oliver Wenz" <oliver.wenz(a)dhbw-mannheim.de>: > >> Hi Dmitriy, > >> thanks for your answer! Yes, we do use swift and its use as glance backend > >> is > >> intentional. > >> > >> I got the following from the swift-proxy-server logs in the swift container > >> on > >> the infra host after taking a snapshot: > >> > >> Mar 22 08:43:43 infra1-swift-proxy-container-27169fa7 proxy-server[87]: > >> Client > >> disconnected without sending last chunk (txn: > >> txa7c64547baf0450eb0034-006058588b) (client_ip: 192.168.110. > >> 106) > >> Mar 22 08:43:43 infra1-swift-proxy-container-27169fa7 proxy-server[87]: > >> 192.168.110.106 192.168.110.211 22/Mar/2021/08/43/43 PUT > >> /v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/3ec63ec2-aa3b-4c3b-a904-b55d1a6ec878 > >> -00001 HTTP/1.0 499 - python-swiftclient-3.10.1 gAAAAABgWFiKMz9R... > >> 204800000 89 > >> - txa7c64547baf0450eb0034-006058588b - 52.2991 - - 1616402571.623875856 > >> 1616402623.922997952 0 > >> > >> On the swift host some services logs contain errors. E.g. the > >> swift-container-updater service: > >> > >> Mar 22 08:42:19 bc1bl12 systemd[1]: swift-container-updater.service: Main > >> process exited, code=exited, status=1/FAILURE > >> Mar 22 08:42:19 bc1bl12 systemd[1]: swift-container-updater.service: Failed > >> with > >> result 'exit-code'. > >> Mar 22 08:42:21 bc1bl12 systemd[1]: swift-container-updater.service: > >> Scheduled > >> restart job, restart counter is at 162982. > >> Mar 22 08:42:21 bc1bl12 systemd[1]: Stopped swift-container-updater > >> service. > >> Mar 22 08:42:21 bc1bl12 systemd[1]: Started swift-container-updater > >> service. > >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: Traceback (most > >> recent > >> call last): > >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: File > >> "/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/swift/common/utils.py", > >> line 803, in config_fallocate_value > >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: reserve_value = > >> float(reserve_value[:-1]) > >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: ValueError: could > >> not > >> convert string to float: '1%' > >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: During handling of > >> the > >> above exception, another exception occurred: > >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: Traceback (most > >> recent > >> call last): > >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: File > >> "/openstack/venvs/swift-22.1.0/bin/swift-container-updater", line 23, in > >> <module> > >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: > >> run_daemon(ContainerUpdater, conf_file, **options) > >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: File > >> "/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/swift/common/daemon.py", > >> line 304, in run_daemon > >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: > >> utils.config_fallocate_value(conf.get('fallocate_reserve', '1%')) > >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: File > >> "/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/swift/common/utils.py", > >> line 809, in config_fallocate_value > >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: raise > >> ValueError('Error: %s is an invalid value for fallocate' > >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: ValueError: Error: > >> 1%% > >> is an invalid value for fallocate_reserve. > >> > >> The same exit-code and traceback shows in the logs of > >> swift-container-auditor, > >> swift-account-auditor and swift-account-reaper services. Does this tell you > >> anything useful? > >> > >> We didn't experience any problems when uploading files to containers, only > >> when > >> taking snapshots of instances. > >> > >> Kind regards, > >> Oliver > >> > >>> ------------------------------ > >>> > >>> Message: 4 > >>> Date: Thu, 18 Mar 2021 12:44:47 +0200 > >>> From: Dmitriy Rabotyagov <noonedeadpunk(a)ya.ru> > >>> To: "openstack-discuss(a)lists.openstack.org" > >>> <openstack-discuss(a)lists.openstack.org> > >>> Subject: Re: [glance][openstack-ansible] Snapshots disappear during > >>> saving > >>> Message-ID: <374941616064157(a)mail.yandex.ru> > >>> Content-Type: text/plain; charset=utf-8 > >>> > >>> Hi Olver, > >>> > >>> Am I right that you're also using OpenStack Swift and it's intentional to > >>> store images there? > >>> Since the issue is related to the upload process into the Swift. So also > >>> checking Swift logs > >>> be usefull as well. > > > > -- > > Kind Regards, > > Dmitriy Rabotyagov > > > -- > Kind Regards, > Dmitriy Rabotyagov > > > > ------------------------------ > > Message: 4 > Date: Mon, 22 Mar 2021 12:35:37 +0100 > From: Bernard Cafarelli <bcafarel(a)redhat.com> > To: OpenStack Discuss <openstack-discuss(a)lists.openstack.org> > Subject: Re: [neutron][stable] neutron-lib stable branches core > reviewers > Message-ID: > <CABHdKwrEN99b3VS89U0M8BRYevV2oqGLSPkSfVzdjCZNFoKHpA(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > On Thu, 18 Mar 2021 at 14:48, Slawek Kaplonski <skaplons(a)redhat.com> wrote: > > > Hi, > > > > I just noticed that neutron-lib project has got own group > > "neutron-lib-stable-maint" which has +2 powers in neutron-lib stable > > branches [1]. As I see now in gerrit that group don't have any members. > > Would it be maybe possible to remove that group and add > > "neutron-stable-maint" to the neutron-lib stable branches instead? If yes, > > should I simply propose patch to change [1] or is there any other way which > > I should do it? > > > I guess we never spotted that "neutron-lib-stable-maint" before, as it > seems neutron cores had +2 powers before: > https://review.opendev.org/c/openstack/neutron-lib/+/717093 > > But yes it could be useful to have either of these groups (neutron cores or > stable cores) in, backports to neutron-lib are rare but some happen - I > remember one some time ago where it was valid to have a patch backported. > > > > [1] > > https://github.com/openstack/project-config/blob/master/gerrit/acls/opensta… > > > > -- > > Slawek Kaplonski > > Principal Software Engineer > > Red Hat > > > > -- > Bernard Cafarelli >

1 0

[neutron] Bug deputy report for week of March 15th
by Brian Haley 22 Mar '21

22 Mar '21

Hi, I was Neutron bug deputy last week. Below is a short summary about the reported bugs. -Brian Critical bugs ------------- * UT are failing when runs with neutron-lib master - https://bugs.launchpad.net/neutron/+bug/1919280 - https://review.opendev.org/c/openstack/neutron/+/780802 * Functional tests failing due to timeout while spawning metadata proxy - https://bugs.launchpad.net/neutron/+bug/1919344 * Functional tests failing due to timeout while killing process - https://bugs.launchpad.net/neutron/+bug/1919345 * [OVN] "OVS_SYSCONFDIR" should be created before writing the OVS system-id file - https://bugs.launchpad.net/neutron/+bug/1920634 - Devstack patch proposed by Slawek High bugs --------- * [OVN] Do not enable send_periodic on router ports which are connected to provider networks - https://bugs.launchpad.net/neutron/+bug/1919347 - https://review.opendev.org/c/openstack/neutron/+/780916 * Project administrators are allowed to view networks across projects - https://bugs.launchpad.net/neutron/+bug/1919386 - https://review.opendev.org/c/openstack/neutron-lib/+/781075 * Creation of floating ip with new rbac policies fails - https://bugs.launchpad.net/neutron/+bug/1920001 Low bugs -------- * [L3][L2 pop] Is it necessary to enforce enabling the option arp_responder and l2_population? - https://bugs.launchpad.net/neutron/+bug/1919107 - Can be reverted now that other DVR changes are being reverted * [OVN][FT] FT tests log a SQL exception - https://bugs.launchpad.net/neutron/+bug/1919352 - https://review.opendev.org/c/openstack/neutron/+/780926 Wishlist bugs ------------- * Automatic rescheduling of BGP speakers on DrAgents - https://bugs.launchpad.net/neutron/+bug/1920065 - RFE

1 0

[nova] When are "_del" directories deleted?
by Eugen Block 22 Mar '21

22 Mar '21

Hi *, I stumbled across a different issue when I noticed there's a <UUID>_del directory in /var/lib/nova/instances/. It's a shared directory to enable live-migration. I noticed these "_del" directories before but I didn't really investigate, everything seems to work as expected. But I would like to understand what exactly happens with such a directory, for example currently there's just one instance having also a _del directory. The timestamp is from 2 years ago when the instance was resized the first time. Looking in the nova code [1] I found the reference to the resize function, I just don't quite understand what would happen with that instance if it was shutdown. Would nova delete it to "clean up"? Can I remove that directory safely to prevent nova from doing anything destructive to that instance? I'd appreciate any insights about this. Just to be safe I created an RBD snapshot of the disk to prevent a deletion. Best regards, Eugen [1] https://opendev.org/openstack/nova/src/commit/146b27f22327f8d60c1017c22ccf1…

2 2

[ironic] How to move node from active state to manageable
by Igal Katzir 22 Mar '21

22 Mar '21

Hello Forum, I have an overcloud that gone bad and I am trying to re-deploy it, Running rhos16.1 with one director and two overcloud nodes (compute and controller) I have re-installed undercloud and having both nodes in an active provisioning state. Do I need to run introspection again? Here is the outputted for baremetal node list: (undercloud) [stack@interop010 ~]$ openstack baremetal node list +--------------------------------------+------------+--------------------------------------+-------------+--------------------+-------------+ | UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance | +--------------------------------------+------------+--------------------------------------+-------------+--------------------+-------------+ | 97b9a603-f64f-47c1-9fb4-6c68a5b38ff6 | interop025 | c7bf16b7-eb3c-4022-88de-7c5a78cda174 | power on | active | False | | 4b02703a-f765-4ebb-85ed-75e88b4cbea5 | interop026 | 99223f65-6985-4815-92ff-e19a28c2aab1 | power on | active | False | +--------------------------------------+------------+--------------------------------------+-------------+--------------------+-------------+ When I want to move each node from active > manage I get an error: (undercloud) [stack@interop010 ~]$ openstack baremetal node manage 4b02703a-f765-4ebb-85ed-75e88b4cbea5 The requested action "manage" can not be performed on node "4b02703a-f765-4ebb-85ed-75e88b4cbea5" while it is in state "active". (HTTP 400) How do I get to a state which is ready for deployment (available) ? Thanks, Igal

2 3

Re: [glance][openstack-ansible] Snapshots disappear during saving
by Oliver Wenz 22 Mar '21

22 Mar '21

Hi Dmitriy, thanks for your answer! Yes, we do use swift and its use as glance backend is intentional. I got the following from the swift-proxy-server logs in the swift container on the infra host after taking a snapshot: Mar 22 08:43:43 infra1-swift-proxy-container-27169fa7 proxy-server[87]: Client disconnected without sending last chunk (txn: txa7c64547baf0450eb0034-006058588b) (client_ip: 192.168.110. 106) Mar 22 08:43:43 infra1-swift-proxy-container-27169fa7 proxy-server[87]: 192.168.110.106 192.168.110.211 22/Mar/2021/08/43/43 PUT /v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/3ec63ec2-aa3b-4c3b-a904-b55d1a6ec878 -00001 HTTP/1.0 499 - python-swiftclient-3.10.1 gAAAAABgWFiKMz9R... 204800000 89 - txa7c64547baf0450eb0034-006058588b - 52.2991 - - 1616402571.623875856 1616402623.922997952 0 On the swift host some services logs contain errors. E.g. the swift-container-updater service: Mar 22 08:42:19 bc1bl12 systemd[1]: swift-container-updater.service: Main process exited, code=exited, status=1/FAILURE Mar 22 08:42:19 bc1bl12 systemd[1]: swift-container-updater.service: Failed with result 'exit-code'. Mar 22 08:42:21 bc1bl12 systemd[1]: swift-container-updater.service: Scheduled restart job, restart counter is at 162982. Mar 22 08:42:21 bc1bl12 systemd[1]: Stopped swift-container-updater service. Mar 22 08:42:21 bc1bl12 systemd[1]: Started swift-container-updater service. Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: Traceback (most recent call last): Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: File "/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/swift/common/utils.py", line 803, in config_fallocate_value Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: reserve_value = float(reserve_value[:-1]) Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: ValueError: could not convert string to float: '1%' Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: During handling of the above exception, another exception occurred: Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: Traceback (most recent call last): Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: File "/openstack/venvs/swift-22.1.0/bin/swift-container-updater", line 23, in <module> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: run_daemon(ContainerUpdater, conf_file, **options) Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: File "/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/swift/common/daemon.py", line 304, in run_daemon Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: utils.config_fallocate_value(conf.get('fallocate_reserve', '1%')) Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: File "/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/swift/common/utils.py", line 809, in config_fallocate_value Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: raise ValueError('Error: %s is an invalid value for fallocate' Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: ValueError: Error: 1%% is an invalid value for fallocate_reserve. The same exit-code and traceback shows in the logs of swift-container-auditor, swift-account-auditor and swift-account-reaper services. Does this tell you anything useful? We didn't experience any problems when uploading files to containers, only when taking snapshots of instances. Kind regards, Oliver > ------------------------------ > > Message: 4 > Date: Thu, 18 Mar 2021 12:44:47 +0200 > From: Dmitriy Rabotyagov <noonedeadpunk(a)ya.ru> > To: "openstack-discuss(a)lists.openstack.org" > <openstack-discuss(a)lists.openstack.org> > Subject: Re: [glance][openstack-ansible] Snapshots disappear during > saving > Message-ID: <374941616064157(a)mail.yandex.ru> > Content-Type: text/plain; charset=utf-8 > > Hi Olver, > > Am I right that you're also using OpenStack Swift and it's intentional to > store images there? > Since the issue is related to the upload process into the Swift. So also > checking Swift logs > be usefull as well. >

2 2

[oslo] Cancel meeting today
by Herve Beraud 22 Mar '21

22 Mar '21

Hello Osloers, I won't be around to run the meeting today. If someone else from the Oslo team wants to run it, please feel free. Thanks for your understanding -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

1 0

[release] Xena PTG
by Herve Beraud 22 Mar '21

22 Mar '21

Oï releasers, The PTG is fast approaching (Apr 19 - 23). To help to organize the gathering: 1) please fill the doodle[1] with the time slots that fit well for you; 2) please add your PTG topics in our etherpad[2]. Voting will be closed on the 21st of March. Thanks for your reading, [1] https://doodle.com/poll/8d8n2picqnhchhsv [2] https://etherpad.opendev.org/p/xena-ptg-os-relmgt -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

1 2

Authenticating to devstack and listing nova flavors + servers
by Joe Wolfe 22 Mar '21

22 Mar '21

Hi All, I am working on an api and would like to get some more context behinds authenticating into the api - so I am trying to get a list out of available flavors 1 and servers to start off with. I am able to get a token that I can interact with against keystone which shows up as https://mydevstackurl/identity/ <https://mydevstackurl/identity/> however when I try and run a simple curl against my compute as follows: curl -H "X-Auth-Token: $OS_TOKEN" -s https://http://mydevstackurl/compute/api/v2.1/servers <https://http//mydevstackurl/compute/api/v2.1/servers> I got no response back from curl - no 200 ,no 30x no 4xx nothing. Can anyone help to get me started ? The documents seem to be pretty detailed - but the methods provided don’t seem to work either in the tutorial: for example: curl -s -H "X-Auth-Token: $OS_TOKEN" \ $OS_COMPUTE_API/flavors \ | python -m json.tool Only returns: zsh: no matches found: http://mydevstackurl/identity/v3/auth/tokens?nocatalog <http://mydevstackurl/identity/v3/auth/tokens?nocatalog> Expecting value: line 1 column 1 (char 0) Thanks

2 1

Re: [openstack-community] Authenticating to devstack and listing nova flavors + servers
by Jeremy Stanley 22 Mar '21

22 Mar '21

On 2021-03-21 16:39:42 -0400 (-0400), Joe Wolfe wrote: > I am working on an api and would like to get some more context > behinds authenticating into the api - so I am trying to get a list > out of available flavors 1 and servers to start off with. I am > able to get a token that I can interact with against keystone > which shows up as https://mydevstackurl/identity/ however when I > try and run a simple curl against my compute as follows: > > curl -H "X-Auth-Token: $OS_TOKEN" -s https://http://mydevstackurl/compute/api/v2.1/servers > > I got no response back from curl - no 200 ,no 30x no 4xx nothing. > > Can anyone help to get me started ? The documents seem to be > pretty detailed - but the methods provided don’t seem to work > either in the tutorial: for example: > > curl -s -H "X-Auth-Token: $OS_TOKEN" \ > $OS_COMPUTE_API/flavors \ > | python -m json.tool > > Only returns: > > zsh: no matches found: http://mydevstackurl/identity/v3/auth/tokens?nocatalog > Expecting value: line 1 column 1 (char 0) As the description of the community mailing list states, "This list does not provide support for OpenStack software." I have Cc'd the openstack-discuss mailing list which is intended for "Discussion of OpenStack use and development." Please follow up there with your question. I'm keeping you in the To header in case you're not subscribed there, so that people who reply to this will hopefully include you directly. -- Jeremy Stanley

1 0

Fw: Aw: Re: OpenStack Victoria - Kolla-Ansible - Cinder - iSCSI-backend - Update driver status failed: config is uninitialized.
by Stefan Kelber 22 Mar '21

22 Mar '21

1 1

OpenStack Victoria - Kolla-Ansible - Cinder - iSCSI-backend - Update driver status failed: config is uninitialized.
by Stefan Kelber 21 Mar '21

21 Mar '21

3 2

iPXE update/configuration for the undercloud.
by Kendrick 21 Mar '21

21 Mar '21

I have a melanox connectx3 and a few broadcom systems that do not properly pxe boot, using maas the broadcoms work fine. I was looking around and one of the solutions to similar problems from 2017 and 19 were to use a different version of the ipxe files. I was hoping to know what settings/changes/scripts openstack undercloud used so i can make sure the connectx3 support is built in to the ipxe file and what files needed to be copied where from the source buld to the containers since the containers have about 20 sets of the ipxe files from what i can see when i use find to locate them from the host. Regards Kendrick

1 0

[all][tc] Dropping lower-constraints testing from all projects
by Ghanshyam Mann 20 Mar '21

20 Mar '21

Hello Everyone, You might have seen the discussion around dropping the lower constraints testing as it becomes more challenging than the current value of doing it. Few of the ML thread around this discussion: - http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019521… - http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019390… As Oslo and many other project dropping or already dropped it, we should decide it for all other projects also otherwise it can be more changing than it is currently. We have not defined it in PTI or testing runtime so it is always up to projects if they still want to keep it but we should decide a general recommendation here. -gmann

12 46

[tripleo][ci] ovb featureset01/35 and other jobs are down
by Wesley Hayutin 20 Mar '21

20 Mar '21

Greetings, ATM, the RDO team is working w/ Vexxhost to diagnose an outage in vexxhost that has caused various problems w/ heat stacks and now overcloud node registration and introspection. The issue is tracked w/ vexx and https://bugs.launchpad.net/tripleo/+bug/1920101. The third party check jobs are not expected to pass at this time. We'll follow up when we have more information. Thank you

1 0

[tripleo] Nominate David J. Peacock (dpeacock) for Validation Framework Core
by Gaël Chamoulaud 20 Mar '21

20 Mar '21

Hi TripleO Devs, David is already a key member of our team since a long time now, he provided all the needed ansible roles for the Validation Framework into tripleo-ansible-operator. He continuously provides excellent code reviews and he is a source of great ideas for the future of the Validation Framework. That's why we would highly benefit from his addition to the core reviewer team. Assuming that there are no objections, we will add David to the core team next week. Thanks, David, for your excellent work! -- Gaël Chamoulaud - (He/Him/His) .::. Red Hat .::. OpenStack .::. .::. DFG:DF Squad:VF .::.

9 12

[Cinder] Review pretty please
by Sam Morrison 19 Mar '21

19 Mar '21

Hi Cinder cores, I've had a simple review [1] waiting for several months, could I please get some feedback on this. Thanks! Sam [1] https://review.opendev.org/c/openstack/cinder/+/764875

3 2

[ops] Bandwidth problem on computes
by Jahson Babel 19 Mar '21

19 Mar '21

Hello everyone, I have a bandwidth problem between the computes nodes of an openstack cluster. This cluster runs on Rocky version with OpenVSwitch. To simplify I'll just pick 3 servers, one controller and two computes nodes all connected to the same switch. Every server is configured with two 10G links. Those links are configured in LACP /teaming. From what I understand of teaming and this configuration I should be able to get 10Gbps between all three nodes. But if I iperf we are way below this : compute1 # sudo iperf3 -c compute2 -p 5201 Connecting to host compute2, port 5201 [ 4] local X.X.X.X port 44946 connected to X.X.X.X port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 342 MBytes 2.87 Gbits/sec 137 683 KBytes [ 4] 1.00-2.00 sec 335 MBytes 2.81 Gbits/sec 8 501 KBytes Plus the problem seems to be only present with incoming traffic. Which mean I can almost get the full 10gbps if I iperf from a compute to the controller. compute1 # sudo iperf3 -c controller -p 5201 Connecting to host controller, port 5201 [ 4] local X.X.X.X port 39008 connected to X.X.X.X port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.10 GBytes 9.41 Gbits/sec 0 691 KBytes [ 4] 1.00-2.00 sec 1.09 GBytes 9.38 Gbits/sec 0 803 KBytes If I do the opposite I get the same results I was getting between the 2 computes. From the tests we've done it seems related to the openstack's services, specifically neutron or OpenVSwitch. From the time those services are running we can't get the full bandwidth. Stopping the services won't fix the issue, in our case removing the packages and rebooting is the only way to obtain the full bandwidth between computes. I voluntarily didn't mention VMs to simplify the question but of course this behavior can also be observed in VMs Knowing that we can achieve 10Gbps it doesn't seems related to the hardware nor the OS. That why we suspect OpenStack's services. But I couldn't find any evidence or misconfiguration that could confirm that. So if anyone got some hints about that kind of setup and/or how mitigate bandwidth decrease I would appreciate. Let know if you need more info. Thanks in advance, Jahson

2 2

Re: [nova] Train: how to use newer API microversion
by Eugen Block 19 Mar '21

19 Mar '21

Alright, I made it work it seems. After I added the new endpoints (v2.1) and disabled the old ones (v2) I restarted nova-api service. This allowed me to use microversion 2.37 to use the option "--nic none", so that works as expected now. I just don't know yet if there are any clients that require v2 so for now I let them in "disabled" state. Thanks, Eugen Zitat von Eugen Block <eblock(a)nde.ag>: >> If you (or your end users) rely on the legacy APIs, you can have both >> like this (192.0.2.40 is the example IP address of the API host; the >> /compute could be :some_port if you prefer that): > > I started with disabling the old endpoints but then I couldn't use > compute at all to create a new instance. There must be something > still relying on the old endpoint but I can't find it right now. > I'll give it another shot tomorrow. Thanks for your input. > > > Zitat von Radosław Piliszek <radoslaw.piliszek(a)gmail.com>: > >> On Thu, Mar 18, 2021 at 3:50 PM Eugen Block <eblock(a)nde.ag> wrote: >>> I also found that already, but it doesn't describe how I can move from >>> v2 to v2.1 in order to use only the newer API. Do you have any >>> information about that? >> >> If you (or your end users) rely on the legacy APIs, you can have both >> like this (192.0.2.40 is the example IP address of the API host; the >> /compute could be :some_port if you prefer that): >> >> +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ >> | ID | Region | Service Name | Service Type | Enabled | >> Interface | URL | >> +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ >> | xxx | RegionOne | nova | compute | True | public >> | http://192.0.2.40/compute/v2.1 | >> | xxx | RegionOne | nova_legacy | compute_legacy | True | public >> | http://192.0.2.40/compute/v2/$(project_id)s | >> +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ >> >> If you don't need the legacy APIs, you can delete their endpoints. >> >> -yoctozepto

1 0

[keystone][kolla][horizon] versioned or unversioned keystone endpoints?
by Mark Goddard 19 Mar '21

19 Mar '21

Hi, There seems to be some inconsistency around whether Keystone endpoints in the catalog should be versioned or not. Specifically, should they have a /v3 suffix? I did a survey of code using keystone-manage bootstrap via codesearch: * Kolla Ansible: unversioned * DevStack: unversioned * puppet-keystone: unversioned * Keystone install guides: versioned * OSA: unversioned I imagine that when v2 was around, the endpoint needed to be unversioned. But was there a point when it became recommended to add a version? I'm asking because of an issue in horizon when downloading a clouds.yaml file for application credentials. If the auth_url field is unversioned, then some older client libraries will fail with a Not Found error during e.g. openstack server list. I tested using Stein and Train U-C. The former fails, the latter works. If I use Stein U-C, then update keystoneauth1 to 3.17.3 (from Train U-C), it works. Thanks, Mark

1 0

[neutron] Drivers meeting agenda - 19.03.2021
by Slawek Kaplonski 19 Mar '21

19 Mar '21

Hi, First of all sorry for sending it so late - I simply forgot to send this email yesterday :) For today's drivers meeting we have 1 RFE to discuss: * https://bugs.launchpad.net/neutron/+bug/1904559 - it is continuation of the discussion from last week. Michael Johnson from Designate team provided his feedack on it so I think that based on that we can make final decision about that RFE on our today's meeting. -- Slawek Kaplonski Principal Software Engineer Red Hat

1 1

[release] Release countdown for week R-3 Mar 22 - Mar 26
by Herve Beraud 19 Mar '21

19 Mar '21

Development Focus ----------------- The Release Candidate (RC) deadline is next Thursday, 25 March, 2021. Work should be focused on fixing any release-critical bugs. General Information ------------------- All deliverables released under a cycle-with-rc model should have a first release candidate by the end of the week, from which a stable/wallaby branch will be cut. This branch will track the Wallaby release. Once stable/wallaby has been created, the master branch will be ready to switch to Xena development. While the master branch will no longer be feature-frozen, please prioritize any work necessary for completing Wallaby plans. Release-critical bugfixes will need to be merged in the master branch first, then backported to the stable/wallaby branch before a new release candidate can be proposed. Actions ------- Early in the week, the release team will be proposing RC1 patches for all cycle-with-rc projects, using the latest commit from the master branch. If your team is ready to go for cutting RC1, please let us know by leaving a +1 on these patches. If there are still a few more patches needed before RC1, you can -1 the patch and update it later in the week with the new commit hash you would like to use. Remember, stable/wallaby branches will be created with this, so you will want to make sure you have what you need included to avoid needing to backport changes from the master branch (which will technically then be Xena) to this stable branch for any additional RCs before the final release. The release team will also be proposing releases for any deliverable following a cycle-with-intermediary model that has not produced any Wallaby release so far. Finally, now is a good time to finalize release highlights. Release highlights help shape the messaging around the release and make sure that your work is properly represented. Upcoming Deadlines & Dates -------------------------- RC1 deadline: 25 March, 2021 (R-3 week) Final RC deadline: 08 April, 2021 (R-1 week) Final Wallaby release: 14 April, 2021 Xena virtual PTG: 19 - 23 April, 2021 Thanks for your reading -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

1 0

OSP13 - Queens - Inconsistent data between in DB between ipamallocation and ipallocation
by Laurent Dumont 19 Mar '21

19 Mar '21

Hey everyone, We just troubleshooted (and fixed) a weird issue with ipamallocation. It looked like ipallocations we're not properly removed from the ipamallocation table. This caused port creations to fail with an error about the IP being already allocated (when there wasn't an actual port using the IP). We found this old issue https://bugs.launchpad.net/neutron/+bug/1884532 but not a whole lot of details in there. Has anyone seen this before? Thanks!

2 2

[neutron] Unit test jobs broken
by Slawek Kaplonski 19 Mar '21

19 Mar '21

Hi, Our UT gate jobs are now broken due to new neutron-lib release. Fix for that is already proposed in [1]. So if Your UT jobs are failing like e.g. in [2] please don't recheck Your patch before [1] will be merged. [1] https://review.opendev.org/c/openstack/neutron/+/780802 [2] https://8ee33ec1a424812a1857-16390f05fde22eb723929856dcc38fcd.ssl.cf5.rackc… -- Slawek Kaplonski Principal Software Engineer Red Hat

1 1

[Neutron] [Designate] Private / Internal DNS Zones with custom records for i.e. service discovery
by Christian Rohmann 19 Mar '21

19 Mar '21

Hey Openstack-Discuss, apart from the standardized and auto-created records for ports / floating-ips and instances (https://docs.openstack.org/neutron/latest/admin/config-dns-int.html) - is there any way to allow users to add their own records which then only resolve internally? The Neutron API (https://docs.openstack.org/api-ref/network/v2/#id52) seems to be all about the resources it manages, so no additional or custom records there. Looking at the Designate API https://docs.openstack.org/api-ref/dns/?expanded=create-zone-detail#create-… is does not seem to be an option to mark a zone as "internal" or "private". But maybe there is another way to add records to the internal zone? I am thinking of an only internally resolvable / valid DNS zone carrying records for i.e. service discovery / cluster forming. There are more and more tools just looking up a DNS records to find cluster members ... * ElasticSearch: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-dis… * Hazelcast: https://github.com/hazelcast/hazelcast-kubernetes#understanding-discovery-m… * HiveMQ: https://github.com/hivemq/hivemq-dns-cluster-discovery-extension/blob/maste… * RabbitMQ: https://www.rabbitmq.com/cluster-formation.html#peer-discovery-dns [...] and with Kubernetes and the headless service concept there are more tools (ab)using DNS for this every week. So having internal dns zones which only resolve within the project would be really helpful. The hyperscalers call this feature * AWS "Private hosted zones" (https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-priv…) * Azure "Private DNS" (https://medium.com/azure-architects/exploring-azure-private-dns-be65de08f780) * GCP "Private zone" (https://cloud.google.com/blog/products/networking/introducing-private-dns-z…) * Alibaba Cloud "DNS PrivateZone" (https://www.alibabacloud.com/product/private-zone) Regards Christian

3 2

[tripleo] launchpad wallaby-3 milestone closed use wallaby-rc1
by Marios Andreou 19 Mar '21

19 Mar '21

Hi tripleo o/ as just mentioned on irc #tripleo, I closed out the wallaby-3 launchpad milestone yesterday. If you want to set a milestone in bugs please use wallaby-rc1 (wallaby-3 is no longer in the "assign milestone" list). I moved the wallaby-3 unclosed bugs to wallaby-rc1 with our helper script [1] results at [2]. Please try and keep bug status updated. Things like moving it to fix-released once we're done merging fixes - it is easy to forget about that. We have 155 bugs triaged at https://launchpad.net/tripleo/+milestone/wallaby-rc1 at least some of those can be moved to fixed/closed nofix etc. Yes I know, #cool story bro thank you for reading ;) and for your help on bug status regards, marios [1] https://opendev.org/openstack/tripleo-ci/src/branch/master/scripts/move_bug… [2] https://gist.github.com/marios/b3155fe3b1318cc26bfa4bc15c764a26#gistcomment…

1 0

Re: [nova] Train: how to use newer API microversion
by Radosław Piliszek 19 Mar '21

19 Mar '21

On Thu, Mar 18, 2021 at 3:50 PM Eugen Block <eblock(a)nde.ag> wrote: > I also found that already, but it doesn't describe how I can move from > v2 to v2.1 in order to use only the newer API. Do you have any > information about that? If you (or your end users) rely on the legacy APIs, you can have both like this (192.0.2.40 is the example IP address of the API host; the /compute could be :some_port if you prefer that): +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ | xxx | RegionOne | nova | compute | True | public | http://192.0.2.40/compute/v2.1 | | xxx | RegionOne | nova_legacy | compute_legacy | True | public | http://192.0.2.40/compute/v2/$(project_id)s | +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ If you don't need the legacy APIs, you can delete their endpoints. -yoctozepto

1 0

[ironic] [infra] Making Glean work with IPA for static IP assignment
by Dmitry Tantsur 18 Mar '21

18 Mar '21

Hi Ian, Given our timezone difference, I've decided to send you an email, adding openstack-discuss in CC for greater exposure. We're trying to make IPA (our agent ramdisk) to work without relying on DHCP for cases like Edge deployments. We've settled on the network_data.json format for the API side and wanted to use Glean on the ramdisk to apply it. You can read more details in [1]. The problem is, I cannot make Glean work with any ramdisk I build. The crux of the problem seems to be that NetworkManager (used by default in RHEL, CentOS, Fedora and Debian at least) starts very early, creates the default connection and ignores whatever files Glean happens to write afterwards. On Debian running `systemctl restart networking` actually helped to pick the new configuration, but I'm not sure we want to do that in Glean. I haven't been able to make NetworkManager pick up the changes on RH systems so far. I build ramdisks using IPA-builder [2] by adding the simple-init element. I've tried removing dhcp-all-interfaces (which we depend on by default) to no effect. I've tried disabling the DHCP server, ended up with no IP connectivity at all. I haven't tried to shutdown and restart a connection as recommended in [3] since it's not trivial to do via SSH. Do you maybe have any hints how to proceed? I'd be curious to know how static IP assignment works in the infra setup. Do you have images with NetworkManager there? Do you use the simple-init element? Any help is very appreciated. Dmitry [1] https://specs.openstack.org/openstack/ironic-specs/specs/not-implemented/L3… [2] https://opendev.org/openstack/ironic-python-agent-builder [3] https://mail.gnome.org/archives/networkmanager-list/2014-January/msg00032.h… -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

4 9

[nova] Train: how to use newer API microversion
by Eugen Block 18 Mar '21

18 Mar '21

Hi *, I'm a little confused and could use some guidance regarding endpoints and microversions. Just recently I did a lot of maintenance in our old cloud, I believe it was created with Kilo and is now Train. Anyway, I wanted to use the "server create" option "--nic none" but couldn't because of the v2 endpoint for the nova service, right? According to the cli help this option is available from microversion 2.37+. This was the first attempt before adding new endpoints: controller:~ # openstack server create --image <UUID> --nic none --flavor 3 test-nic-none nics must be a list or a tuple, not <class 'str'> These were the endpoints before I added new ones: controller:~ # openstack endpoint list | grep nova | <UUID> | RegionOne | nova | compute | True | public | http://controller:<PORT>/v2/%(tenant_id)s | | <UUID> | RegionOne | nova | compute | True | internal | http://controller:<PORT>/v2/%(tenant_id)s | | <UUID> | RegionOne | nova | compute | True | admin | http://controller:<PORT>/v2/%(tenant_id)s | I checked with the Train install guide and created new endpoints for nova: controller:~ # openstack endpoint create --region RegionOne compute public http://controller:<PORT>/v2.1 controller:~ # openstack endpoint create --region RegionOne compute internal http://controller:<PORT>/v2.1 controller:~ # openstack endpoint create --region RegionOne compute admin http://controller:<PORT>/v2.1 I should mention that I haven't restarted any services yet, I wanted to check each step carefully since people are working in that cloud. In the debug output from creating a new instance I saw that it's still trying to use v2 api, so I disabled it, resulting in error messages that there's no compute service available. So I reenabled the old endpoints and now have both versions active. Then I tried this: controller:~ # openstack --os-compute-api-version 2.37 server create --image <UUID> --nic none --flavor 3 test-nic-none Invalid input for field/attribute networks. Value: none. 'none' is not of type 'array' (HTTP 400) (Request-ID: req-b3c53ab2-8040-42cf-bf07-d03e8f46cfd5) Now I'm running out of ideas how to upgrade our api endpoints so the services will actually be able to use them properly. I read Train release notes (we're planning to upgrade to Ussuri soon) and multiple docs about microversions but nothing seems to cover my issue. Could anyone shed some light? I must admit, during the upgrade cycles in the past years I just hoped everything would keep working and it did, I didn't put too much attention in the api versions yet. Any help is appreciated! Thanks and best regards, Eugen

2 2

[glance][openstack-ansible] Snapshots disappear during saving
by Oliver Wenz 18 Mar '21

18 Mar '21

Hi! We are currently experiencing problems with our OpenStack Ansible Victoria cloud when trying to create snapshots from instances. In some cases, everything works but often the pending snapshots just disappear. When this happens, the following glance-api.service errors show up: Mar 17 08:33:12 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:33:11.975 85 INFO glance.api.v2.image_data [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token. Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.283 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] REQ: curl -i http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance… -X PUT -H "X-Auth-Token: gAAAAABgUb7IBvZ_..." Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.285 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP STATUS: 504 Gateway Time-out Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.285 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP HEADERS: {'content-length': '92', 'cache-control': 'no-cache', 'content-type': 'text/html', 'connection': 'close'} Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.286 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP BODY: b"<html><body><h1>504 Gateway Time-out</h1>\nThe server didn't respond in time.\n</body></html>\n" Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.306 85 ERROR glance_store._drivers.swift.store [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Error during chunked upload to backend, deleting stale chunks.: swiftclient.exceptions.ClientException: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload. Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.320 85 ERROR glance_store._drivers.swift.store [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to add object to Swift. Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload..: swiftclient.exceptions.ClientException: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload. Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.327 85 ERROR glance.api.v2.image_data [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to upload image data due to internal error: glance_store.exceptions.BackendException: Failed to add object to Swift. Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Caught error: Failed to add object to Swift. Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload..: glance_store.exceptions.BackendException: Failed to add object to Swift. Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload.. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1014, in add 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self._delete_stale_chunks( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise() 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise value 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1003, in add 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi manager.get_connection().put_object( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1960, in put_object 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return self._retry(reset_func, put_object, container, obj, contents, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1843, in _retry 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi reset_func(func, *args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1940, in _default_reset 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise ClientException('put_object(%r, %r, ...) failure and no ' 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi swiftclient.exceptions.ClientException: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi During handling of the above exception, another exception occurred: 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1347, in __call__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1391, in dispatch 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return method(*args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/utils.py", line 416, in wrapped 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 298, in upload 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self._restore(image_repo, image) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise() 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise value 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 163, in upload 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/domain/proxy.py", line 208, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 501, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise() 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise value 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 447, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/policy.py", line 198, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/quota/__init__.py", line 318, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 567, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 458, in _upload_to_store 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 398, in add_with_multihash 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return store_add_to_backend_with_multihash( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 480, in store_add_to_backend_with_multihash 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/driver.py", line 279, in add_adapter 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/capabilities.py", line 176, in op_checker 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1082, in add 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise glance_store.BackendException(msg) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi glance_store.exceptions.BackendException: Failed to add object to Swift. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload.. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Mar 17 08:35:16 infra1-glance-container-99614ac2 uwsgi[85]: Wed Mar 17 08:35:16 2021 - uwsgi_response_writev_headers_and_body_do(): Connection reset by peer [core/writer.c line 306] during PUT /v2/images/1b9af05a-f7c9-4315-9354-e09f1df66321/file (192.168.110.215) Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.441 85 CRITICAL glance [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unhandled error: OSError: write error 2021-03-17 08:35:16.441 85 ERROR glance OSError: write error 2021-03-17 08:35:16.441 85 ERROR glance It seems as if the problems occur more often with large instances, i.e. there are fewer problems with new Ubuntu 20.04 instances but after an 'apt-upgrade' on the instance, the error occurs every time a snapshot is taken. Any help is much appreciated! Kind regards, Oliver

2 1