September 2021 - openstack-discuss

Quay.io availability in China
by Radosław Piliszek 10 Sep '21

10 Sep '21

Dears, I have a question (mostly to our readers from China) whether https://quay.io/ is freely accessible from there. Asking because Kolla is planning to switch from Docker Hub (which has some local mirror in China as far as we know) to Quay.io for public images. -yoctozepto

3 3

Horizon connection errors from object store
by Michel Niyoyita 10 Sep '21

10 Sep '21

Hello team , I am facing an issue when I am trying to connect to the object store containers on the horizon dashboad . Once click on containers it automatically disconnect. please find below logs I am getting and help for further analysis. [Thu Sep 09 06:35:22.185771 2021] [wsgi:error] [pid 167:tid 139887608641280] [remote 10.10.29.150:55130] Attempted scope to domain Default failed, will attempt to scope to another domain. [Thu Sep 09 06:35:22.572522 2021] [wsgi:error] [pid 167:tid 139887608641280] [remote 10.10.29.150:55130] Login successful for user "admin" using domain "Default", remote address 10.10.29.150. [Thu Sep 09 06:35:51.494815 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" [Thu Sep 09 06:35:51.495140 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 Unauthorized [Thu Sep 09 06:35:51.495541 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: {'Content-Length': '119', 'X-Trans-Id': 'tx00000000000000000000f-006139ab44-9fc1a-default', 'X-Openstack-Request-Id': 'tx00000000000000000000f-006139ab44-9fc1a-default', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Thu, 09 Sep 2021 06:35:51 GMT', 'Connection': 'Keep-Alive'} [Thu Sep 09 06:35:51.495792 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP BODY: b'{"Code":"AccessDenied","RequestId":"tx00000000000000000000f-006139ab44-9fc1a-default","HostId":"9fc1a-default-default"}' [Thu Sep 09 06:35:51.498743 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] Unauthorized: /api/swift/containers/ [Thu Sep 09 06:35:52.924169 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" [Thu Sep 09 06:35:52.924520 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 Unauthorized [Thu Sep 09 06:35:52.924789 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: {'Content-Length': '119', 'X-Trans-Id': 'tx000000000000000000010-006139ab48-9fc1a-default', 'X-Openstack-Request-Id': 'tx000000000000000000010-006139ab48-9fc1a-default', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Thu, 09 Sep 2021 06:35:52 GMT', 'Connection': 'Keep-Alive'} [Thu Sep 09 06:35:52.925034 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP BODY: b'{"Code":"AccessDenied","RequestId":"tx000000000000000000010-006139ab48-9fc1a-default","HostId":"9fc1a-default-default"}' [Thu Sep 09 06:35:52.929398 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] Unauthorized: /api/swift/containers/ [Thu Sep 09 06:35:52.935799 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:56016] Logging out user "admin". [Thu Sep 09 06:35:53.061489 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] Logging out user "". [Thu Sep 09 06:35:54.541593 2021] [wsgi:error] [pid 165:tid 139887608641280] [remote 10.10.29.150:55852] The request's session was deleted before the request completed. The user may have logged out in a concurrent request, for example. [Thu Sep 09 06:35:54.542896 2021] [wsgi:error] [pid 165:tid 139887608641280] [remote 10.10.29.150:55852] Bad Request: /api/swift/policies/ [Thu Sep 09 06:35:54.566055 2021] [wsgi:error] [pid 167:tid 139887608641280] [remote 10.10.29.150:55860] The request's session was deleted before the request completed. The user may have logged out in a concurrent request, for example. [Thu Sep 09 06:35:54.567130 2021] [wsgi:error] [pid 167:tid 139887608641280] [remote 10.10.29.150:55860] Bad Request: /api/swift/info/ (kolla-open1) stack@kolla-open1 :/var/lib/docker/volumes/kolla_logs/_data/horizon$ Michel

2 6

Cinder API timeout on single-control node
by Eugen Block 10 Sep '21

10 Sep '21

Hi *, since your last responses were quite helpful regarding rabbitmq I would like to ask a different question for the same environment. It's an older openstack version (Pike) running with only one control node. There already were lots of volumes (way more than 1000) in that cloud, but after adding a bunch more (not sure how many exactly) in one project the whole cinder api became extremely slow. Both horizon and CLI run into timeouts: [Wed Sep 01 13:18:52.109178 2021] [wsgi:error] [pid 60440] [client <IP>:58474] Timeout when reading response headers from daemon process 'horizon': /srv/www/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi, referer: http://<control>/project/volumes/ [Wed Sep 01 13:18:53.664714 2021] [wsgi:error] [pid 13007] Not Found: /favicon.ico Sometimes the volume creation succeeds if you just retry, but it often fails. The dashboard shows a "504 gateway timeout" after two minutes (also after four minutes after I increased the timeout for the apache dashboard config). The timeout also shows even if I try to get into the volumes tab of an empty project. A couple of weeks ago I already noticed some performance issues with cinder api if there are lots of attached volumes, if there are many "available" volumes it doesn't seem to slow things down. But since then the total number of volumes has doubled. At the moment there are more than 960 attached volumes across all projects and more than 750 detached volumes. I searched the cinder.conf for any helpful setting but I'm not sure which would actually help. And since it's a production cloud I would like to avoid restarting services all the time just to try something. Maybe some of you can point me in the right direction? I would appreciate any help! If there's more information I can provide just let me know. Thanks! Eugen

2 2

[tc][operators][all] Querying operators about their use of TC tags
by Radosław Piliszek 10 Sep '21

10 Sep '21

Dear OpenStack Operators, As part of Technical Committee's (TC's) efforts for this cycle (Xena) [1] we are working on an analysis of the usefulness of the Tags framework [2]. To this end, a summarising spreadsheet with tag distribution and applicable notes was started by me. [3] As the Tags framework is aimed at operators, we (TC) have a question to you (operators) whether you are aware of the existence of the Tags framework and whether it is something that you actually use to make deployment decisions. Please reply to this mail if you use the Tags framework and let us know how. [1] https://etherpad.opendev.org/p/tc-xena-tracker [2] https://governance.openstack.org/tc/reference/tags/index.html [3] https://docs.google.com/spreadsheets/d/18GXibtdQnSkIwA7DsBvX9dPwbw9JH76AhhR… Kind regards, Radosław Piliszek (yoctozepto) OpenStack TC vice-chair

1 0

[cinder] review patches with FFE first
by Brian Rosmaita 10 Sep '21

10 Sep '21

For anyone who missed yesterday's cinder meeting, we are conveniently tracking the patches that have FFEs on this etherpad: https://etherpad.opendev.org/p/cinder-xena-ffe Please give them the highest review priority. cheers, brian

1 0

[all][elections][ptl] Project Team Lead Election Conclusion and Results
by Ian Y. Choi 09 Sep '21

09 Sep '21

Thank you to the electorate, to all those who voted and to all candidates who put their name forward for Project Team Lead (PTL) in this election. A healthy, open process breeds trust in our decision making capability thank you to all those who make this process possible. Now for the results of the PTL election process, please join me in extending congratulations to the following PTLs: * Barbican : Douglas Mendizábal * Blazar : Pierre Riteau * Cinder : Brian Rosmaita * Cloudkitty : Rafael Weingartner * Cyborg : Bailin Zhang * Designate : Michael Johnson * Ec2 Api : Andrey Pavlov * Freezer : ge cong * Glance : Abhishek Kekane * Horizon : Vishal Manchanda * Ironic : Iury Gregory Melo Ferreira * Kolla : Michal Nasiadka * Kuryr : Maysa de Macedo Souza * Magnum : Feilong Wang * Manila : Goutham Pacha Ravi * Masakari : Radosław Piliszek * Murano : Rong Zhu * Neutron : Lajos Katona * Nova : Sylvain Bauza * Octavia : Gregory Thiemonge * OpenStack Charms : Alex Kavanagh * Openstack Chef : Lance Albertson * OpenStack Helm : Gage Hugo * OpenStackAnsible : Dmitriy Rabotyagov * OpenStackSDK : Artem Goncharov * Quality Assurance : Martin Kopec * Rally : Andrey Kurilin * Release Management : Elod Illes * Requirements : Matthew Thode * Senlin : XueFeng Liu * Solum : Rong Zhu * Storlets : Takashi Kajinami * Swift : Tim Burke * Tacker : Yasufumi Ogawa * Telemetry : Matthias Runge * Tripleo : James Slagle * Trove : Lingxian Kong * Vitrage : Eyal Bar-Ilan * Watcher : chen ke * Winstackers : Lucian Petrut * Zaqar : wang hao Elections: * Cyborg: https://civs1.civs.us/cgi-bin/results.pl?id=E_807146df7d6a1d3b Election process details and results are also available here: https://governance.openstack.org/election/ Thank you to all involved in the PTL election process, Ian Y. Choi, on behalf of the OpenStack Technical Election Officials

3 2

[all] Dangerous trend toward too many release-independent components
by Thomas Goirand 09 Sep '21

09 Sep '21

Hi, As I'm packaging Xena in Debian, I've seen more and more components switching to release-independent. This poses a number of problems which I'm really unsure how to address. Let me explain. First of all, once an OpenStack release reaches a distribution stable release, there will be no update of it (ie: no new upstream release), unless there's a security problems, or important bugfixes that cannot be backported individually. At least, that's how Debian works, and why we call our suite "stable" (ie: it doesn't change). However, if a component is declared release-independent, the OpenStack community expects it to be updated for all downward releases. But that's not what's going on in reality. Also, if there's a security fix, we're supposed to get this through an update of a new version, which may include changes not suitable for a distribution update. The danger is that the update may be refused by the Debian Stable release team, for very good reasons (which I agree on). I'm also questioning myself: since we have pinned global requirements, will these new updates be included in the tests of older OpenStack releases? Is this really the case right now? So, while I do understand why it's nicer to set release-independent tags on some components, it is my opinion that this doesn't help stability and poses numerous problems to downstream distributions. Could we limit the number of such release-independent released components to the strict minimum low-profile (ie: with the least amount of code) components, rather than constantly increasing the list? Your thoughts? Cheers, Thomas Goirand (zigo)

4 12

[OSSA-2021-006] Neutron: Routes middleware memory leak for nonexistent controllers (CVE-2021-40797)
by Jeremy Stanley 09 Sep '21

09 Sep '21

======================================================================== OSSA-2021-006: Routes middleware memory leak for nonexistent controllers ======================================================================== :Date: September 09, 2021 :CVE: CVE-2021-40797 Affects ~~~~~~~ - Neutron: <16.4.1, >=17.0.0 <17.2.1, >=18.0.0 <18.1.1 Description ~~~~~~~~~~~ Slawek Kaplonski with Red Hat reported a vulnerability in Neutron's routes middleware. By making API requests involving nonexistent controllers, an authenticated user may cause the API worker to consume increasing amounts of memory, resulting in API performance degradation or denial of service. All Neutron deployments are affected. Patches ~~~~~~~ - https://review.opendev.org/807638 (Queens) - https://review.opendev.org/807637 (Rocky) - https://review.opendev.org/807636 (Stein) - https://review.opendev.org/807635 (Train) - https://review.opendev.org/807634 (Ussuri) - https://review.opendev.org/807633 (Victoria) - https://review.opendev.org/807632 (Wallaby) - https://review.opendev.org/807335 (Xena) Credits ~~~~~~~ - Slawek Kaplonski from Red Hat (CVE-2021-40797) References ~~~~~~~~~~ - https://launchpad.net/bugs/1942179 - http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-40797 Notes ~~~~~ - The stable/train, stable/stein, stable/rocky, and stable/queens branches are under extended maintenance and will receive no new point releases, but patches for them are provided as a courtesy. -- Jeremy Stanley

1 0

[neutron] Drivers meeting agenda - 10.09.2021
by Slawek Kaplonski 09 Sep '21

09 Sep '21

Hi, Agenda for our tomorrow's drivers meeting is at [1]. We have 3 RFEs to discuss: * https://bugs.launchpad.net/neutron/+bug/1830014 - [RFE] add API for neutron debug tool "probe" - there is also spec proposed for that one [2] * https://bugs.launchpad.net/neutron/+bug/1916428 - dibbler tool for dhcpv6 is concluded * https://bugs.launchpad.net/neutron/+bug/1870319 - [RFE] Network cascade deletion API call All of them are pretty old but we need to get back to them and hopefully decide something :) See You all on the meeting tomorrow. [1] https://wiki.openstack.org/wiki/Meetings/NeutronDrivers#Agenda [2] https://review.opendev.org/c/openstack/neutron-specs/+/662541 -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[heat] Issue happend during server_group creation with policies
by ueha.ayumu＠fujitsu.com 09 Sep '21

09 Sep '21

Hi heat team. I reported the following bug report [1] on Sep 2. [1] https://storyboard.openstack.org/#!/story/2009164 Have you confirmed it? This error seems to appear to have occurred after patch [2] was merged. [2] https://review.opendev.org/c/openstack/heat/+/783971 Please kindly check it, thanks. Regards, Ayumu Ueha

1 0