September 2021 - openstack-discuss

[cyborg] RFE requested for Cyborg
by Brin Zhang(张百林) 10 Sep '21

10 Sep '21

Dear Release Team, We experienced strange freezes and timeouts for a long time on our python-cyborgclient need to update the lost feature of device profile 'description' [1] And we have merged this at 7, Sep, that we can update [2]'s hash then to publish it. [1] https://review.opendev.org/c/openstack/python-cyborgclient/+/803886 [2] https://review.opendev.org/c/openstack/releases/+/807624 brinzhang

3 2

[all]Introduction to venus which is the project of log management and has been contributed to the OpenStack community
by Liye Pang(逄立业) 10 Sep '21

10 Sep '21

Hello everyone, after feedback from a large number of operations and maintenance personnel in InCloud OpenStack, we developed the log management project “Venus” for the OpenStack projects and that has contributed to the OpenStack community. The following is an introduction to “Venus”. If there is interest in the community, we are interested in proposing it to become an official OpenStack project in the future. Background In the day-to-day operation and maintenance of large-scale cloud platform, the following problems are encountered： l Time-consuming for log querying while the server increasing to thousands. l Difficult to retrieve logs, since there are many modules in the platform, e.g. systems service, compute, storage, network and other platform services. l The large amount and dispersion of log make faults are difficult to be discovered. l Because of distributed and interaction between components of the cloud platform, and scattered logs between components, it will take more time to locate problems. About Venus According to the key requirements of OpenStack in log storage, retrieval, analysis and so on, we introduced Venus project, a unified log management module. This module can provide a one-stop solution to log collection, cleaning, indexing, analysis, alarm, visualization, report generation and other needs, which involves helping operator or maintainer to quickly solve retrieve problems, grasp the operational health of the platform, and improve the management capabilities of the cloud platform. Additionally, this module plans to use machine learning algorithms to quickly locate IT failures and root causes, and improve operation and maintenance efficiency. Application scenario Venus played a key role in the following scenarios： l Retrieval: Provide a simple and easy-to-use way to retrieve all log and the context. l Analysis: Realize log association, field value statistics, and provide multi-scene and multi-dimensional visual analysis reports. l Alerts：Convert retrieval into active alerts to realize the error finding in massive logs. l Issue location: Establish a chain relationship and knowledge graphs to quickly locate problems. Overall structure The architecture of log management system based on Venus and elastic search is as follows: Diagram 0: Architecture of Venus venus_api: API module，provide API、rest-api service. venus_manager: Internal timing task module to realize the core functions of the log system. Current progress The current progress of the Venus project is as follows: l Collection：Develop fluentd collection tasks based on collectd to read, filter, format and send plug-ins for OpenStack, operating systems, and platform services, etc. l Index：Dealing with multi-dimensional index data in elasticsearch, and provide more concise and comprehensive authentication interface to return query results. l Analysis：Analyzing and display the related module errors, Mariadb connection errors, and Rabbitmq connection errors. l Alerts：Develop alarm task code to set threshold for the number of error logs of different modules at different times, and provides alarm services and notification services. l Location：Develop the call chain analysis function based on global_requested series, which can show the execution sequence, time and error information, etc., and provide the export operation. l Management：Develop configuration management functions in the log system, such as alarm threshold setting, timing task management, and log saving time setting, etc. Application examples Two examples of Venus application scenarios are as follows. 1. The virtual machine creation operation was performed on the cloud platform and it was found that the virtual machine was not created successfully. First, we can find the request id of the operation and jump to the virtual machine creation call chain page. Then, we can query the calling process, view and download the details of the log of the call. 2. In the cloud platform, the error log of each module can be converted into alarms to remind the users. Further, we can retrieve the details of the error log and error log statistics. Next step The next step of the Venus project is as follows: * Collection：In addition to fluent, other collection plugins such as logstash will be integrated. * Analysis: Explore more operation and maintenance scenarios, and conduct statistical analysis and alarm on key data. * display: The configuration, analysis and alarm of Venus will be integrated into horizon in the form of plugin. * location: Form clustering log and construct knowledge map, and integrate algorithm class library to locate the root cause of the fault. Venus Project Registry Venus library: https://opendev.org/inspur/venus You can grab the source code using the following git command: git clone https://opendev.org/inspur/venus.git Venus Demo Youtu.be: <https://youtu.be/mE2MoEx3awM> https://youtu.be/mE2MoEx3awM

7 15

Quay.io availability in China
by Radosław Piliszek 10 Sep '21

10 Sep '21

Dears, I have a question (mostly to our readers from China) whether https://quay.io/ is freely accessible from there. Asking because Kolla is planning to switch from Docker Hub (which has some local mirror in China as far as we know) to Quay.io for public images. -yoctozepto

3 3

Horizon connection errors from object store
by Michel Niyoyita 10 Sep '21

10 Sep '21

Hello team , I am facing an issue when I am trying to connect to the object store containers on the horizon dashboad . Once click on containers it automatically disconnect. please find below logs I am getting and help for further analysis. [Thu Sep 09 06:35:22.185771 2021] [wsgi:error] [pid 167:tid 139887608641280] [remote 10.10.29.150:55130] Attempted scope to domain Default failed, will attempt to scope to another domain. [Thu Sep 09 06:35:22.572522 2021] [wsgi:error] [pid 167:tid 139887608641280] [remote 10.10.29.150:55130] Login successful for user "admin" using domain "Default", remote address 10.10.29.150. [Thu Sep 09 06:35:51.494815 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" [Thu Sep 09 06:35:51.495140 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 Unauthorized [Thu Sep 09 06:35:51.495541 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: {'Content-Length': '119', 'X-Trans-Id': 'tx00000000000000000000f-006139ab44-9fc1a-default', 'X-Openstack-Request-Id': 'tx00000000000000000000f-006139ab44-9fc1a-default', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Thu, 09 Sep 2021 06:35:51 GMT', 'Connection': 'Keep-Alive'} [Thu Sep 09 06:35:51.495792 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP BODY: b'{"Code":"AccessDenied","RequestId":"tx00000000000000000000f-006139ab44-9fc1a-default","HostId":"9fc1a-default-default"}' [Thu Sep 09 06:35:51.498743 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] Unauthorized: /api/swift/containers/ [Thu Sep 09 06:35:52.924169 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" [Thu Sep 09 06:35:52.924520 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 Unauthorized [Thu Sep 09 06:35:52.924789 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: {'Content-Length': '119', 'X-Trans-Id': 'tx000000000000000000010-006139ab48-9fc1a-default', 'X-Openstack-Request-Id': 'tx000000000000000000010-006139ab48-9fc1a-default', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Thu, 09 Sep 2021 06:35:52 GMT', 'Connection': 'Keep-Alive'} [Thu Sep 09 06:35:52.925034 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP BODY: b'{"Code":"AccessDenied","RequestId":"tx000000000000000000010-006139ab48-9fc1a-default","HostId":"9fc1a-default-default"}' [Thu Sep 09 06:35:52.929398 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] Unauthorized: /api/swift/containers/ [Thu Sep 09 06:35:52.935799 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:56016] Logging out user "admin". [Thu Sep 09 06:35:53.061489 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] Logging out user "". [Thu Sep 09 06:35:54.541593 2021] [wsgi:error] [pid 165:tid 139887608641280] [remote 10.10.29.150:55852] The request's session was deleted before the request completed. The user may have logged out in a concurrent request, for example. [Thu Sep 09 06:35:54.542896 2021] [wsgi:error] [pid 165:tid 139887608641280] [remote 10.10.29.150:55852] Bad Request: /api/swift/policies/ [Thu Sep 09 06:35:54.566055 2021] [wsgi:error] [pid 167:tid 139887608641280] [remote 10.10.29.150:55860] The request's session was deleted before the request completed. The user may have logged out in a concurrent request, for example. [Thu Sep 09 06:35:54.567130 2021] [wsgi:error] [pid 167:tid 139887608641280] [remote 10.10.29.150:55860] Bad Request: /api/swift/info/ (kolla-open1) stack@kolla-open1 :/var/lib/docker/volumes/kolla_logs/_data/horizon$ Michel

2 6

Cinder API timeout on single-control node
by Eugen Block 10 Sep '21

10 Sep '21

Hi *, since your last responses were quite helpful regarding rabbitmq I would like to ask a different question for the same environment. It's an older openstack version (Pike) running with only one control node. There already were lots of volumes (way more than 1000) in that cloud, but after adding a bunch more (not sure how many exactly) in one project the whole cinder api became extremely slow. Both horizon and CLI run into timeouts: [Wed Sep 01 13:18:52.109178 2021] [wsgi:error] [pid 60440] [client <IP>:58474] Timeout when reading response headers from daemon process 'horizon': /srv/www/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi, referer: http://<control>/project/volumes/ [Wed Sep 01 13:18:53.664714 2021] [wsgi:error] [pid 13007] Not Found: /favicon.ico Sometimes the volume creation succeeds if you just retry, but it often fails. The dashboard shows a "504 gateway timeout" after two minutes (also after four minutes after I increased the timeout for the apache dashboard config). The timeout also shows even if I try to get into the volumes tab of an empty project. A couple of weeks ago I already noticed some performance issues with cinder api if there are lots of attached volumes, if there are many "available" volumes it doesn't seem to slow things down. But since then the total number of volumes has doubled. At the moment there are more than 960 attached volumes across all projects and more than 750 detached volumes. I searched the cinder.conf for any helpful setting but I'm not sure which would actually help. And since it's a production cloud I would like to avoid restarting services all the time just to try something. Maybe some of you can point me in the right direction? I would appreciate any help! If there's more information I can provide just let me know. Thanks! Eugen

2 2

[tc][operators][all] Querying operators about their use of TC tags
by Radosław Piliszek 10 Sep '21

10 Sep '21

Dear OpenStack Operators, As part of Technical Committee's (TC's) efforts for this cycle (Xena) [1] we are working on an analysis of the usefulness of the Tags framework [2]. To this end, a summarising spreadsheet with tag distribution and applicable notes was started by me. [3] As the Tags framework is aimed at operators, we (TC) have a question to you (operators) whether you are aware of the existence of the Tags framework and whether it is something that you actually use to make deployment decisions. Please reply to this mail if you use the Tags framework and let us know how. [1] https://etherpad.opendev.org/p/tc-xena-tracker [2] https://governance.openstack.org/tc/reference/tags/index.html [3] https://docs.google.com/spreadsheets/d/18GXibtdQnSkIwA7DsBvX9dPwbw9JH76AhhR… Kind regards, Radosław Piliszek (yoctozepto) OpenStack TC vice-chair

1 0

[cinder] review patches with FFE first
by Brian Rosmaita 09 Sep '21

09 Sep '21

For anyone who missed yesterday's cinder meeting, we are conveniently tracking the patches that have FFEs on this etherpad: https://etherpad.opendev.org/p/cinder-xena-ffe Please give them the highest review priority. cheers, brian

1 0

[all][elections][ptl] Project Team Lead Election Conclusion and Results
by Ian Y. Choi 09 Sep '21

09 Sep '21

Thank you to the electorate, to all those who voted and to all candidates who put their name forward for Project Team Lead (PTL) in this election. A healthy, open process breeds trust in our decision making capability thank you to all those who make this process possible. Now for the results of the PTL election process, please join me in extending congratulations to the following PTLs: * Barbican : Douglas Mendizábal * Blazar : Pierre Riteau * Cinder : Brian Rosmaita * Cloudkitty : Rafael Weingartner * Cyborg : Bailin Zhang * Designate : Michael Johnson * Ec2 Api : Andrey Pavlov * Freezer : ge cong * Glance : Abhishek Kekane * Horizon : Vishal Manchanda * Ironic : Iury Gregory Melo Ferreira * Kolla : Michal Nasiadka * Kuryr : Maysa de Macedo Souza * Magnum : Feilong Wang * Manila : Goutham Pacha Ravi * Masakari : Radosław Piliszek * Murano : Rong Zhu * Neutron : Lajos Katona * Nova : Sylvain Bauza * Octavia : Gregory Thiemonge * OpenStack Charms : Alex Kavanagh * Openstack Chef : Lance Albertson * OpenStack Helm : Gage Hugo * OpenStackAnsible : Dmitriy Rabotyagov * OpenStackSDK : Artem Goncharov * Quality Assurance : Martin Kopec * Rally : Andrey Kurilin * Release Management : Elod Illes * Requirements : Matthew Thode * Senlin : XueFeng Liu * Solum : Rong Zhu * Storlets : Takashi Kajinami * Swift : Tim Burke * Tacker : Yasufumi Ogawa * Telemetry : Matthias Runge * Tripleo : James Slagle * Trove : Lingxian Kong * Vitrage : Eyal Bar-Ilan * Watcher : chen ke * Winstackers : Lucian Petrut * Zaqar : wang hao Elections: * Cyborg: https://civs1.civs.us/cgi-bin/results.pl?id=E_807146df7d6a1d3b Election process details and results are also available here: https://governance.openstack.org/election/ Thank you to all involved in the PTL election process, Ian Y. Choi, on behalf of the OpenStack Technical Election Officials

3 2

[all] Dangerous trend toward too many release-independent components
by Thomas Goirand 09 Sep '21

09 Sep '21

Hi, As I'm packaging Xena in Debian, I've seen more and more components switching to release-independent. This poses a number of problems which I'm really unsure how to address. Let me explain. First of all, once an OpenStack release reaches a distribution stable release, there will be no update of it (ie: no new upstream release), unless there's a security problems, or important bugfixes that cannot be backported individually. At least, that's how Debian works, and why we call our suite "stable" (ie: it doesn't change). However, if a component is declared release-independent, the OpenStack community expects it to be updated for all downward releases. But that's not what's going on in reality. Also, if there's a security fix, we're supposed to get this through an update of a new version, which may include changes not suitable for a distribution update. The danger is that the update may be refused by the Debian Stable release team, for very good reasons (which I agree on). I'm also questioning myself: since we have pinned global requirements, will these new updates be included in the tests of older OpenStack releases? Is this really the case right now? So, while I do understand why it's nicer to set release-independent tags on some components, it is my opinion that this doesn't help stability and poses numerous problems to downstream distributions. Could we limit the number of such release-independent released components to the strict minimum low-profile (ie: with the least amount of code) components, rather than constantly increasing the list? Your thoughts? Cheers, Thomas Goirand (zigo)

4 12

[OSSA-2021-006] Neutron: Routes middleware memory leak for nonexistent controllers (CVE-2021-40797)
by Jeremy Stanley 09 Sep '21

09 Sep '21

======================================================================== OSSA-2021-006: Routes middleware memory leak for nonexistent controllers ======================================================================== :Date: September 09, 2021 :CVE: CVE-2021-40797 Affects ~~~~~~~ - Neutron: <16.4.1, >=17.0.0 <17.2.1, >=18.0.0 <18.1.1 Description ~~~~~~~~~~~ Slawek Kaplonski with Red Hat reported a vulnerability in Neutron's routes middleware. By making API requests involving nonexistent controllers, an authenticated user may cause the API worker to consume increasing amounts of memory, resulting in API performance degradation or denial of service. All Neutron deployments are affected. Patches ~~~~~~~ - https://review.opendev.org/807638 (Queens) - https://review.opendev.org/807637 (Rocky) - https://review.opendev.org/807636 (Stein) - https://review.opendev.org/807635 (Train) - https://review.opendev.org/807634 (Ussuri) - https://review.opendev.org/807633 (Victoria) - https://review.opendev.org/807632 (Wallaby) - https://review.opendev.org/807335 (Xena) Credits ~~~~~~~ - Slawek Kaplonski from Red Hat (CVE-2021-40797) References ~~~~~~~~~~ - https://launchpad.net/bugs/1942179 - http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-40797 Notes ~~~~~ - The stable/train, stable/stein, stable/rocky, and stable/queens branches are under extended maintenance and will receive no new point releases, but patches for them are provided as a courtesy. -- Jeremy Stanley

1 0

[neutron] Drivers meeting agenda - 10.09.2021
by Slawek Kaplonski 09 Sep '21

09 Sep '21

Hi, Agenda for our tomorrow's drivers meeting is at [1]. We have 3 RFEs to discuss: * https://bugs.launchpad.net/neutron/+bug/1830014 - [RFE] add API for neutron debug tool "probe" - there is also spec proposed for that one [2] * https://bugs.launchpad.net/neutron/+bug/1916428 - dibbler tool for dhcpv6 is concluded * https://bugs.launchpad.net/neutron/+bug/1870319 - [RFE] Network cascade deletion API call All of them are pretty old but we need to get back to them and hopefully decide something :) See You all on the meeting tomorrow. [1] https://wiki.openstack.org/wiki/Meetings/NeutronDrivers#Agenda [2] https://review.opendev.org/c/openstack/neutron-specs/+/662541 -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[heat] Issue happend during server_group creation with policies
by ueha.ayumu＠fujitsu.com 09 Sep '21

09 Sep '21

Hi heat team. I reported the following bug report [1] on Sep 2. [1] https://storyboard.openstack.org/#!/story/2009164 Have you confirmed it? This error seems to appear to have occurred after patch [2] was merged. [2] https://review.opendev.org/c/openstack/heat/+/783971 Please kindly check it, thanks. Regards, Ayumu Ueha

1 0

Horizon connection errors from object store
by Michel Niyoyita 09 Sep '21

09 Sep '21

Hello team , I am facing an issue when I am trying to connect to the object store containers on the horizon dashboad . Once click on containers it automatically disconnect. please find below logs I am getting and help for further analysis. [Thu Sep 09 06:35:22.185771 2021] [wsgi:error] [pid 167:tid 139887608641280] [remote 10.10.29.150:55130] Attempted scope to domain Default failed, will attempt to scope to another domain. [Thu Sep 09 06:35:22.572522 2021] [wsgi:error] [pid 167:tid 139887608641280] [remote 10.10.29.150:55130] Login successful for user "admin" using domain "Default", remote address 10.10.29.150. [Thu Sep 09 06:35:51.494815 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" [Thu Sep 09 06:35:51.495140 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 Unauthorized [Thu Sep 09 06:35:51.495541 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: {'Content-Length': '119', 'X-Trans-Id': 'tx00000000000000000000f-006139ab44-9fc1a-default', 'X-Openstack-Request-Id': 'tx00000000000000000000f-006139ab44-9fc1a-default', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Thu, 09 Sep 2021 06:35:51 GMT', 'Connection': 'Keep-Alive'} [Thu Sep 09 06:35:51.495792 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP BODY: b'{"Code":"AccessDenied","RequestId":"tx00000000000000000000f-006139ab44-9fc1a-default","HostId":"9fc1a-default-default"}' [Thu Sep 09 06:35:51.498743 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] Unauthorized: /api/swift/containers/ [Thu Sep 09 06:35:52.924169 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" [Thu Sep 09 06:35:52.924520 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 Unauthorized [Thu Sep 09 06:35:52.924789 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: {'Content-Length': '119', 'X-Trans-Id': 'tx000000000000000000010-006139ab48-9fc1a-default', 'X-Openstack-Request-Id': 'tx000000000000000000010-006139ab48-9fc1a-default', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Thu, 09 Sep 2021 06:35:52 GMT', 'Connection': 'Keep-Alive'} [Thu Sep 09 06:35:52.925034 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP BODY: b'{"Code":"AccessDenied","RequestId":"tx000000000000000000010-006139ab48-9fc1a-default","HostId":"9fc1a-default-default"}' [Thu Sep 09 06:35:52.929398 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] Unauthorized: /api/swift/containers/ [Thu Sep 09 06:35:52.935799 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:56016] Logging out user "admin". [Thu Sep 09 06:35:53.061489 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] Logging out user "". [Thu Sep 09 06:35:54.541593 2021] [wsgi:error] [pid 165:tid 139887608641280] [remote 10.10.29.150:55852] The request's session was deleted before the request completed. The user may have logged out in a concurrent request, for example. [Thu Sep 09 06:35:54.542896 2021] [wsgi:error] [pid 165:tid 139887608641280] [remote 10.10.29.150:55852] Bad Request: /api/swift/policies/ [Thu Sep 09 06:35:54.566055 2021] [wsgi:error] [pid 167:tid 139887608641280] [remote 10.10.29.150:55860] The request's session was deleted before the request completed. The user may have logged out in a concurrent request, for example. [Thu Sep 09 06:35:54.567130 2021] [wsgi:error] [pid 167:tid 139887608641280] [remote 10.10.29.150:55860] Bad Request: /api/swift/info/ (kolla-open1) stack@kolla-open1 :/var/lib/docker/volumes/kolla_logs/_data/horizon$ [image: image.png] [image: image.png] Michel

1 0

[neutron][neutron-vpnaas] Propose to EOL stable/queens, stable/rocky and stable/stein branches or neutron-vpnaas
by Slawek Kaplonski 09 Sep '21

09 Sep '21

Hi, On our last meeting of the CI team we discussed the problem with broken stable branches (stein and older) in the neutron-vpnaas project [1]. I checked that last patches which were merged in those branches are more than year ago: * Queens - last merged patch in April 2019 * Rocky - last merged patch in February 2020 * Stein - last merged patch in September 2019 Giving the above and the current status of the CI in those branches, I propose to make them End Of Life. I will wait until end of next week for anyone who would like to maybe step up as maintainer of those branches and who would at least try to fix CI of them. If there will be no volunteers for that, I will EOL those branches in neutron- vpnaas. [1] https://meetings.opendev.org/meetings/neutron_ci/2021/neutron_ci. 2021-09-07-15.00.log.html#l-64 -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

Question about vDPA support for live migration
by Jeffrey Zhang 09 Sep '21

09 Sep '21

Hey everyone, I am testing vDPA on OpenStack Wallaby. I have set it up successfully. But I found there are several features that are not implemented, including live/code migration[0], which I wanted. Anyone know what's the latest progress about this? I found no talk about this. [0] https://review.opendev.org/c/openstack/nova/+/780866/3/releasenotes/notes/v… <https://signaldomn.online/click?redirect=https%3A%2F%2Freview.opendev.org%2…> --- Regards, Jeffrey Zhang Blog: http://xcodest.me <https://signaldomn.online/click?redirect=http%3A%2F%2Fxcodest.me%2F&dID=163…>

1 0

[neutron] Slow router provisioning during full resync of L3 Agent
by Patryk Jakuszew 09 Sep '21

09 Sep '21

Hello, I have a long-standing issue with L3 Agent which I would like to finally solve - *very* slow router provisioning in L3 Agent. We are operating a Rocky-based OpenStack deployment with three bare-metal L3 Agent nodes running in legacy mode. After restarting the L3 node, it takes a really long time for the L3 agent to become fully operational. There are two parts of resync which take much time: getting a list of routers from neutron-server and actually recreate them in the L3 node. While the long running time of router list retrieval is somewhat understandable, the router provisioning process itself proves to be very troublesome in our operations tasks. In our production deployment with around 250 routers, it takes around 2 hours (!) to recreate the router namespaces and have the L3 node fully functional again. Two hours of router re-provisioning is actually an optimistic scenario, this proved to be much longer during the outages we encountered (sometimes the sync took nearly 6-8 hours). This effectively prolongs any maintenance upgrades, configuration changes and OpenStack release upgrades. Another thing is, on that same production environment the first 100 routers usually get provisioned fast (around 30 minutes), after that it slows down with each router - this kind of non deterministic behavior makes it hard to communicate the maintenance finish ETA for our users. We also have a test environment with Stein already installed, where this problem is also present - full resync of 150 routers, with only one external gateway ports, takes around an hour to complete. Are there any operators here who also encountered that issue? Does anyone have any experience with similar situation and are willing to share their observations and optimizations? -- Regards, Patryk Jakuszew

3 3

[all][tc] Technical Committee next weekly meeting on Sept 9th at 1500 UTC
by Ghanshyam Mann 08 Sep '21

08 Sep '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for Sept 9th at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Sept 8th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

[devstack][openstack-qa] DevStack on VirtualBox with Host-only Adapter
by os-user157 08 Sep '21

08 Sep '21

Hi all, I'm trying to deploy DevStack inside an Ubuntu Server 20.04.3 LTS virtual machine inside VirtualBox, but I'm struggling a lot, I got a lot of different errors. My use case: I want to test Zuul and DevStack, so I'll create a virtual machine for DevStack and one for Zuul, both on VirtualBox. I'm using a Host-only adapter with CIDR 192.168.200.0/24, and I need DevStack to be accessible on this network, and to use this network as the floating range for giving floating IP addresses to instances inside DevStack, so that Zuul can also access them. My local.conf: https://paste.openstack.org/show/809182/ Screenshots of the network configuration of my VirtualBox VM: First adapter, NAT (used for internet connectivity): https://i.imgur.com/C7XTiLJ.png Second adapter, host-only, used for communication with my Windows host and other VMs in the same network: https://i.imgur.com/Qeciuqx.png DevStack is erroring with the arping command on lib/neutron-legacy, arping gets 0 responses and exits the script with exit code 1 Does anyone have experience in dpeloying DevStack inside VirtualBox in such manner? I really need some help with this, if anyone could give me some ideas of waht could be wrong, it'd very much appreciated! Thanks.

1 0

Tenant vlan Network Issue
by Karera Tony 08 Sep '21

08 Sep '21

Hello Team, Whenever I create an internal network with type vlan. The Instances don't get IPs but for external Network it's fine. Below is the etc/kolla/config/neutron/ml2_conf.ini file [ml2] type_drivers = flat,vlan,vxlan tenant_network_types = vlan,flat mechanism_drivers = openvswitch,l2population [ml2_type_vlan] network_vlan_ranges = physnet1 [ml2_type_flat] flat_networks = physnet1 Is there something I need to change or I need to configure the interface differently Regards Tony Karera

4 12

[TripleO] Yoga PTG topics etherpad
by James Slagle 08 Sep '21

08 Sep '21

I've started an etherpad where we can collect topics for discussion at the upcoming PTG: https://etherpad.opendev.org/p/tripleo-yoga-topics Please add topics over the next couple of weeks, and then we'll work on a schedule. Thanks. -- -- James Slagle --

1 0

Re: [all][tc] Minium python version for yoga and centos 9 stream.
by Sean Mooney 08 Sep '21

08 Sep '21

resending to correct list On Tue, 2021-09-07 at 10:51 +0100, Sean Mooney wrote: > hi i was jsut looking at the yoga runtimes an notice that we have not really updated it > with regards to minium python versions. > > currently > > the runtime defintion is as follows https://github.com/openstack/governance/blob/master/reference/runtimes/yoga… > > It is the :doc:`policy <../../resolutions/20181024-python-update-process>` that each OpenStack release cycle will target the latest available version of Python; default Python runtimes on the > distributions listed above; and versions used in integration tests at the start of the cycle, at least until the point when all projects have migrated to a later version. > > Based on the criteria above, all Python-based projects must target and test against, at a minimum: > > Python 3.6 (default in CentOS 8 Stream) > Python 3.8 (latest available; default in Ubuntu 20.04) > > looking at to look at https://repology.org/project/python/versions what python version are shpping by default now in many distro > more and more of them are moving to python 3.9 which is currently not in the test list and wer are starting to fall quite far behind > > debian 11/12/stable are currently using 3.9.2 > centos 8 defaults to 3.6.8 but ships 3.8.6 and 3.9.2 as appstreams > centos 9 stream will default to python3-3.9.6-6.el9.x86_64.rpm https://composes.stream.centos.org/test/latest-CentOS-Stream/compose/BaseOS… > ubuntu 20.04 default to 3.8.2 but also ships 3.9.5 in universe > ubuntu 21.04 default to 3.9.5 so we can expect that when yoga ships on ubuntu 22.04 it will be based on at least python 3.9 and proably 3.10 > fedora well they basically ship everythin but i belive 3.9 is not the default based on complaints some of my fellow contiutors had with > runnign unit and functional tests. > > openSUSE Leap 15.3 main/oss python39 3.9.4 ship 3.9 but default to 3.6 > they aslo nolonger have an openstack product but it would still be nice to keep them in mind. > > > over all i think we shoudl consider raising or min tested/supported python version to python 3.8 for yoga and consider updating from centos 8 to centos 9. > i also think it worth including python 3.9 in the required testing versions for python projects. > > several distros will be releaseing openstack xena/wallaby with python 3.9 and also yoga likely with 3.10 in some cases. > if we only offically test with 3.6 and 3.8 we could have some hard probalems to solve going forward. the optional testing with unit and functional tests > that some project have for python 3.9 certenly helps but actully having an integrated tempest gate job that runs on 3.9 i think is going to be increasingly important. > > readring centos 9 stream, it was orginally planned of release at the end of q2. now that date has obviosly passed but i know that they are still hopeful to have > a release ready in the next month or so. cloud images are avaibel at https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/… although the repos are still only > partly complete https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/… and there are currently no active mirros so its still officaly > not released https://www.centos.org/centos-stream/ and i don not have any ETA beyond soonTM. as such im aware we cant offically declare centos 9 stream as part of the tested > runtimes at this point but it might be at least worht listing it as an experimtal runtime as i suspec there will be a desire to swtich to it during the yoga cycle for some proejct. > ooo/rdo come to mind as they discussed this for xena but obviously that was not possibel since it was not released in time. > > > regards > sean. > > ps removing python 3.6 and replacing it with 3.9 instead of just adding 3.9 as an addtional runtime is more to manage our testing > there isnt nessarly any python feature that i explcitly would like to be able to use that are added in 3.8 that would be enabled > by making that the new min python verion that is motivating this suggestion. there are some nice thingin 3.9/3.10 such as pattern > mataching that are commign but it will be some time (2-4 years)before we can actully use those as our min version so i was just being practial > and suggestion we raise to 3.8 and add 3.9 as required with 3.10 as optional/experimal for those that want it. >

5 7

[cinder] Bug deputy report for week of 2021-08-09
by Sofia Enriquez 08 Sep '21

08 Sep '21

This is a bug report from 2021-01-09 to 2021-08-09. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- High - https://bugs.launchpad.net/cinder/+bug/1942696 "RateLimitingMiddleware still belongs to v2 API code" Medium - https://bugs.launchpad.net/cinder/+bug/1942953 "Upload volume to image will terminate the vm's volume attachment". Assigned to xuanyandong - https://bugs.launchpad.net/cinder/+bug/1942342 "Huawei FC driver fails to create a volume from a snapshot with Dorado V6 - Bad or unexpected response from the storage volume backend API: Create luncopy error". Unassigned. - https://bugs.launchpad.net/cinder/+bug/1934036 "Undefined CONF.quota_metadata_items is used in unit tests". Assigned to Takashi Kajinami. Cheers, -- L. Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

Fwd: [ceph-users] Re: Replacing swift with RGW
by Michel Niyoyita 08 Sep '21

08 Sep '21

---------- Forwarded message --------- From: Eugen Block <eblock(a)nde.ag> Date: Thu, Sep 2, 2021 at 10:39 AM Subject: Re: [ceph-users] Re: Replacing swift with RGW To: Michel Niyoyita <micou12(a)gmail.com> You should continue this thread on the openstack-discuss mailing list (openstack-discuss(a)lists.openstack.org) since this is not a ceph issue. I'm not familiar with kolla and I don't know the requirements to install openstack-swift, you'll need to ask the openstack community. Zitat von Michel Niyoyita <micou12(a)gmail.com>: > Below are errors I am getting once I try to run swift commands . the second > one is the error I get once try to install python-swiftclient > > (kolla-open) [stack@kolla-open ~]$ swift -v stat > -bash: swift: command not found > (kolla-open) [stack@kolla-open ~]$ sudo yum -y install python-swiftclient > Last metadata expiration check: 0:59:21 ago on Thu 02 Sep 2021 09:21:53 AM > CAT. > No match for argument: python-swiftclient > Error: Unable to find a match: python-swiftclient > (kolla-open) [stack@kolla-open ~]$ > > Waiting for your help > > On Thu, Sep 2, 2021 at 10:17 AM Eugen Block <eblock(a)nde.ag> wrote: > >> I can't tell for sure, but yes, I believe you need the openstack-swift >> package (with dependencies). What errors do you get? The more >> information you share the better people can help. >> >> >> Zitat von Michel Niyoyita <micou12(a)gmail.com>: >> >> > I tried to install "sudo yum -y install python-swiftclient" on openstack >> > side but fails . are there openastack-shwift packages which are needed? >> > if are there please help me to get . may be also it is the cause I am >> > failing to run swift command on openstack cli side. >> > >> > thank you for your continued support. >> > >> > Micheal >> > >> > On Thu, Sep 2, 2021 at 9:14 AM Eugen Block <eblock(a)nde.ag> wrote: >> > >> >> I only configured the endpoints for the clients to directly access the >> >> RGWs, but you'll probably need to install the openstack-swift package. >> >> Or have you done that already? >> >> >> >> >> >> Zitat von Michel Niyoyita <micou12(a)gmail.com>: >> >> >> >> > Thank you Eugen for your prompt response. >> >> > >> >> > Now the commands provided work. but I am not finding the object >> storage >> >> on >> >> > the horizon dashboard , but it appears in the system information >> >> services. >> >> > [image: image.png] >> >> > so my question is how to configure it in order that it can appear in >> the >> >> > dashboard . >> >> > >> >> > Michel >> >> > >> >> > On Wed, Sep 1, 2021 at 3:49 PM Eugen Block <eblock(a)nde.ag> wrote: >> >> > >> >> >> Sorry, one little detail slipped through, the '--region' flag has to >> >> >> be put before the 'service' name. The correct command would be: >> >> >> >> >> >> openstack endpoint create --region RegionOne swift admin >> >> >> http://ceph-osd3:8080/swift/v1 >> >> >> >> >> >> and respectively for the other interfaces. >> >> >> >> >> >> >> >> >> Zitat von Eugen Block <eblock(a)nde.ag>: >> >> >> >> >> >> > Hi, >> >> >> > >> >> >> > this is not a ceph issue but your openstack cli command as the >> error >> >> >> > message states. >> >> >> > >> >> >> > Try one interface at a time: >> >> >> > >> >> >> > openstack endpoint create swift public >> >> >> > http://ceph-osd3:8080/swift/v1 --region RegionOne swift >> >> >> > openstack endpoint create swift admin >> http://ceph-osd3:8080/swift/v1 >> >> >> > --region RegionOne swift >> >> >> > openstack endpoint create swift internal >> >> >> > http://ceph-osd3:8080/swift/v1 --region RegionOne swift >> >> >> > >> >> >> > >> >> >> > Zitat von Michel Niyoyita <micou12(a)gmail.com>: >> >> >> > >> >> >> >> Hello , >> >> >> >> >> >> >> >> Below are errors I am getting once I am trying to integrate swift >> >> with >> >> >> >> Radosgateway. >> >> >> >> >> >> >> >> From openstack side once i try to endpoint which will point to the >> >> >> >> radosgateway : >> >> >> >> >> >> >> >> (kolla-open) [stack@kolla-open kolla]$ openstack endpoint create >> >> >> >> --publicurl http://ceph-osd3:8080/swift/v1 --adminurl >> >> >> >> http://ceph-osd3:8080/swift/v1 --internal >> >> >> http://ceph-osd3:8080/swift/v1 >> >> >> >> --region RegionOne swift >> >> >> >> usage: openstack endpoint create [-h] [-f >> >> {json,shell,table,value,yaml}] >> >> >> >> [-c COLUMN] [--noindent] [--prefix >> >> >> PREFIX] >> >> >> >> [--max-width <integer>] >> [--fit-width] >> >> >> >> [--print-empty] [--region >> >> <region-id>] >> >> >> >> [--enable | --disable] >> >> >> >> <service> <interface> <url> >> >> >> >> openstack endpoint create: error: argument <interface>: invalid >> >> choice: >> >> >> ' >> >> >> >> http://ceph-osd3:8080/swift/v1' (choose from 'admin', 'public', >> >> >> 'internal') >> >> >> >> >> >> >> >> (kolla-open) [stack@kolla-open kolla]$ >> >> >> >> >> >> >> >> below are my /etc/ceph/ceph.conf file : >> >> >> >> >> >> >> >> [client.rgw.ceph-osd3] >> >> >> >> rgw_dns_name = ceph-osd3 >> >> >> >> host = ceph-osd3 >> >> >> >> keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3/keyring >> >> >> >> log file = /var/log/ceph/ceph-rgw-ceph-osd3.log >> >> >> >> rgw frontends = civetweb port=10.10.29.110:8080 num_threads=100 >> >> >> >> rgw_keystone_url=http://10.10.29.150:35357 >> >> >> >> rgw_keystone_admin_user=admin >> >> >> >> rgw_keystone_admin_password=admin >> >> >> >> rgw_keystone_admin_tenant=admin >> >> >> >> rgw_keystone_accepted_role=admin Member swiftoperator >> >> >> >> rgw_keystone_token_cache_size=200 >> >> >> >> rgw_keystone_revocation_interval=300 >> >> >> >> >> >> >> >> [client.rgw.ceph-osd3.rgw0] >> >> >> >> host = ceph-osd3 >> >> >> >> keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3.rgw0/keyring >> >> >> >> log file = /var/log/ceph/ceph-rgw-ceph-osd3.rgw0.log >> >> >> >> rgw frontends = beast endpoint=10.10.29.110:8080 >> >> >> >> rgw thread pool size = 512 >> >> >> >> >> >> >> >> please note that my rgw_dns_name = ceph_osd3 with 10.10.29.110 as >> IP >> >> >> >> >> >> >> >> and 10.10.29.150 all-in-one IP >> >> >> >> >> >> >> >> >> >> >> >> Please crosscheck where I might make mistake and try to correct. >> >> >> >> >> >> >> >> Best regards >> >> >> >> >> >> >> >> Michel >> >> >> >> >> >> >> >> On Mon, Aug 30, 2021 at 11:25 AM Etienne Menguy < >> >> >> etienne.menguy(a)croit.io> >> >> >> >> wrote: >> >> >> >> >> >> >> >>> Hi, >> >> >> >>> >> >> >> >>> There are some information on Ceph documentation >> >> >> >>> https://docs.ceph.com/en/latest/radosgw/keystone/ < >> >> >> >>> https://docs.ceph.com/en/latest/radosgw/keystone/> . >> >> >> >>> - Use keystone as auth for RGW >> >> >> >>> - Create service and register your RGW as swift >> >> >> >>> >> >> >> >>> Étienne >> >> >> >>> >> >> >> >>>> On 27 Aug 2021, at 15:47, Michel Niyoyita <micou12(a)gmail.com> >> >> wrote: >> >> >> >>>> >> >> >> >>>> Hello , >> >> >> >>>> >> >> >> >>>> I have configured RGW in my ceph cluster deployed using ceph >> >> ansible >> >> >> and >> >> >> >>>> create sub user to access the created containers and would like >> to >> >> >> >>> replace >> >> >> >>>> swift by RGW in the openstack side. Anyone can help on >> >> configuration >> >> >> to >> >> >> >>> be >> >> >> >>>> done in the OpenStack side in order to integrate those >> services. I >> >> >> have >> >> >> >>>> deployed OpenStack wallaby using Kolla-ansible on ubuntu 20.04. >> and >> >> >> ceph >> >> >> >>>> pacific 16.2.5 was deployed using ansible on ubuntu 20.04 >> >> >> >>>> >> >> >> >>>> Kindly help for the configuration or documentation. >> >> >> >>>> >> >> >> >>>> Best Regards >> >> >> >>>> >> >> >> >>>> Michel >> >> >> >>>> _______________________________________________ >> >> >> >>>> ceph-users mailing list -- ceph-users(a)ceph.io >> >> >> >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> >>> >> >> >> >>> _______________________________________________ >> >> >> >>> ceph-users mailing list -- ceph-users(a)ceph.io >> >> >> >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> >>> >> >> >> >> _______________________________________________ >> >> >> >> ceph-users mailing list -- ceph-users(a)ceph.io >> >> >> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> ceph-users mailing list -- ceph-users(a)ceph.io >> >> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>

4 7

[sdk]: Block storage service update volume status from reserved to available
by dmeng 08 Sep '21

08 Sep '21

Hello there, Hope everything is going well. I would like to know if there is any method that could update the volume status or reset its state. While we creating volumes use the block storage service or creating vms with volumes use the compute service, sometimes we got volumes stuck in creating/deleteing/attaching/reserved state. And we couldn't remove them or reset their state without an admin account even we own the project. The way we create volumes and boot vms is like the following: # create the volume cv = cinder.create_volume(name=bv_name, size=size, image_id=imageobj.id, volume_type=volume_type, is_bootable=True) # format mapping data bdm = [{'uuid': cv.id, 'source_type': 'volume', 'destination_type': 'volume', 'delete_on_termination': True, 'boot_index': 0}] # create the vm nova.create_server(name=hostname, image_id=imageobj.id, flavor_id=flavorl.id, key_name=key_name, block_device_mapping=bdm, user_data=format_userdata, networks=netid, security_groups=sec_groups) In the normal situation, everything works. But in the case of any network issue, the volume might stuck in the process of creating, deleting, or attaching. And if the volume got created successfully, when it tries to create the vm, the volume will be reserved for it, but if anything wrong happen when creating the vm or get timed out, the volume will be in the reserved stage forever. We would like to know if we could update the state or delete them without the admin account? We tried the command line tools 'openstack volume set --state available <volume_id>' and 'cinder reset-state --state available <volume_id>', but both got: Policy doesn't allow volume_extension:volume_admin_actions:reset_status to be performed. (HTTP 403) (Request-ID: req-b50236c3-5662-4cc0-8602-19e79e80219d) ERROR: Unable to reset the state for the specified entity(s). We are using the chameleon openstack cloud but we are not the admin user there, so wondering if there is any method that we could update the volume in our own project? Thanks and have a great day! Catherine

3 2

Error adding a Plain Volume created through GUI to Instance
by KK CHN 08 Sep '21

08 Sep '21

ExecutionError: Unexpected error while running command. Command: rbd map volume-4dcb78d5-e0ce-4bfc-95ae-36a4417c35d8 --pool volumes --id volumes --mon_host 10.236.223.131:6789 --mon_host 10.236.223.132:6789 Exit code: 6 Stdout: 'RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable volumes/volume-4dcb78d5-e0ce-4bfc-95ae-36a4417c35d8 journaling".\nIn some cases useful info is found in syslog - try "dmesg | tail".\n' Stderr: 'rbd: sysfs write failed\nrbd: map failed: (6) No such device or address\n' I am trying to add a 20 GB plain volume created through Horizon GUI to an existing Windows 2012 R VM running with VirtIO drivers installed. But this volume adding results in the error as shown above, trace tells to fix it by rbd feature disable .. what is the permanent fix for this issue ? and why it happens ? ( OpenStack Ussuri, Glance, Ceph , qemu-kvm) root@ctrl1:/home/cloud# ceph --version ceph version 14.2.19 (bb796b9b5bab9463106022eef406373182465d11) nautilus (stable) root@ctrl1:/home/cloud#

3 2