- openstack-discuss - lists.openstack.org

[all][tc] Technical Committee next weekly meeting on Sept 30th at 1500 UTC
by Ghanshyam Mann 29 Sep '21

29 Sep '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for Sept 30th at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Sept 29th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

DPDK Poor performance
by Satish Patel 29 Sep '21

29 Sep '21

Folk, I have deployed DPDK on one of my compute nodes to reply to my sriov deployment which doesn't support bonding. This is what I am doing to do a performance benchmark. Spun up VM with 8 vCPU and 8GB memory with single virtual nic. Then install nuttcp tool and start nuttcp -S (server mode) and then use Physical server outside my lab and use following command to blast some traffic nuttcp -T 30 -i -u -w4m -R 2G 10.69.7.130 I found 120kpps rate working fine but after that I noticed a packet drop on nuttcp. Here is the full details of my deployment and load-test result - https://paste.opendev.org/show/809675/ Small confusion related hugepage - I am seeing following in /dev/hugepages/ I believe rtemap_0 created by DPDK root@ovn-lab-comp-dpdk-1:~# ls -l /dev/hugepages/rtemap_0 -rw------- 1 root root 1073741824 Sep 29 02:28 /dev/hugepages/rtemap_0 Why is the qemu instance file empty? Does that mean it's not using a huge page for dpdk performance? root@ovn-lab-comp-dpdk-1:~# ls -l /dev/hugepages/libvirt/qemu/1-instance-00000045/ total 0 I have hw:mem_page_size='large' setting in flavor so it should use hugepage. am i missing something?

1 0

[largescale-sig] Next meeting: Sept 29th, 15utc
by Thierry Carrez 29 Sep '21

29 Sep '21

Hi everyone, The Large Scale SIG meeting is back this Wednesday in #openstack-operators on OFTC IRC, at 15UTC. You can doublecheck how that time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20210929T15 A number of topics have already been added to the agenda, including discussing the final details of our next OpenInfra.Live show! Feel free to add other topics to our agenda at: https://etherpad.openstack.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

[kolla] Yoga PTG
by Michał Nasiadka 29 Sep '21

29 Sep '21

Hola Koalas! We are approaching the time of the next PTG. Kolla Yoga etherpad is available and initially populated[1], feel free to add any topics that shall be discussed and register your planned attendance. Don't forget to register! [2]. [1] https://etherpad.opendev.org/p/kolla-yoga-ptg [3] https://openinfra-ptg.eventbrite.com Best regards, Michal Nasiadka

1 0

[cinder] Bug deputy report for week of 2021-29-09
by Sofia Enriquez 29 Sep '21

29 Sep '21

This is a bug report from 2021-15-09 to 2021-29-09. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Medium - https://bugs.launchpad.net/os-brick/+bug/1943977 "os_bricks fails to rerty "iscsiadm -m session" when iser_use_multipath and iscsi_use_multipath are set in nova.conf". Assigned to Sophie Huang. - https://bugs.launchpad.net/os-brick/+bug/1945323 "Can't attach NVMe volume" Low - https://bugs.launchpad.net/cinder/+bug/1945169 " [swift] volume backup's status keep in 'restoring' and Lost connection to MySQL server during query". Assigned to silphon zhang. Wishlist - https://bugs.launchpad.net/cinder/+bug/1945249 '[charm] no upgrade path to cinder-lvm subordinate'. Unassigned. Cheers, -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[Cinder] [Nova] [Interop] "compute-volume-list" API guideline
by Vida Haririan 29 Sep '21

29 Sep '21

Hi Cinder and Nova teams, The Interop team is reviewing all related open stories to determine next steps. We researched the current status of 1638112:“os-volumes should be flagged for removal in the compute API guidelines“ [1] and noticed that "compute-volume-list" was removed in 2017.01 guidelines [2][3]. However, two related tests still remain in tempest [4][5] and we are reaching out to determine if the API is still in use or has been moved/renamed. Please let us know if you have any questions. [1] https://storyboard.openstack.org/#!/story/1638112 [2] https://opendev.org/osf/interop/src/commit/dd7ae2660de5daa82ab4fe080e875d63… [3] https://opendev.org/osf/interop/src/branch/master/guidelines/next.json [4] https://opendev.org/openstack/tempest/src/commit/ae41052a51f5dbb748eb6bf4f2… [5] https://opendev.org/openstack/tempest/src/commit/ae41052a51f5dbb748eb6bf4f2… Thank you, -- *Vida Haririan* Senior Software Quality Engineer Red Hat EMEA <https://www.redhat.com/>

2 1

[announce][debian][xena] general availability of OpenStack Xena in Debian
by Thomas Goirand 29 Sep '21

29 Sep '21

Hi! It's my pleasure to announce the general availability of OpenStack Xena in Debian. Indeed, I've just finished uploading everything to Debian Experimental at the beginning of this week, and was able yesterday, to pop my first VM over on top of Xena. This marks the first readiness of the release, from the Debian point of view (even though there probably will be things to fix, like after addressing https://bugs.launchpad.net/oslo.policy/+bug/1945336 for example, and uploading the final release to replace the RCs when that becomes available). The Bullseye backports are available the usual way, for example using extrepo: apt-get install extrepo extrepo enable openstack_xena apt-get update ... or directly setting-up the http://bullseye-xena.debian.net repository the usual way. I'm currently in the process of uploading all of Xena to Debian Unstable (but please bare with me: there's 225+ packages in this update). Note that Xena can already be installed by OCI [1]. Please report any issue you may find, on OCI or on the Debian packages, through the Debian BTS as usual. Cheers, Thomas Goirand (zigo) [1] https://salsa.debian.org/openstack-team/debian/openstack-cluster-installer

1 0

[kolla][neutron][victoria] metering agent on compute nodes
by Tomasz Rutkowski 29 Sep '21

29 Sep '21

Hi All, I have issue with neutron-metering-agent - on network node it works ok, however on compute nodes the list of routers is constantly empty (and l3-agent works normal)... how to debug why outside network node the metering-agent does not recieve list of routers? with best regards -- Tomasz Rutkowski

2 2

[all] Gate status: All devstack based jobs are failing for placement url issue in devstack
by Ghanshyam Mann 28 Sep '21

28 Sep '21

Hello Everyone, All the devstack based jobs are failing with placement URL issue in devstack. The root cause is not known yet. You will see the below error in devstack/summary log- + ./stack.sh:main:1455 : /usr/local/bin/nova-status --config-file /etc/nova/nova.conf upgrade check Placement logs: INFO placement.requestlog [None req-3ad7ab8b-7c85-477a-b156-f5ca652a38cc service placement] 10.208.226.48 "GET /placemen//" status: 404 len: 162 microversion: 1.0 Please do not recheck until the issue is fixed. -gmann

5 8

OpenInfra Live - September 30th at 9am CT
by Erin Disney 28 Sep '21

28 Sep '21

Hi everyone, This week’s OpenInfra Live episode is brought to you by the OpenStack and Ceph communities. Ceph is a highly scalable distributed-storage open source solution offering object, block, and file storage. Join us as various Community members discuss the basics, ongoing development and integration of OpenStack with Ceph. Date and time: September 30th at 9am CT (1400 UTC) You can watch us live on: YouTube: https://www.youtube.com/watch?v=zJVoleSpSOk LinkedIn: https://www.linkedin.com/video/event/urn:li:ugcPost:6846862917225320448/ Facebook: https://www.facebook.com/104139126308032/posts/4364096940312208/ WeChat: Recording will be posted on OpenStack WeChat after the live stream Speakers: Kendall Nelson (OpenInfra Foundation) John Fulton (Red Hat) Francesco Pantano (Red Hat) Tom Barron (Red Hat) Loan Harrouin (Societe Generale) Pawel Sadowski (OVH) Mohammed Naser (VEXXHOST) Have an idea for a future episode? Share it now at ideas.openinfra.live <https://openinfrafoundation.formstack.com/forms/openinfralive>. Register now for OpenInfra Live: Keynotes, a special edition of OpenInfra Live on November 17-18th starting at 1500 UTC: https://openinfralivekeynotes.eventbrite.com/ Thanks, Erin Erin Disney Event Marketing Open Infrastructure Foundation

1 0

Issue on accessing object storage containers from outside
by Michel Niyoyita 28 Sep '21

28 Sep '21

Dear all I have created containers and upload objects into containers . I can access these containers and objects trough horizon dashboard but once make them public I can not reach them through the provided link . If any one have an idea on how to access containers and their object using the link from outside he can help and guide. I deployed openstack wallaby using kolla-ansible and ceph pacific using ansible , both are using ubuntu 20.04 as OS Michel

3 4

[openstack-qa] opendev.org mirrors
by Eduardo Santos 28 Sep '21

28 Sep '21

Hi folks, On US-based servers, a DevStack deploy takes 15 minutes. When deploying it from Brazil (where I live) however, it takes more than an hour. Examining the stack.sh logs, git_timed appears to be the culprit. I'm using a 250 Mbps connection. Do opendev.org mirrors exist for other regions other than the US? If not, has this been discussed yet? I believe this is more of a financial discussion othet than technical, but I'm curious if other users have raised this question before. I'm not sure if openstack-qa is the most appropriate flag. If not, please feel free to correct. Thanks!

3 3

Creating multiple servers with a shared volume
by open infra 28 Sep '21

28 Sep '21

Hi, I am planning to spin hundreds of servers with a shared storage. I have created one instance and attached a 1 TB volume (multiattach) and created a snapshot of the instance. But when I spin another server using the snapshot, I noticed that instead of expected shared volume, individual 1 TB size additional (dedicated) volume has been attached to each VM. Is there a mechanism where I can create multiple instances along with a shared volume attached to the instance instead of attaching later? Regards, Danishka

3 5

[Tripleo] Support of Alma/Rocky Linux
by Anirudh Gupta 28 Sep '21

28 Sep '21

Hi Team As per the document, the supporting OS mentioned in the documentation is only Centos Stream and No Stream version (as of now) Since Centos has officially stopped releasing new versions and looking for EOL. I Just wanted to check if there is a possibility that we can install Tripleo on a specific version of Alma Linux or Rocky Linux Regards Anirudh Gupta

2 3

RE: Problems with Fluentd buffer [fluentd][kolla-ansible]
by Adam Tomas 28 Sep '21

28 Sep '21

Hi, I’ve turned off „central_logging” and now I have only several errors like below on all nodes (3-7 per day) 0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 04:49:32.728326628 +0200 chunk="5ccf11fdc0d6876abdef813211371285" error_class=RestClient::RequestTimeout error="408 Request Timeout” On compute node, which had errors like this : /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag=„openstack_python" I’ve restarted fluentd container on that node and now it’s not showing errors like this anymore. So what’s the problem? Performance of Monasca/Kibana/Elasticsearch? But the node (telemetry) on which this stack is running is not overloaded - 20-30% CPU (24 cores on that node) , 40% RAM (there is 64GB), a lot of space on SSD. Best regards Adam Tomas > Hi, > after some time of using newly deployed kolla-ansible I have fluentd errors: > > on all controllers, storage and network nodes: > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 01:40:46 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 01:40:47.847564807 +0200 chunk="5ccee7b1762157f0ae10dcbeb4e14fd8" error_class=RestClient::GatewayTimeout error="504 Gateway Timeout" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 04:49:31 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 04:49:32.728326628 +0200 chunk="5ccf11fdc0d6876abdef813211371285" error_class=RestClient::RequestTimeout error="408 Request Timeout” > > > on compute nodes: > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="openstack_python" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="kolla.var.log.kolla.monasca.agent-statsd.log" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"action"=>:throw_exception, "message"=>"failed to write data into buffer by buffer overflow action=:throw_exception"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"error"=>"#<Fluent::Plugin::Buffer::BufferOverflowError: buffer space has too many data>", "location"=>"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'", "tag"=>"openstack_python", "message"=>"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'\" tag=\"openstack_python\""} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"error"=>"#<Fluent::Plugin::Buffer::BufferOverflowError: buffer space has too many data>", "location"=>"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'", "tag"=>"kolla.var.log.kolla.monasca.agent-statsd.log", "message"=>"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'\" tag=\"kolla.var.log.kolla.monasca.agent-statsd.log\""} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="openstack_python" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="kolla.var.log.kolla.monasca.agent-forwarder.log" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"action"=>:throw_exception, "message"=>"failed to write data into buffer by buffer overflow action=:throw_exception"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data” > > and on monitoring node: > > 2021-09-27 14:40:50 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=84.05643947119825 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:42:53 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=122.65328024700284 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:44:15 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=82.32426812895574 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:46:05 +0200 [warn]: #0 Bulk sending messages to monasca-api threw exception exceptionew=#<RestClient::Exceptions::ReadTimeout: Timed out reading data from server> > 2021-09-27 14:46:05 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 14:46:06.042150289 +0200 chunk="5ccf89c9ec54b09e616f7167d5f93cc1" error_class=RestClient::Exceptions::ReadTimeout error="Timed out reading data from server" > 2021-09-27 14:46:05 +0200 [warn]: #0 suppressed same stacktrace > 2021-09-27 14:47:58 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=112.90601075813174 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:47:58 +0200 [warn]: #0 retry succeeded. chunk_id="5ccf89c9ec54b09e616f7167d5f93cc1" > 2021-09-27 14:50:16 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=136.24769522389397 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:52:27 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=129.86474119895138 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:54:08 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=100.71324555086903 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:56:31 +0200 [warn]: #0 Bulk sending messages to monasca-api threw exception exceptionew=#<RestClient::Exceptions::ReadTimeout: Timed out reading data from server> > >

2 1

key_repository does not exist or Keystone does not have sufficient permission to access it [kolla-ansible][keystone]
by Adam Tomas 27 Sep '21

27 Sep '21

Hi, some next problem with my kolla-ansible deployment. On controller nodes I have (several times per hour) error messages like below: ERROR keystone.common.fernet_utils [req-386e223e-5f08-4959-bd0b-3dbbfd3b534f 73c7a9a7175e46e9971181963267225b efa5d1ce80324476b4ce189686dda2d1 - default default] Either [None] key_repository does not exist or Keystone does not have sufficient permission to access it: /etc/keystone/credential-keys/ I’ve checked - such path doesn’t exist inside keystone containers. I found bug #1863643 (https://bugs.launchpad.net/kolla/+bug/1863643 <https://bugs.launchpad.net/kolla/+bug/1863643>) but I don’t see any progress on it nor information that victoria is not affected (so I assume it’s the same bug and Victoria is affected?). What should be done to get rid of this errors? Best regards Adam Tomas

1 0

[cinder] reminder: this week's meeting in video+IRC
by Brian Rosmaita 27 Sep '21

27 Sep '21

Quick reminder that this week's Cinder team meeting on Wednesday 29 September, being the final meeting of the month, will be held in both videoconference and IRC at the regularly scheduled time of 1400 UTC. These are the video meeting rules we've agreed to: * Everyone will keep IRC open during the meeting. * We'll take notes in IRC to leave a record similar to what we have for our regular IRC meetings. * Some people are more comfortable communicating in written English. So at any point, any attendee may request that the discussion of the current topic be conducted entirely in IRC. * The meeting will be recorded. connection info: https://bluejeans.com/3228528973 meeting agenda: https://etherpad.opendev.org/p/cinder-xena-meetings cheers, brian

1 0

[qa] Office Hour Sep 28th is cancelled
by Martin Kopec 27 Sep '21

27 Sep '21

Hi everyone, the next QA Office Hour on September 28th at 2pm UTC is cancelled due to a public holiday in my country and I won't be able to moderate it. Thank you for your understanding, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA

1 0

[OSSA-2021-002] Nova: Open Redirect in noVNC proxy (CVE-2021-3654)
by Jeremy Stanley 27 Sep '21

27 Sep '21

=========================================== OSSA-2021-002: Open Redirect in noVNC proxy =========================================== :Date: July 29, 2021 :CVE: CVE-2021-3654 Affects ~~~~~~~ - Nova: <21.2.3, >=22.0.0 <22.2.3, >=23.0.0 <23.0.3 Description ~~~~~~~~~~~ Swe Aung, Shahaan Ayyub, and Salman Khan with the Monash University Cyber Security team reported a vulnerability affecting Nova's noVNC proxying implementation which exposed access to a well-known redirect behavior in the Python standard library's http.server.SimpleHTTPRequestHandler and thus noVNC's WebSockifyRequestHandler which uses it. By convincing a user to follow a specially-crafted novncproxy URL, the user could be redirected to an unrelated site under control of the attacker in an attempt to convince them to divulge credentials or other sensitive data. All Nova deployments with novncproxy enabled are affected. Errata ~~~~~~ The initial fix did not take into account the possibility of bypass using exactly three slashes. This update provides a more thorough revised fix for the issue. The affected versions list has been updated to indicate versions expected to include the newer solution. Patches ~~~~~~~ - https://review.opendev.org/791807 (Train) - https://review.opendev.org/806629 (errata 1) (Train) - https://review.opendev.org/791806 (Ussuri) - https://review.opendev.org/806628 (errata 1) (Ussuri) - https://review.opendev.org/791805 (Victoria) - https://review.opendev.org/806626 (errata 1) (Victoria) - https://review.opendev.org/791577 (Wallaby) - https://review.opendev.org/805818 (errata 1) (Wallaby) - https://review.opendev.org/791297 (Xena) - https://review.opendev.org/805654 (errata 1) (Xena) Credits ~~~~~~~ - Swe Aung from Monash University Cyber Security team (CVE-2021-3654) - Shahaan Ayyub from Monash University Cyber Security team (CVE-2021-3654) - Salman Khan from Monash University Cyber Security team (CVE-2021-3654) References ~~~~~~~~~~ - https://launchpad.net/bugs/1927677 - https://bugs.python.org/issue32084 - http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3654 Notes ~~~~~ - The stable/train branch is under extended maintenance and will receive no new point releases, but a patch for it is provided as a courtesy. OSSA History ~~~~~~~~~~~~ - 2021-09-27 - Errata 1 - 2021-07-29 - Original Version -- Jeremy Stanley

1 0

Not registering Hypervisors
by Karera Tony 27 Sep '21

27 Sep '21

Hello Team, I have deployed Openstack Victoria using Kolla-ansible on Ubuntu 20.04 and ceph as the backend storage for Nova, Cinder and Glance. It finished with no error but it has failed to register any on the Compute Nodes under Hypervisors. Any idea on how to resolve this ? Regards Tony Karera

5 22

[neutron] upgrading port binding:profile.allocation value for port-resource-request-groups API extension
by Balazs Gibizer 27 Sep '21

27 Sep '21

Hi Neutrinos! We found a technical challenge during implementing the port-resource-request-groups API extension[1]. That extension changes the format of the port.resoruce_request as well as the format of the port.binding:profile.allocation. The former is a calculated field on the port so that is easy. However the bindig:profile is persisted in the database so data migration is needed. What is the canonical way to do such DB data translation in Neutron? Can we translate the data in place during alembic migration? Or should we do some kind of online data migration when the data is translated by neutron when it is read from the db? cheers, gibi [1] https://review.opendev.org/c/openstack/neutron/+/805637/5

2 6

Problems with Fluentd buffer [fluentd][kolla-ansible]
by Adam Tomas 27 Sep '21

27 Sep '21

Hi, after some time of using newly deployed kolla-ansible I have fluentd errors: on all controllers, storage and network nodes: /var/log/kolla/fluentd/fluentd.log:2021-09-27 01:40:46 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 01:40:47.847564807 +0200 chunk="5ccee7b1762157f0ae10dcbeb4e14fd8" error_class=RestClient::GatewayTimeout error="504 Gateway Timeout" /var/log/kolla/fluentd/fluentd.log:2021-09-27 04:49:31 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 04:49:32.728326628 +0200 chunk="5ccf11fdc0d6876abdef813211371285" error_class=RestClient::RequestTimeout error="408 Request Timeout” on compute nodes: /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="openstack_python" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="kolla.var.log.kolla.monasca.agent-statsd.log" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"action"=>:throw_exception, "message"=>"failed to write data into buffer by buffer overflow action=:throw_exception"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"error"=>"#<Fluent::Plugin::Buffer::BufferOverflowError: buffer space has too many data>", "location"=>"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'", "tag"=>"openstack_python", "message"=>"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'\" tag=\"openstack_python\""} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"error"=>"#<Fluent::Plugin::Buffer::BufferOverflowError: buffer space has too many data>", "location"=>"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'", "tag"=>"kolla.var.log.kolla.monasca.agent-statsd.log", "message"=>"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'\" tag=\"kolla.var.log.kolla.monasca.agent-statsd.log\""} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="openstack_python" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="kolla.var.log.kolla.monasca.agent-forwarder.log" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"action"=>:throw_exception, "message"=>"failed to write data into buffer by buffer overflow action=:throw_exception"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data” and on monitoring node: 2021-09-27 14:40:50 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=84.05643947119825 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:42:53 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=122.65328024700284 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:44:15 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=82.32426812895574 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:46:05 +0200 [warn]: #0 Bulk sending messages to monasca-api threw exception exceptionew=#<RestClient::Exceptions::ReadTimeout: Timed out reading data from server> 2021-09-27 14:46:05 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 14:46:06.042150289 +0200 chunk="5ccf89c9ec54b09e616f7167d5f93cc1" error_class=RestClient::Exceptions::ReadTimeout error="Timed out reading data from server" 2021-09-27 14:46:05 +0200 [warn]: #0 suppressed same stacktrace 2021-09-27 14:47:58 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=112.90601075813174 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:47:58 +0200 [warn]: #0 retry succeeded. chunk_id="5ccf89c9ec54b09e616f7167d5f93cc1" 2021-09-27 14:50:16 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=136.24769522389397 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:52:27 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=129.86474119895138 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:54:08 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=100.71324555086903 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:56:31 +0200 [warn]: #0 Bulk sending messages to monasca-api threw exception exceptionew=#<RestClient::Exceptions::ReadTimeout: Timed out reading data from server> 2021-09-27 14:56:31 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 14:56:32.140741991 +0200 chunk="5ccf89e1feb471b7ce25a26d37977baa" error_class=RestClient::Exceptions::ReadTimeout error="Timed out reading data from server" 2021-09-27 14:56:31 +0200 [warn]: #0 suppressed same stacktrace 2021-09-27 14:58:03 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=90.91785193886608 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:58:03 +0200 [warn]: #0 retry succeeded. chunk_id="5ccf89e1feb471b7ce25a26d37977baa" 2021-09-27 15:00:01 +0200 [info]: #0 detected rotation of /var/log/kolla/kafka/server.log; waiting 5 seconds 2021-09-27 15:00:01 +0200 [info]: #0 following tail of /var/log/kolla/kafka/server.log 2021-09-27 15:00:10 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=127.54797655600123 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 15:02:29 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=137.74220423400402 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 15:03:08 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=38.81670297612436 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 15:03:13 +0200 [info]: #0 detected rotation of /var/log/kolla/kafka/controller.log; waiting 5 seconds 2021-09-27 15:03:13 +0200 [info]: #0 following tail of /var/log/kolla/kafka/controller.log What seems to be the problem? Is there any way to run more than one fluentd worker in kolla? Best regards Adam Tomas

1 0

[keystone] Correct policies setup for System Administrators
by Jan Wasilewski 27 Sep '21

27 Sep '21

Hello, I am preparing policies configuration before an upgrade to the newer OpenStack release(Stein) and I would like to create a group of System Administrators to be able to get i.e. a list of all projects in the OpenStack cloud. I was following a description from this page [1] but it seems my admin user is able to get only a list of projects where it is directly added(i.e. with member role, reader role, or admin role). I am just wondering if we can list all of the OpenStack projects by System Administrator user without role reader added to every single project? To summarize what steps were done so far: - Original policy.json file which was used is here [2] - Only one option was changed so far: from: "identity:list_projects": "rule:cloud_admin or rule:admin_and_matching_domain_id", to: "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)", - Output for command: openstack role assignment list --system all --role member --role reader +----------------------------------+------+----------------------------------+---------+--------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +----------------------------------+------+----------------------------------+---------+--------+--------+-----------+ | e39e97c23bfe45d1a2f9689b6985f990 | | a0841b83f583477887219f27dd95477b | | | all | False | +----------------------------------+------+----------------------------------+---------+--------+--------+-----------+Shows only role reader, not role member, which is a bit strange if we compare with linked page above. But we have this in implied roles:openstack implied role list +----------------------------------+-----------------+----------------------------------+-------------------+ | Prior Role ID | Prior Role Name | Implied Role ID | Implied Role Name | +----------------------------------+-----------------+----------------------------------+-------------------+ | a3c7bb5d06884b048c1bfb4403b82b42 | admin | 3f20cb7be46346a8b2ba65c4684d50a3 | member | | a3c7bb5d06884b048c1bfb4403b82b42 | admin | 9fe2ff9ee4384b1894a90878d3e92bab | _member_ | | 3f20cb7be46346a8b2ba65c4684d50a3 | member | e39e97c23bfe45d1a2f9689b6985f990 | reader | +----------------------------------+-----------------+----------------------------------+-------------------+- Admin roles are grouped in a group ATM.Admin: openstack role assignment list --names --system all --role admin: +-------+---------------------+-------------------+---------+--------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +-------+---------------------+-------------------+---------+--------+--------+-----------+ | admin | | ATM.Admin@Default | | | all | False | | admin | admin@Default | | | | all | False | | admin | jwasilewski@Default | | | | all | False | +-------+---------------------+-------------------+---------+--------+--------+-----------+Just to be sure that IDs are linked, we can check it here:openstack role assignment list --system all --role admin +----------------------------------+----------------------------------+----------------------------------+---------+--------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +----------------------------------+----------------------------------+----------------------------------+---------+--------+--------+-----------+ | a3c7bb5d06884b048c1bfb4403b82b42 | | a0841b83f583477887219f27dd95477b | | | all | False | | a3c7bb5d06884b048c1bfb4403b82b42 | 19416fe5a2da45c88eb66c3aaf856c73 | | | | all | False | | a3c7bb5d06884b048c1bfb4403b82b42 | f42df418fd404d04b9bdabf2f1b49fd9 | | | | all | False | +----------------------------------+----------------------------------+----------------------------------+---------+--------+--------+-----------+ So by linking roles(implied roles): admin(a3c7bb5d06884b048c1bfb4403b82b42 ) -> member(3f20cb7be46346a8b2ba65c4684d50a3) -> reader(e39e97c23bfe45d1a2f9689b6985f990). Correlation is visible, based on that my user(jwasilewski) should retrieve a full project list, but it seems only three projects are visible where this user is an admin. I do not want to add my user as a reader to every single project to be able to list all of them. Is there a way how to make it or the only way is to add this role(reader) for a user to all projects? Thank you in advance for any suggestions. Best regards, Jan Wasilewski [1] https://docs.openstack.org/keystone/stein/admin/service-api-protection.html… -> https://docs.openstack.org/keystone/latest/admin/service-api-protection.htm… [2] https://paste.openstack.org/show/bq0HgyqouZF1vywKVkGn/

1 0

[neutron] CI update
by Slawek Kaplonski 27 Sep '21

27 Sep '21

Hi, Due to very urgent things which I had yesterday, I wasn't able to run Neutron CI meeting as usually. Fortunatelly we don't have many new issues in our CI. There is only one new issue in our scenario jobs which I wanted to discuss [1]. It's impacting Ironic gates but I noticed it also in the Neutron CI as well. See [2] or [3] for example. I'm not sure about Ironic jobs but in Neutron I saw it mostly (or only, I'm not sure) in the multinode jobs. [1] https://bugs.launchpad.net/neutron/+bug/1944201 [2] https:// e36beaa2ff297ebe7d5f-5944c3d62ed334b8cdf50b534c246731.ssl.cf5.rackcdn.com/ 805849/9/check/neutron-ovs-tempest-dvr-ha-multinode-full/f83fa96/compute1/ logs/screen-q-agt.txt [3] https://storage.bhs.cloud.ovh.net/v1/ AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_88f/803045/12/check/ neutron-ovs-tempest-slow/88f8bb7/job-output.txt -- Slawek Kaplonski Principal Software Engineer Red Hat

5 8

[neutron] Bug deputy report (week starting on 2021-09-20)
by Bernard Cafarelli 27 Sep '21

27 Sep '21

Hello neutrinos, here is the list of bugs reported for last week, overall all bugs have assignees or fixes in progress. Both critical bugs are merged and with pending xena backports (this week is the final RCs week) In details: Critical * neutron-openvswitch-agent crashes on start with firewall config of br-int - https://bugs.launchpad.net/neutron/+bug/1944201 First seen in Ironic jobs but we also see it in our jobs, fix by ralonsoh merged: https://review.opendev.org/c/openstack/neutron/+/810592 (pending xena backport) * neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle fails with Port already has an attached device - https://bugs.launchpad.net/tripleo/+bug/1943708 Previously reported as https://bugs.launchpad.net/neutron/+bug/1943679, fix with revert proposed by slaweq and merged: https://review.opendev.org/c/openstack/neutron/+/809550 (pending xena backport) High * Traffic leaked from dhcp port before vlan tag is applied - https://bugs.launchpad.net/neutron/+bug/1930414 Possible security issue that was under investigation and now public, assigned to rubasov * OVN does not update the network segment tag - https://bugs.launchpad.net/neutron/+bug/1944708 Assigned to lucasagomes, fix in review: https://review.opendev.org/c/openstack/neutron/+/810704 Medium * Fix OVN metadata agent functional tests for Chassis_Private - https://bugs.launchpad.net/neutron/+bug/1944460 Functional OVN tests failing when using newer OVN, fix merged: https://review.opendev.org/c/openstack/neutron/+/810309 * IPv6 slaac subnet creation causes FixedIpsSubnetsNotOnSameSegment error - https://bugs.launchpad.net/neutron/+bug/1944948 Error raised when creating an IPv6 slaac subnet with multisegment, assigned to elvira Incomplete * Instances with SRIOV ports loose access after failed live migrations - https://bugs.launchpad.net/neutron/+bug/1944619 SRIOV live migration issue, liuyulong gave some pointers -- Bernard Cafarelli

1 0

[release] Release countdown for week R-1, Sep 27 - Oct 1
by Herve Beraud 27 Sep '21

27 Sep '21

Development Focus ----------------- We are on the final mile of the Xena development cycle! Remember that the Xena final release will include the latest release candidate (for cycle-with-rc deliverables) or the latest intermediary release (for cycle-with-intermediary deliverables) available. September 30 is the deadline for final Xena release candidates as well as any last cycle-with-intermediary deliverables. We will then enter a quiet period until we tag the final release on October 6, 2021. Teams should be prioritizing fixing release-critical bugs, before that deadline. Otherwise it's time to start planning the Yoga development cycle, including discussing sessions content in preparation of the PTG on the week of October 18. Actions ------- Watch for any translation patches coming through on the stable/xena branch and merge them quickly. If you discover a release-critical issue, please make sure to fix it on the master branch first, then backport the bugfix to the stable/xena branch before triggering a new release. Please drop by #openstack-release with any questions or concerns about the upcoming release ! Upcoming Deadlines & Dates -------------------------- Final Xena release: October 6 Yoga PTG: October 18-22 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud

1 0

[neutron][ovn] support for stateless NAT for floating ip in ml2 ovn
by Moshe Levi 26 Sep '21

26 Sep '21

Hi all, OVN support stateless NAT operations [1] for use case of 1:1 mapped between inner and external ips, i.e dnat_and_snat rule. In openstack is the floating ip use-case. Looking on ml2 ovn support it seem that it only support floating ip with connection tracking. Can ml2 ovn support also the stateless NAT option? Is there concerns using stateless NAT? [1] - https://patchwork.ozlabs.org/project/openvswitch/cover/1570154179-14525-1-g…

3 8

[requirements] Please consider bumping docker to 5.0.2 in Xena u-c
by Pierre Riteau 26 Sep '21

26 Sep '21

Hello, Xena upper constraints currently contain docker 5.0.1, released on August 31. This version includes a critical issue [1] which was fixed by version 5.0.2 released the next day. Except for a docstring update this is the only change, according to git diff. Would it be possible to update Xena u-c to use docker 5.0.2? We currently have to work around it in Kayobe to avoid installing the buggy version [2]. Thanks, Pierre Riteau (priteau) [1] https://github.com/docker/docker-py/issues/2885 [2] https://review.opendev.org/c/openstack/kayobe/+/807345/9/ansible/kolla-targ…

3 2

[zun][requirements] request to blacklist docker 5.0.1 for X release
by Hongbin Lu 26 Sep '21

26 Sep '21

Hi, It appears that docker 5.0.1 is broken [1]. All the OpenStack projects that depends on docker-py (i.e. Zun [2]) are impacted. Would the requirements team consider blacklist docker 5.0.1 for X release? Please note that Zun must have such blacklist. Otherwise, the X release will be totally broken. [1] https://github.com/docker/docker-py/issues/2885 [2] https://bugs.launchpad.net/zun/+bug/1945088 Best regards, Hongbin

2 2

anyone running ovn with ovs+dpdk
by Satish Patel 25 Sep '21

25 Sep '21

Folks, Anyone running ovn with dpdk on ubuntu 20.04 (Focal) with ovs version 2.13.3 ? I am trying to do that but my ovs switch keep crashing as soon as i spin up vm and it creates socket file in /var/run/openvswitch and I am not able to understand what is wrong with it. (no generating crash dump also) Please advise or share your thoughts.

1 0

All Devstack Based Jobs Failing to Create Swap on master
by Clark Boylan 24 Sep '21

24 Sep '21

Hello everyone, At approximately 12:53 UTC September 24 (today) [0] Devstack's Xena branch was created. Due to an unexpected side effect in Devstack's Zuul configs this has caused all Devstack based jobs against master and possible stable/xena to RETRY_LIMIT and fail. The underlying issue you'll see in the jobs is that mkswap cannot create swap because it already exists [1]. This happens because Devstack's pre-run playbook is running twice [2] and the configure-swap role is not idempotent. The role is running twice because the Devstack base job is loading master and stable/xena's configs and composing them together which results in two pre-run playbooks being selected one for each branch. This happens because Devstack's stable/xena branch explicitly states that its configs apply to master and feature/r1 branches [3]. I've pushed up a change [4] to remove this Zuul configuration from the Xena branch which I expect will fix things for us. [0] https://zuul.opendev.org/t/openstack/build/ebe77aee0c104c9e9b917ff28a4731ee… [1] https://zuul.opendev.org/t/openstack/build/52fa83b18e1a476b80afc41617fe68f9… [2] https://zuul.opendev.org/t/openstack/build/52fa83b18e1a476b80afc41617fe68f9… [3] https://opendev.org/openstack/devstack/src/branch/stable/xena/.zuul.yaml#L1… [4] https://review.opendev.org/c/openstack/devstack/+/810947

2 2

[all][tc] What's happening in Technical Committee: summary 24th Sept, 21: Reading: 5 min
by Ghanshyam Mann 24 Sep '21

24 Sep '21

Hello Everyone, Here is this week's summary of the Technical Committee activities. 1. TC Meetings: ============ * TC this week IRC meeting held on Sept 23rd Thursday. * Most of the meeting discussions are summarized in the below sections (Completed or in-progress activities section). To know more details, you can check the complete logs @ https://meetings.opendev.org/meetings/tc/2021/tc.2021-09-23-15.00.log.html * We will have next week's meeting on Sept 30th, Thursday 15:00 UTC on IRC, feel free the topic in agenda [1] by Sept 29th. 2. What we completed this week: ========================= * Moved Cloud Research SIG to active state[2] 3. Activities In progress: ================== TC Tracker for Xena cycle ------------------------------ * TC is using the etherpad[3] for Xena cycle working item. We will be checking and updating the status biweekly on the same etherpad. * Current status is: 7 completed, 6 in-progress Open Reviews ----------------- * Five open reviews for ongoing activities[4]. Add project health check tool ----------------------------------- * We are automating the project health checks. Currently, every TC member is assigned a number of projects to keep checking the project health/ listening to their issues but this is not so successful so far. * Rico has proposed the code[5] to collect the contribution stats and we are in the process of documenting the usage and interpretation of those stats. Stable Core team process change --------------------------------------- * As discussed in Shanghai and Xena PTG, there are some concerns over the current stable core team process (especially now when there are very few global stable team members) and to improve the process, mnaser has proposed the draft resolution[6] . Feel free to provide early feedback if you have any. Removing stable release 'Unmaintained' phase ------------------------------------------------------- * 'Unmaintained' phase in the stable release process has been confusing than adding any value. * Elod has proposed the removal of the 'unmaintained' phase[7], feel free to reply if any objection otherwise we will merge the patch on Monday(27th Sept). Places for projects to spreading the word ------------------------------------------------- * Amy has started to list down all the places that projects can use to spread the word out about your project which will help to get additional contributors and users[8]. She also sent emails to PTLs. * Please make note of it and use such places/blogs to highlights/educate operators/users/community etc. TC tags analysis ------------------- * As discussed in the last PTG, TC is working on an analysis of the usefulness of the Tags framework[9] or what all tags can be cleaned up. * We are still waiting for the operator's response to the email on openstack-disscuss ML[10]. If you are an operator, please respond to the email and based on the feedback we will continue the discussion in PTG. Project updates ------------------- * Add the cinder-netapp charm to Openstack charms[11] * Retiring js-openstack-lib [12] * Retire puppet-freezer[13] Yoga release community-wide goal ----------------------------------------- * Please add the possible candidates in this etherpad [14]. * Current status: "Secure RBAC" is selected for Yoga cycle[15]. PTG planning ---------------- * We are collecting the PTG topics in etherpad[16], feel free to add any topic you would like to discuss. * We discussed the live stream of one of the TC PTG sessions like we did last time. Once we will have more topics in etherpad then we can select the appropriate one. Test support for TLS default: ---------------------------------- * Rico has started a separate email thread over testing with tls-proxy enabled[17], we encourage projects to participate in that testing and help to enable the tls-proxy in gate testing. 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[18]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15 UTC [19] 3. Office hours: The Technical Committee offers a weekly office hour every Tuesday at 0100 UTC [20] 4. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [2] https://review.opendev.org/c/openstack/governance-sigs/+/809887 [3] https://etherpad.opendev.org/p/tc-xena-tracker [4] https://review.opendev.org/q/projects:openstack/governance+status:open [5] https://review.opendev.org/c/openstack/governance/+/810037 [6] https://review.opendev.org/c/openstack/governance/+/810721 [7] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/02501… [8] https://review.opendev.org/c/openstack/project-team-guide/+/810766 [9] https://governance.openstack.org/tc/reference/tags/index.html [10] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/02480… [11] https://review.opendev.org/c/openstack/governance/+/809011 [12] https://review.opendev.org/c/openstack/governance/+/798540 [13] https://review.opendev.org/c/openstack/governance/+/807163 [14] https://etherpad.opendev.org/p/y-series-goals [15] https://review.opendev.org/c/openstack/governance/+/803783 [16] https://etherpad.opendev.org/p/tc-yoga-ptg [17] http://lists.openstack.org/pipermail/openstack-discuss/2021-June/023000.html [18] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [19] http://eavesdrop.openstack.org/#Technical_Committee_Meeting [20] http://eavesdrop.openstack.org/#Technical_Committee_Office_hours -gmann

1 0

[ops][neutron] Is it possible to "lock" a floating IP to an instance ?
by Massimo Sgaravatto 24 Sep '21

24 Sep '21

Hello I have the following use case: A user creates a VM and associates a floating IP to such instance Is in some way possible to prevent that the floating IP is disassociated from that instance by another user of the same project ? If it helps, the user owning the instance could be admin (but allowing only the admin user to manage floating IPs is not an option) Thanks, Massimo

2 2

[Sahara]Currently about the development of the Sahara community there are some points
by Juntingqiu Qiujunting (邱军婷) 24 Sep '21

24 Sep '21

Hi all: Currently about the development of the Sahara community there are some points as following: 1. About the schedule of the regular meeting of the Sahara project? What is your suggestion? How about the regular meeting time every Wednesday afternoon 15:00 to 16:30? 2. Regarding the Sahara project maintenance switch from StoryBoard to launchpad. https://storyboard.openstack.org/ https://blueprints.launchpad.net/openstack/ The reasons are as follows: 1. OpenSatck core projects are maintained on launchpad, such as nova, cinder, neutron, etc. 2. Most OpenStack contributors are used to working on launchpad. 3. Do you have any suggestions? If you think this is feasible, I will post this content in the Sahara community later. Thank you for your help. Thank you Fossen. --------------------------------- Fossen Qiu | 邱军婷 CBRD | 云计算与大数据研发部 T: 18249256272 E: qiujunting(a)inspur.com<mailto:qiujunting@inspur.com> 浪潮云海

3 2

Neutron FIP Qos with OVN Backend
by Ammad Syed 24 Sep '21

24 Sep '21

Hello, I am using Neutron wallaby with OVN backend. I am testing QoS FIP and it works fine. I have created a QoS policy and attached it with FIP, It works fine. Is it possible to set one of the QoS policy as default that each time I acquire a floating IP, the QoS should be attached to it by default ? Ammad

4 4

[devstack][requirements][stable/ussuri] New setuptools doesn't like suds-jurko?
by Neil Jerram 24 Sep '21

24 Sep '21

I use the stable/ussuri branches of devstack and tempest and have recently started seeing this stack.sh failure: Collecting suds-jurko===0.6 Downloading suds-jurko-0.6.tar.bz2 (143 kB) ERROR: Command errored out with exit status 1: command: /usr/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-wsiug8gg/suds-jurko/setup.py'"'"'; __file__='"'"'/tmp/pip-install-wsiug8gg/suds-jurko/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-h8s2x8w4 cwd: /tmp/pip-install-wsiug8gg/suds-jurko/ Complete output (1 lines): error in suds-jurko setup command: use_2to3 is invalid. IIUC that last error is from setuptools, specifically version 58.0.4. In previous runs with setuptools 57.4.0, this error did not occur. Does it mean that stable/ussuri global-requirements.txt for setuptools should have "<58" ? I haven't investigated why suds-jurko is pulled in, but I guess it's from the reference Neutron code - as opposed to something that my particular testing has added - and so this could affect anyone using stable/ussuri devstack. Best wishes, Neil -- Neil Jerram Principal Software Engineer Tigera neil(a)tigera.io | @neiljerram <http://twitter.com/neiljerram> Follow Tigera: Blog <https://blog.tigera.io/> | Twitter <https://twitter.com/tigeraio> | LinkedIn <https://www.linkedin.com/company/tigera/> Leader in Kubernetes Security and Observability <https://tigera.io/>

2 2

[release][qa][requirements][i18n] time to finalize stable branch setup
by Előd Illés 24 Sep '21

24 Sep '21

Hi, All the projects that are enabled in devstack by default have been branched, so it's time to - [requirements]: create branch for 'requirements' to be able to lift the requirements freeze - [qa]: do the necessary QA preparation patches ( https://wiki.openstack.org/wiki/QA/releases <https://wiki.openstack.org/wiki/QA/releases> ) - [i18n]: update the translation tools for the new stable series So I ask the PTLs of these teams to do the above necessary preparation. Thanks, Előd

1 0

[ironic] Yoga PTG
by Iury Gregory 23 Sep '21

23 Sep '21

Hello Ironicers! We are approaching the time of the next PTG (it's in October but we need to make some decisions). Our yoga etherpad is available[1], feel free to add any topics we shall discuss. In the ethercalc[2] you can find the slots we already booked, we may add more slots for APAC if someone is willing to run the meeting and discussions. Don't forget to register! [3]. [1] https://etherpad.opendev.org/p/ironic-yoga-ptg [2] https://ethercalc.openstack.org/8tum5yl1bx43 [3] https://openinfra-ptg.eventbrite.com -- *Att[]'sIury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Part of the ironic-core and puppet-manager-core team in OpenStack* *Software Engineer at Red Hat Czech* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory(a)gmail.com <iurygregory(a)gmail.com>*

1 1

[neutron] Drivers meeting - Friday 24.09.2021 - cancelled
by Lajos Katona 23 Sep '21

23 Sep '21

Hi Neutron Team, Due to lack of agenda, let's cancel tomorrow's drivers meeting. Let's concentrate on the ongoing release and see You on the meeting next week :-) Lajos Katona (lajoskatona)

1 0

Re: RabbitMQ annoying disconnections [kolla-ansible, rabbitmq]
by bkslash 23 Sep '21

23 Sep '21

Hi, I’ve tried to set heartbeat_in_pthread = true, but nothing has changed after kolla-ansible reconfigure - still got disconnections and heartbeat_in_pthread option was present inside configs on all nodes. But in the meantime I tried to alternate rabbit configuration and I realized that Kolla ignored /etc/kolla/config/rabbitmq.conf file (of course at first I checked permissions, etc.) So I tried to put heartbeat = 600 inside /etc/kolla/config/rabbitmq/rabbitmq.conf - this overwrote rabbitmq.conf totally (I mean Kolla didn’t just add heartbeat = 600 to rabbitmq.conf on each controller node, but put ONLY this line in all rabbitmq.conf files ! - rest of the file was cleared). So finally I tried to create separate config for each controller node. So I copied default content of /etc/kolla/rabbitmq/rabbitmq.conf on each controller (because of course these files differs from each other - they have different IPs inside) to /etc/kolla/config/rabbitmq/controller1(2,3)/rabbitmq.conf And finally heartbeat = 600 started to work - no more disconnections. I left Oslo heartbeats and timeouts intact (with the values I’ve set previously) and I don’t have disconnections from services side too. So there are two questions: 1. Why heartbeat_in_pthread didn’t change situation? And if it’s deprecated it was kind of risky to use it long term (after upgrade to wallaby it would stop working anyway?) 2. Why Kolla-ansible ignored rabbitmq.conf file if it is located in /etc/kolla/config, and overwrites ALL content of rabbitmq.conf file when it is located in /etc/kolla/config/rabbitmq/ ? Best regards Adam Tomas > Wiadomość napisana przez Herve Beraud <hberaud(a)redhat.com <mailto:hberaud@redhat.com>> w dniu 21.09.2021, o godz. 08:12: > > > > Le sam. 18 sept. 2021 à 16:12, bkslash <bkslash(a)poczta.onet.pl <mailto:bkslash@poczta.onet.pl>> a écrit : > Hi Ben, > Thank you for the answer. I’m using open stack Victoria deployed with Kolla-ansible. > So I assume I should add (in /etc/kolla/config/global.conf) > > [oslo_messaging_rabbit] > heartbeat_in_pthread = True > > > But for Victoria (nova sample configuration file) it seems to be deprecated? > # DEPRECATED: Run the health check heartbeat thread through a native python > # thread by default. If this option is equal to False then the health check > # heartbeat will inherit the execution model from the parent process. For > # example if the parent process has monkey patched the stdlib by using > # eventlet/greenlet then the heartbeat will be run through a green thread > # (boolean value) > # This option is deprecated for removal. > # Its value may be silently ignored in the future. > #heartbeat_in_pthread = true > So will it work? > > > Yes it will work. > > Few months ago I proposed to remove this deprecation and to keep the option available [1], unfortunately my removal isn't yet merged. > Only the default value is changed and the option will be turned on by default. > > [1] https://review.opendev.org/c/openstack/oslo.messaging/+/800621 <https://review.opendev.org/c/openstack/oslo.messaging/+/800621> > > Best regards > Adam Tomas > > >> Wiadomość napisana przez Ben Nemec <openstack(a)nemebean.com <mailto:openstack@nemebean.com>> w dniu 17.09.2021, o godz. 18:57: >> >> This sounds like you're hitting the heartbeat/eventlet bug. Basically eventlet won't schedule the heartbeat thread unless something is regularly waking up the api threads. Depending on what release you're running, you may be able to set heartbeat_in_pthread[0] to fix this. >> >> 0: https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messagi… <https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messagi…> >> >> On 9/17/21 7:00 AM, Adam Tomas wrote: >>> Hi, >>> after some struggling I have almost „clear” logs (clear=error free :) ) Almost…. RabbitMQ keeps disconnecting sessions and there is a huge amount of disconnect errors in all logs (see below). I found this bug description: >>> https://bugzilla.redhat.com/show_bug.cgi?id=1711794 <https://bugzilla.redhat.com/show_bug.cgi?id=1711794> <https://bugzilla.redhat.com/show_bug.cgi?id=1711794 <https://bugzilla.redhat.com/show_bug.cgi?id=1711794>> >>> in which we can read as follows: "this is a recoverable issue that is already handled by how oslo.messaging is designed. disconnection is not an error and should not be reported as such in the logs.” >>> but… It is reported :( And produces tons of logs. >>> I tried to modify heartbeat values - helped a little bit, but I had to increase [database] max_pool_size = 5 and that of course multiplied number of disconnection errors by 5 :( >>> [oslo_messaging_rabbit] >>> heartbeat_timeout_threshold = 720 heartbeat_interval = 360 heartbeat_rate = 4 >>> How to avoid this disconnections?? Or what to do to avoid writing this „errors”to logs? >>> Best regards >>> Adam Tomas >>> /var/log/kolla/rabbitmq/rabbitXXX.log >>> 2021-09-17 06:43:08.771 [error] <0.16056.678> closing AMQP connection <0.16056.678> (x.x.x.x00:44882 -> x.x.x.x00:5672 - mod_wsgi:959:aaa12af5-bb0b-456e-a129-a466eccc24f0): >>> missed heartbeats from client, timeout: 60s >>> 2021-09-17 06:43:08.773 [info] <0.7412.679> Closing all channels from connection 'x.x.x.x00:44882 -> x.x.x.x00:5672' because it has been closed >>> 2021-09-17 06:43:09.282 [info] <0.4574.679> accepting AMQP connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.283 [info] <0.28410.678> accepting AMQP connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.285 [info] <0.28410.678> Connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:954:c05a1221-6915-4414-ba93-7db5450ff394 >>> 2021-09-17 06:43:09.285 [info] <0.4574.679> Connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:967:08f29ce5-d6ee-43ae-87a9-e8c8856f8c93 >>> 2021-09-17 06:43:09.286 [info] <0.4574.679> connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672 - magnum-conductor:967:08f29ce5-d6ee-43ae-87a9-e8c8856f8c93): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.286 [info] <0.28410.678> connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672 - magnum-conductor:954:c05a1221-6915-4414-ba93-7db5450ff394): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.504 [error] <0.14465.678> closing AMQP connection <0.14465.678> (x.x.x.x00:44914 -> x.x.x.x00:5672 - mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3): >>> missed heartbeats from client, timeout: 60s >>> 2021-09-17 06:43:09.507 [info] <0.14985.679> Closing all channels from connection 'x.x.x.x00:44914 -> x.x.x.x00:5672' because it has been closed >>> 2021-09-17 06:43:09.521 [info] <0.11574.678> accepting AMQP connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.523 [info] <0.11574.678> Connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672) has a client-provided name: mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3 >>> 2021-09-17 06:43:09.525 [info] <0.11574.678> connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672 - mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.654 [info] <0.31575.679> accepting AMQP connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.656 [info] <0.31575.679> Connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:7:5a1cc3cc-150d-417a-9652-1279597c3db7 >>> 2021-09-17 06:43:09.656 [info] <0.31180.678> accepting AMQP connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.658 [info] <0.31575.679> connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672 - magnum-conductor:7:5a1cc3cc-150d-417a-9652-1279597c3db7): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.658 [info] <0.31180.678> Connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:954:c8deffa4-7d67-46be-9b9d-09fa20544a2b >>> 2021-09-17 06:43:09.660 [info] <0.2757.679> accepting AMQP connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.660 [info] <0.31180.678> connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672 - magnum-conductor:954:c8deffa4-7d67-46be-9b9d-09fa20544a2b): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.662 [info] <0.2757.679> Connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:7:cebf81c9-95f6-48fa-8bde-30c01243d315 >>> 2021-09-17 06:43:09.663 [info] <0.2757.679> connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672 - magnum-conductor:7:cebf81c9-95f6-48fa-8bde-30c01243d315): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.814 [error] <0.22797.678> closing AMQP connection <0.22797.678> (x.x.x.x05:57676 -> x.x.x.x00:5672 - neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e): >>> missed heartbeats from client, timeout: 60s >>> 2021-09-17 06:43:09.817 [info] <0.7903.679> Closing all channels from connection 'x.x.x.x05:57676 -> x.x.x.x00:5672' because it has been closed >>> 2021-09-17 06:43:10.319 [error] <0.18268.678> closing AMQP connection <0.18268.678> (x.x.x.x00:44924 -> x.x.x.x00:5672 - nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09): >>> missed heartbeats from client, timeout: 60s >>> 2021-09-17 06:43:10.322 [info] <0.28032.678> Closing all channels from connection 'x.x.x.x00:44924 -> x.x.x.x00:5672' because it has been closed >>> 2021-09-17 06:43:10.330 [info] <0.19596.678> accepting AMQP connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:10.333 [info] <0.19596.678> Connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672) has a client-provided name: nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09 >>> 2021-09-17 06:43:10.334 [info] <0.19596.678> connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672 - nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:10.831 [info] <0.9838.678> accepting AMQP connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:10.834 [info] <0.9838.678> Connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672) has a client-provided name: neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e >>> 2021-09-17 06:43:10.835 [info] <0.9838.678> connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672 - neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:11.738 [error] <0.15275.678> closing AMQP connection <0.15275.678> (x.x.x.x01:53414 -> x.x.x.x00:5672 - mod_wsgi:968:b60f723c-882e-4780-a162-55c03ebe6179): >>> missed heartbeats from client, timeout: 60s >>> /var/log/kolla/glance/glance-api.log >>> 2021-09-17 13:17:23.212 958 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:17:29.208 955 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:17:42.190 957 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:17:42] "GET / HTTP/1.1" 300 1446 0.001949 >>> 2021-09-17 13:17:46.351 957 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:18:12.336 955 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:18:12] "GET / HTTP/1.1" 300 1446 0.003754 >>> 2021-09-17 13:18:13.703 956 INFO eventlet.wsgi.server [req-1a00db34-349b-4ed3-8625-edfadc939be4 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:13] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.139229 >>> 2021-09-17 13:18:33.065 955 INFO eventlet.wsgi.server [req-a57046eb-549e-4837-95c7-d0994c05449e d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:33] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.096302 >>> 2021-09-17 13:18:33.077 955 INFO eventlet.wsgi.server [req-3a8af281-6732-4afd-b3ec-e502367bce51 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:33] "GET /v2/schemas/image HTTP/1.1" 200 6259 0.003982 >>> 2021-09-17 13:18:36.747 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:18:36.759 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:18:42.460 957 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:18:42] "GET / HTTP/1.1" 300 1446 0.001383 >>> 2021-09-17 13:18:52.659 956 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:18:52.670 956 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:19:12.598 955 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:19:12] "GET / HTTP/1.1" 300 1446 0.001590 >>> 2021-09-17 13:19:16.367 957 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:19:27.127 955 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:19:27.184 955 INFO eventlet.wsgi.server [req-8f68efc9-f46a-450d-8138-928e4f2585b4 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:19:27] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.032917 >>> /var/log/kolla/keystone/keystone.log >>> 2021-09-17 13:13:54.670 958 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:13:55.148 960 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:13:55.150 962 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:13:55.345 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:13:56.044 961 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:18.402 966 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:31.372 963 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:36.580 967 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:38.771 965 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:16:12.226 964 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> /var/log/kolla/nova/nova-compute.log >>> 2021-09-17 13:10:07.710 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56226. >>> 2021-09-17 13:11:11.473 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:11:34.046 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:11:34.721 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:13:07.699 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:13:08.734 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56434. >>> 2021-09-17 13:14:11.501 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:34.070 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:34.745 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:16:08.724 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:16:09.756 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56638. >>> 2021-09-17 13:17:34.098 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:19:09.747 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:19:10.781 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56834. >>> 2021-09-17 13:20:34.124 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:20:34.777 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:20:51.789 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:22:10.772 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:22:11.806 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57038. >>> 2021-09-17 13:23:34.149 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:23:34.803 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:23:51.818 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:25:11.795 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:25:12.842 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57240. >>> 2021-09-17 13:26:34.176 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:26:34.830 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:28:12.812 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:28:13.848 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57446. > > > > -- > Hervé Beraud > Senior Software Engineer at Red Hat > irc: hberaud > https://github.com/4383/ <https://github.com/4383/> > https://twitter.com/4383hberaud <https://twitter.com/4383hberaud> >

4 4

[all][tc] Technical Committee next weekly meeting on Sept 23rd at 1500 UTC
by Ghanshyam Mann 22 Sep '21

22 Sep '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for Sept 23rd at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Sept 22th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

[stable] remove formal 'unmaintained' phase from EM to EOL transition
by Előd Illés 22 Sep '21

22 Sep '21

Hi stable maintainers! On the Xena Virtual PTG there was a discussion about 'Stable Extended Maintenance to End of Life transition' [1] and at some point we touched that the listed 'Unmaintained' phase [2] does not add value in the process (rather some confusion) so that was agreed that it should be removed (line 397). I've seen that the removal was not yet proposed so I created a patch [3] for it now. So this is just a warning for the change. Thanks, Előd [1] Line 342-397 https://etherpad.opendev.org/p/tc-xena-ptg [2] https://docs.openstack.org/project-team-guide/stable-branches.html#maintena… [3] https://review.opendev.org/c/openstack/project-team-guide/+/810499

1 0

[heat] Proposing Brendan Shephard for heat core
by Rico Lin 22 Sep '21

22 Sep '21

Hi all I would like to suggest to add Brendan Shephard(bshephar(a)redhat.com) to the heat core reviewer group. He has been working on heat actively. provide good quality reviews and commits. I think it will be great to have him as the heat core reviewer. *Rico Lin* OIF Individual Board of directors, OpenStack TC, Multi-arch SIG chair, Heat PTL, Senior Software Engineer@EasyStack

3 2

[cinder] Bug deputy report for week of 2021-22-09
by Sofia Enriquez 22 Sep '21

22 Sep '21

This is a bug report from 2021-08-09 to 2021-15-09. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- High - https://bugs.launchpad.net/cinder/+bug/1944118 " cinder-manage db version fails with: AttributeError: 'Engine' object has no attribute 'get_main_option'". Assigned to Stephen Finucane. Medium - https://bugs.launchpad.net/os-brick/+bug/1944474 "iSCSI connections are not reinitiated after host reboot". Assigned to Sophie Huang. - https://bugs.launchpad.net/cinder/+bug/1926412 "When the backup cannot find an available service, update the volume status to none". Assigned to ZhaoYixin - https://bugs.launchpad.net/cinder/+bug/1920729 "PowerStore driver reports no iSCSI targets found". Assigned to Ivan Pchelintsev. Low - https://bugs.launchpad.net/cinder/+bug/1924798 "NetApp ONTAP cannot initialize the driver with SVM scoped mode". Assigned to Felipe Rodrigues. - https://bugs.launchpad.net/cinder/+bug/1924643 "NetApp ONTAP cannot resize a volume LUN with a sub-clone". Assigned to Felipe Rodrigues. Invalid - https://bugs.launchpad.net/cinder/+bug/1922507 "cinder-volume-driver test fails "ModuleNotFoundError: No module named 'fixtures'" Thanks -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[monasca][cloudkitty] [kolla-ansible][victoria] metrics metadata
by Tomasz Rutkowski 22 Sep '21

22 Sep '21

Hi, perhaps I'm missing something - monasca returns metrics metadata when asked for measurements (/metrics/measurements) however cloudkitty asks only for aggregated data (/metrics/statistics). Is there some way to push cloudkitty to work with monasca also by the field-id mappings (metadata values)? with best regards -- Tomasz Rutkowski

1 0

[Outreachy] Call for mentors, project submission by September 26
by Mahati Chamarthy 21 Sep '21

21 Sep '21

Hello! *TL;DR OpenStack is participating in Outreachy for the upcoming December round!* *Please submit projects as soon as possible, the deadline is September 26th, 4 pm UTC:* *https://www.outreachy.org/communities/cfp/openstack/ <https://www.outreachy.org/communities/cfp/openstack/>* Outreachy's goal is to support people from groups underrepresented in the technology industry. Interns will work remotely with mentors from our community. We are seeking mentors to propose projects that Outreachy interns can work on during their internship. If you or your colleagues are interested in mentoring in the next round, please submit your project proposal by *September 26th,* *2021 *over here https://www.outreachy.org/communities/cfp/openstack/ Mentors should read the mentor FAQ https://www.outreachy.org/mentor/mentor-faq and find more details about the Outreachy program and timeline in https://www.outreachy.org/communities/cfp/ . If you want help crafting your project proposal, please contact me Mahati Chamarthy <mahati.chamarthy(a)gmail.com> or Samuel Queiroz < samueldmq(a)gmail.com> and we will be glad to assist you. Thank you, Samuel Queiroz

1 1

[requirements][release][ironic] Exception for ironic deliverables - Xena
by Iury Gregory 21 Sep '21

21 Sep '21

Hello everyone, The ironic CI last week was in a very bad shape and we couldn't merge some patches that we wanted to have in the Xena release, we knew last week was the RC1 and we should have proposed the release for our derivables. Today we were able to merge a workaround to get our CI green again, we will probably have all the required patches merged by EOD or tomorrow morning. I would like to ask for an exception so we can release our deliverables tomorrow. Thanks! -- *Att[]'sIury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Part of the ironic-core and puppet-manager-core team in OpenStack* *Software Engineer at Red Hat Czech* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory(a)gmail.com <iurygregory(a)gmail.com>*

3 2

Constant HA Virtual Router Flapping
by Justin Moravec 21 Sep '21

21 Sep '21

Any ideas welcome, Kolla-ansible Ussuri deployment. Have some projects that are using virtual routers that are in HA setup that are constantly flapping (flipping between one neutron node to another) causing disconnects. This only happens in SOME projects, other projects with HA routers are working as intended. This only affects HA router deployments, non HA routers work as intended no matter the neutron node they are on. Have verified that each virtual router can reach the external network gateway while it is on each of the neutron nodes. Thanks, Justin

1 0

[neutron] Bug deputy report (Sep 13 - 19)
by Akihiro Motoki 21 Sep '21

21 Sep '21

Hi, Here is my bug deputy report last week. Sorry for late. [Summary] * There are several RC2 candidate bugs. I added a DVR one to Xena RC2 milestone. * dynamic-routing and vpnaas bugs need to be investigated. I would be nice if someone can look into the dynamic-routing one. Regarding vpnaas ones, unfortunately I had no time to check them last week but I will visit them this week. [RC candidates] * https://bugs.launchpad.net/neutron/+bug/1943714 DB session commit error in resource_registry.set_resources_dirty In Progress, High, Fix already approved * https://bugs.launchpad.net/neutron/+bug/1943930 [DHCP] new line symbol in opt_name of extra_dhcp_opt causes dnsmasq to fail In Progress, High, Fix already approved * https://bugs.launchpad.net/neutron/+bug/1943846 Create floating agent gw port for DVR fails In Progress, High, Fix already approved NOTE: It is caused by the change in the notification payload during Xena, so I targeted to Xena-RC2. [QoS] * https://bugs.launchpad.net/neutron/+bug/1943724 Change a QoS policy on a bound port does not handle minimum bandwidth direction change In Progress, Medium NOTE: A fix is proposed but it is in the feature patch series of the min pps rule. Hopefully someone familiar with QoS can check the dependency of the fix. [dynamic-routing] * https://bugs.launchpad.net/neutron/+bug/1943725 Automatic cleanup of BGP speakers is too aggressive New, Unassigned [VPNaaS] * https://bugs.launchpad.net/neutron/+bug/1943449 VPNaaS reconfiguration creates duplicate IPtables rules causes the VPN connection to remain DOWN New, Unassigned * https://bugs.launchpad.net/neutron/+bug/1943716 vpn services / vpn connections are stuck in PENDING CREATE New, Unassigned NOTE: RabbitMQ topic mismatch is reported here. It is interesting. I will check later this week. Thanks, Akihiro Motoki (irc: amotoki)

1 0

[release] Release countdown for week R-2, Sep 20 - Sep 24
by Herve Beraud 20 Sep '21

20 Sep '21

Development Focus ----------------- At this point we should have release candidates (RC1 or recent intermediary release) for all the Xena deliverables. Teams should be working on any release-critical bugs that would require another RC or intermediary release before the final release. Actions ------- Early in the week, the release team will be proposing stable/xena branch creation for all deliverables that have not branched yet, using the latest available Xena release as the branch point. If your team is ready to go for creating that branch, please let us know by leaving a +1 on these patches. If you would like to wait for another release before branching, you can -1 the patch and update it later in the week with the new release you would like to use. By the end of the week the release team will merge those patches though, unless an exception is granted. Once stable/xena branches are created, if a release-critical bug is detected, you will need to fix the issue in the master branch first, then backport the fix to the stable/xena branch before releasing out of the stable/xena branch. After all of the cycle-with-rc projects have branched we will branch devstack, grenade, and the requirements repos. This will effectively open them up for Yoga development, though the focus should still be on finishing up Xena until the final release. For projects with translations, watch for any translation patches coming through and merge them quickly. A new release should be produced so that translations are included in the final Xena release. Finally, now is a good time to finalize release notes. In particular, consider adding any relevant "prelude" content. Release notes are targeted for the downstream consumers of your project, so it would be great to include any useful information for those that are going to pick up and use or deploy the Xena version of your project. Upcoming Deadlines & Dates -------------------------- Final RC deadline: September 30 (R-1 week) Final Xena release: October 6 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud

1 0