- openstack-discuss - lists.openstack.org

[all][tc] What's happening in Technical Committee: summary 1st Oct, 21: Reading: 5 min
by Ghanshyam Mann 01 Oct '21

01 Oct '21

Hello Everyone, Here is this week's summary of the Technical Committee activities. 1. TC Meetings: ============ * TC this week IRC meeting held on Sept 30th Thursday. * Most of the meeting discussions are summarized in the below sections (Completed or in-progress activities section). To know more details, you can check the complete logs @ https://meetings.opendev.org/meetings/tc/2021/tc.2021-09-30-15.00.log.html * We will have next week's video call meeting on Oct 7th, Thursday 15:00 UTC, feel free the topic in agenda [1] by Oct 6th. 2. What we completed this week: ========================= * Listed the 'Places for projects to spreading the word'[2] * Removed stable release 'Unmaintained' phase[3] 3. Activities In progress: ================== TC Tracker for Xena cycle ------------------------------ * TC is using the etherpad[4] for Xena cycle working item. We will be checking and updating the status biweekly on the same etherpad. * Current status is: 8 completed, 5 in-progress Open Reviews ----------------- * Five open reviews for ongoing activities[5]. Place to maintain the external hosted ELK, E-R, O-H services ------------------------------------------------------------------------- * We discussed about the technical possibility and the place to add the ELK services maintenance[6]. * As there is no other infra project than OpenStack has shown the interest to use and maintain it, we are disucssing where we can fit this in OpenStack. TACT SIG is one place we are leaning towards. We will continue to discuss it in next TC meeting and prepare some draft plan. Add project health check tool ----------------------------------- * No updates on this than previous week. * We are reviewing Rico proposal on collecting stats tool[7] and TODO of documenting the usage and interpretation of those stats. Stable Core team process change --------------------------------------- * Draft proposal resolution is still under review[8] . Feel free to provide early feedback if you have any. * Elod has raised more issues today[9], that a few projects stable change (even auto generated patch) are not merged and so does their stable branch doc site is not up. For now, we are fine to auto/single core approval of those auto-generated patches and proceed to make stable branch doc site up. Call for 'Technical Writing' SIG Chair/Maintainers ---------------------------------------------------------- * The technical writing SIG[10] provides documentation guidance, assistance, tooling, and style guides for OpenStack project teams. * As you might have read the email from Elod[9], Stephen who is current chair for this SIG is not planning to continue to chair. Please let us know if you are interested to help in this Doc TC tags analysis ------------------- * As discussed in the last PTG, TC is working on an analysis of the usefulness of the Tags framework[11] or what all tags can be cleaned up. * We are still waiting for the operator's response to the email on openstack-disscuss ML[12]. If you are an operator, please respond to the email and based on the feedback we will continue the discussion in PTG. Project updates ------------------- * Add the cinder-netapp charm to Openstack charms[13] * Retiring js-openstack-lib [14] * Retire puppet-freezer[15] Yoga release community-wide goal ----------------------------------------- * Please add the possible candidates in this etherpad [16]. * Current status: "Secure RBAC" is selected for Yoga cycle[17]. PTG planning ---------------- * We are collecting the PTG topics in etherpad[18], feel free to add any topic you would like to discuss. * We discussed the live stream of one of the TC PTG sessions like we did last time. Once we will have more topics in etherpad then we can select the appropriate one. Test support for TLS default: ---------------------------------- * Rico has started a separate email thread over testing with tls-proxy enabled[19], we encourage projects to participate in that testing and help to enable the tls-proxy in gate testing. 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[20]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15 UTC [21] 3. Office hours: The Technical Committee offers a weekly office hour every Tuesday at 0100 UTC [22] 4. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [2] https://docs.openstack.org/project-team-guide/spread-the-word.html [3] https://review.opendev.org/c/openstack/project-team-guide/+/810499 [4] https://etherpad.opendev.org/p/tc-xena-tracke [5] https://review.opendev.org/q/projects:openstack/governance+status:open [6] https://etherpad.opendev.org/p/elk-service-maintenance-plan [7] https://review.opendev.org/c/openstack/governance/+/810037 [8] https://review.opendev.org/c/openstack/governance/+/810721 [9] http://lists.openstack.org/pipermail/openstack-discuss/2021-October/025161.… [10] https://governance.openstack.org/sigs/ [11] https://governance.openstack.org/tc/reference/tags/index.html [12] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/02480… [13] https://review.opendev.org/c/openstack/governance/+/809011 [14] https://review.opendev.org/c/openstack/governance/+/798540 [15] https://review.opendev.org/c/openstack/governance/+/807163 [16] https://etherpad.opendev.org/p/y-series-goals [17] https://review.opendev.org/c/openstack/governance/+/803783 [18] https://etherpad.opendev.org/p/tc-yoga-ptg [19] http://lists.openstack.org/pipermail/openstack-discuss/2021-June/023000.html [20] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [21] http://eavesdrop.openstack.org/#Technical_Committee_Meeting [22] http://eavesdrop.openstack.org/#Technical_Committee_Office_hours

1 0

[release] Release countdown for week R-0, Oct 4 - Oct 8
by Herve Beraud 01 Oct '21

01 Oct '21

Development Focus ----------------- We will be releasing the coordinated OpenStack Xena release next week, on October 6, 2021. Thanks to everyone involved in the Xena cycle! We are now in pre-release freeze, so no new deliverable will be created until final release, unless a release-critical regression is spotted. Otherwise, teams attending the virtual PTG should start to plan what they will be discussing there! General Information ------------------- On release day, the release team will produce final versions of deliverables following the cycle-with-rc release model, by re-tagging the commit used for the last RC. A patch doing just that will be proposed soon. PTLs and release liaisons should watch for that final release patch from the release team. While not required, we would appreciate having an ack from each team before we approve it on the 16th, so that their approval is included in the metadata that goes onto the signed tag. Upcoming Deadlines & Dates -------------------------- Final Xena release: October 6 Yoga PTG: October 18-22 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud

1 0

[oslo] oslo.metrics package for Fedora
by Hirotaka Wakabayashi 01 Oct '21

01 Oct '21

Hello Oslo Team, I am a Fedora packager. I want to package oslo.metrics for Fedora because my package uses oslo.messaging that requires oslo.metrics as you know. oslo.messaging package repository already exists in Fedora. I will take over it from the former package maintainer. the oslo.metrics repository doesn't existso I need to make it. If any concerns with it, please reply. I can update the version as soon as the new version releases by using Fedora's release monitoring system. Thanks in advance, Hirotaka Wakabayashi

4 4

[glance] No meeting on 07th October
by Abhishek Kekane 30 Sep '21

30 Sep '21

Hi All, We decided to cancel our next (October 7th) weekly meeting. According to schedule we will meet directly on October 14th. In case of any queries, reach us on #openstack-glance IRC channel. Thank you, Abhishek

1 0

[cinder][nova][requirements] RFE requested for os-brick
by Brian Rosmaita 30 Sep '21

30 Sep '21

Hello Requirements Team, The Cinder team recently became aware of a regression in os-brick that prevents Nova from attaching NVMe volumes [0]. A fix has been merged to master (yoga development) [1] and has been backported to stable/xena [2]. We've proposed a release of os-brick 5.0.1 from stable/xena [3], and are petitioning for an Requirements Freeze Exception to include os-brick 5.0.1 in the xena release. Thank you for your attention to this matter. [0] https://bugs.launchpad.net/os-brick/+bug/1945323 [1] https://review.opendev.org/c/openstack/os-brick/+/811944 [2] https://review.opendev.org/c/openstack/os-brick/+/812010 [3] https://review.opendev.org/c/openstack/releases/+/812048

2 1

[neutron] Drivers meeting agenda - 01.10.2021
by Lajos Katona 30 Sep '21

30 Sep '21

Hi, The agenda for tomorrow's drivers meeting is at [1]. We have 1 RFE to discuss: * https://bugs.launchpad.net/neutron/+bug/1885261 Add stateless firewall support to OVS firewall ** It is Liu who would like to implement this, see logs of last team meeting at [2] ** Liu can't participate as there's some celebration in China, and he is on holiday, but still we can discuss it to have better understanding. [1] https://wiki.openstack.org/wiki/Meetings/NeutronDrivers#Agenda [2] https://meetings.opendev.org/meetings/networking/2021/networking.2021-09-28… See you at the meeting tomorrow. Lajos Katona (lajoskatona

1 0

Open stack Swift
by Midhunlal Nb 30 Sep '21

30 Sep '21

Hi team, I am using Openstack rocky setup with 1 controller node 2 compute node and 1 sStorage node,Now i am planning to configure a storage backup service Swift. Where do I need to configure swift,It needs an extra Node?Please explain the requirements. Thanks & Regards Midhunlal N B +918921245637

3 2

how dpdk improve guest performance
by Satish Patel 30 Sep '21

30 Sep '21

Folks, If I am running dpdk in my compute nodes then sure it will reduce kernel overhead but how does it help to improve guest network performance? Inside guest vm it's going to use kernel to copy packet to userspace. in my example we run high performance haproxy which handles 300k to 500k connections but I have noticed performance is poor and lots of packet drops. If I run the same workload on SRIOV vms it works fine and I notice less kernel context switching inside the guest VM. I think when you run DPDK in compute then it only help compute but not your guest network performance. How do I improve guest performance also with dpdk based compute?

1 0

[nova] WSGI Apache vs Eventlet HTTP Servers
by Ammad Syed 30 Sep '21

30 Sep '21

Hi, I was reviewing below document. https://docs.openstack.org/nova/wallaby/user/wsgi.html Is apache2 wsgi provide better performance then eventlet http servers for nova api ? If yes, can you guys please share the sample or reference apache-nova-api file for wsgi ? - Ammad

1 0

[GPU] Request for suggestions for OpenStack deployment
by Jay See 30 Sep '21

30 Sep '21

Hi, We are planning to buy some servers with GPU cards from DELL(R740/R750)/HPE/Supermicro with NVIDIA A100 40GB (GPU). I have the following questions: 1. What is the best ratio for Memory to GPU? or It doesn´t matter. 2. Anyone using A100? A100 has any known issues with OpenStack deployment? 3. If you have set up with similar server with GPU, could you share the server configuration? It will help us in deciding / identifying the components we have missed. Thanks in advance. Regards, ~ Jayachander.

1 0

DC DR high availability - Snapshot taking feature and user experience
by KK CHN 29 Sep '21

29 Sep '21

List, I am in the process of setting up a DC and DR sites with openstack(ussuri, qemu kvm with debian as base OS). I would like to know what all are the options in DC and DR sites for High availability . What the architecture methods to follow at DC and DR so that, If VM crashes at one physical machine at DC ( for example 3 controllers in DC), or Host machine crashes( RAM, Hardware failure etc. ) / machine power cable detached accidentally so that no services down / VMs and applications not down. How can achieve this ? share the best practices here. Also If I take snapshots of the running VMs ( what will the user experience ? will they in freeze / logged out from applications right now they are logged in ? ) . Can we avoid the service unavailable for users while taking snapshots ? To DR we are rsyncing these snapshots after converting each VM image to qcow2 then rsyncing . ( All these we are performing on Controller node . Generally 100 GB VM will output 16 GB qcow2 image for example. ) its taking 2 minutes for snapshot creation and 20 to 30 Minutes for qcow2 conversion . ) How many snapshots and conversion to qcow2( qemu-img convert) can be performed on this controller machine where we performing this operation . ( Can we apply parallel processing for this and how ? ) . Seet he controller specs where we perform this. The controller spec ( 20 core CPUs with 1.5 TB RAM total 160 processors ) Then copying to DR ( which is 200 KM away from our DC takes 4 minutes with rsync). Then we populate each VM with these copied qcow2 images and attaching the Network ips and nat there. Is this the practice or any other good way to perform this for a robust DC and DR setup . Kindly share your thoughts. Regards, Kris

1 0

[all][tc] Technical Committee next weekly meeting on Sept 30th at 1500 UTC
by Ghanshyam Mann 29 Sep '21

29 Sep '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for Sept 30th at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Sept 29th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

DPDK Poor performance
by Satish Patel 29 Sep '21

29 Sep '21

Folk, I have deployed DPDK on one of my compute nodes to reply to my sriov deployment which doesn't support bonding. This is what I am doing to do a performance benchmark. Spun up VM with 8 vCPU and 8GB memory with single virtual nic. Then install nuttcp tool and start nuttcp -S (server mode) and then use Physical server outside my lab and use following command to blast some traffic nuttcp -T 30 -i -u -w4m -R 2G 10.69.7.130 I found 120kpps rate working fine but after that I noticed a packet drop on nuttcp. Here is the full details of my deployment and load-test result - https://paste.opendev.org/show/809675/ Small confusion related hugepage - I am seeing following in /dev/hugepages/ I believe rtemap_0 created by DPDK root@ovn-lab-comp-dpdk-1:~# ls -l /dev/hugepages/rtemap_0 -rw------- 1 root root 1073741824 Sep 29 02:28 /dev/hugepages/rtemap_0 Why is the qemu instance file empty? Does that mean it's not using a huge page for dpdk performance? root@ovn-lab-comp-dpdk-1:~# ls -l /dev/hugepages/libvirt/qemu/1-instance-00000045/ total 0 I have hw:mem_page_size='large' setting in flavor so it should use hugepage. am i missing something?

1 0

[largescale-sig] Next meeting: Sept 29th, 15utc
by Thierry Carrez 29 Sep '21

29 Sep '21

Hi everyone, The Large Scale SIG meeting is back this Wednesday in #openstack-operators on OFTC IRC, at 15UTC. You can doublecheck how that time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20210929T15 A number of topics have already been added to the agenda, including discussing the final details of our next OpenInfra.Live show! Feel free to add other topics to our agenda at: https://etherpad.openstack.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

[kolla] Yoga PTG
by Michał Nasiadka 29 Sep '21

29 Sep '21

Hola Koalas! We are approaching the time of the next PTG. Kolla Yoga etherpad is available and initially populated[1], feel free to add any topics that shall be discussed and register your planned attendance. Don't forget to register! [2]. [1] https://etherpad.opendev.org/p/kolla-yoga-ptg [3] https://openinfra-ptg.eventbrite.com Best regards, Michal Nasiadka

1 0

[cinder] Bug deputy report for week of 2021-29-09
by Sofia Enriquez 29 Sep '21

29 Sep '21

This is a bug report from 2021-15-09 to 2021-29-09. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Medium - https://bugs.launchpad.net/os-brick/+bug/1943977 "os_bricks fails to rerty "iscsiadm -m session" when iser_use_multipath and iscsi_use_multipath are set in nova.conf". Assigned to Sophie Huang. - https://bugs.launchpad.net/os-brick/+bug/1945323 "Can't attach NVMe volume" Low - https://bugs.launchpad.net/cinder/+bug/1945169 " [swift] volume backup's status keep in 'restoring' and Lost connection to MySQL server during query". Assigned to silphon zhang. Wishlist - https://bugs.launchpad.net/cinder/+bug/1945249 '[charm] no upgrade path to cinder-lvm subordinate'. Unassigned. Cheers, -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[Cinder] [Nova] [Interop] "compute-volume-list" API guideline
by Vida Haririan 29 Sep '21

29 Sep '21

Hi Cinder and Nova teams, The Interop team is reviewing all related open stories to determine next steps. We researched the current status of 1638112:“os-volumes should be flagged for removal in the compute API guidelines“ [1] and noticed that "compute-volume-list" was removed in 2017.01 guidelines [2][3]. However, two related tests still remain in tempest [4][5] and we are reaching out to determine if the API is still in use or has been moved/renamed. Please let us know if you have any questions. [1] https://storyboard.openstack.org/#!/story/1638112 [2] https://opendev.org/osf/interop/src/commit/dd7ae2660de5daa82ab4fe080e875d63… [3] https://opendev.org/osf/interop/src/branch/master/guidelines/next.json [4] https://opendev.org/openstack/tempest/src/commit/ae41052a51f5dbb748eb6bf4f2… [5] https://opendev.org/openstack/tempest/src/commit/ae41052a51f5dbb748eb6bf4f2… Thank you, -- *Vida Haririan* Senior Software Quality Engineer Red Hat EMEA <https://www.redhat.com/>

2 1

[announce][debian][xena] general availability of OpenStack Xena in Debian
by Thomas Goirand 29 Sep '21

29 Sep '21

Hi! It's my pleasure to announce the general availability of OpenStack Xena in Debian. Indeed, I've just finished uploading everything to Debian Experimental at the beginning of this week, and was able yesterday, to pop my first VM over on top of Xena. This marks the first readiness of the release, from the Debian point of view (even though there probably will be things to fix, like after addressing https://bugs.launchpad.net/oslo.policy/+bug/1945336 for example, and uploading the final release to replace the RCs when that becomes available). The Bullseye backports are available the usual way, for example using extrepo: apt-get install extrepo extrepo enable openstack_xena apt-get update ... or directly setting-up the http://bullseye-xena.debian.net repository the usual way. I'm currently in the process of uploading all of Xena to Debian Unstable (but please bare with me: there's 225+ packages in this update). Note that Xena can already be installed by OCI [1]. Please report any issue you may find, on OCI or on the Debian packages, through the Debian BTS as usual. Cheers, Thomas Goirand (zigo) [1] https://salsa.debian.org/openstack-team/debian/openstack-cluster-installer

1 0

[kolla][neutron][victoria] metering agent on compute nodes
by Tomasz Rutkowski 29 Sep '21

29 Sep '21

Hi All, I have issue with neutron-metering-agent - on network node it works ok, however on compute nodes the list of routers is constantly empty (and l3-agent works normal)... how to debug why outside network node the metering-agent does not recieve list of routers? with best regards -- Tomasz Rutkowski

2 2

[all] Gate status: All devstack based jobs are failing for placement url issue in devstack
by Ghanshyam Mann 28 Sep '21

28 Sep '21

Hello Everyone, All the devstack based jobs are failing with placement URL issue in devstack. The root cause is not known yet. You will see the below error in devstack/summary log- + ./stack.sh:main:1455 : /usr/local/bin/nova-status --config-file /etc/nova/nova.conf upgrade check Placement logs: INFO placement.requestlog [None req-3ad7ab8b-7c85-477a-b156-f5ca652a38cc service placement] 10.208.226.48 "GET /placemen//" status: 404 len: 162 microversion: 1.0 Please do not recheck until the issue is fixed. -gmann

5 8

OpenInfra Live - September 30th at 9am CT
by Erin Disney 28 Sep '21

28 Sep '21

Hi everyone, This week’s OpenInfra Live episode is brought to you by the OpenStack and Ceph communities. Ceph is a highly scalable distributed-storage open source solution offering object, block, and file storage. Join us as various Community members discuss the basics, ongoing development and integration of OpenStack with Ceph. Date and time: September 30th at 9am CT (1400 UTC) You can watch us live on: YouTube: https://www.youtube.com/watch?v=zJVoleSpSOk LinkedIn: https://www.linkedin.com/video/event/urn:li:ugcPost:6846862917225320448/ Facebook: https://www.facebook.com/104139126308032/posts/4364096940312208/ WeChat: Recording will be posted on OpenStack WeChat after the live stream Speakers: Kendall Nelson (OpenInfra Foundation) John Fulton (Red Hat) Francesco Pantano (Red Hat) Tom Barron (Red Hat) Loan Harrouin (Societe Generale) Pawel Sadowski (OVH) Mohammed Naser (VEXXHOST) Have an idea for a future episode? Share it now at ideas.openinfra.live <https://openinfrafoundation.formstack.com/forms/openinfralive>. Register now for OpenInfra Live: Keynotes, a special edition of OpenInfra Live on November 17-18th starting at 1500 UTC: https://openinfralivekeynotes.eventbrite.com/ Thanks, Erin Erin Disney Event Marketing Open Infrastructure Foundation

1 0

Issue on accessing object storage containers from outside
by Michel Niyoyita 28 Sep '21

28 Sep '21

Dear all I have created containers and upload objects into containers . I can access these containers and objects trough horizon dashboard but once make them public I can not reach them through the provided link . If any one have an idea on how to access containers and their object using the link from outside he can help and guide. I deployed openstack wallaby using kolla-ansible and ceph pacific using ansible , both are using ubuntu 20.04 as OS Michel

3 4

[openstack-qa] opendev.org mirrors
by Eduardo Santos 28 Sep '21

28 Sep '21

Hi folks, On US-based servers, a DevStack deploy takes 15 minutes. When deploying it from Brazil (where I live) however, it takes more than an hour. Examining the stack.sh logs, git_timed appears to be the culprit. I'm using a 250 Mbps connection. Do opendev.org mirrors exist for other regions other than the US? If not, has this been discussed yet? I believe this is more of a financial discussion othet than technical, but I'm curious if other users have raised this question before. I'm not sure if openstack-qa is the most appropriate flag. If not, please feel free to correct. Thanks!

3 3

Creating multiple servers with a shared volume
by open infra 28 Sep '21

28 Sep '21

Hi, I am planning to spin hundreds of servers with a shared storage. I have created one instance and attached a 1 TB volume (multiattach) and created a snapshot of the instance. But when I spin another server using the snapshot, I noticed that instead of expected shared volume, individual 1 TB size additional (dedicated) volume has been attached to each VM. Is there a mechanism where I can create multiple instances along with a shared volume attached to the instance instead of attaching later? Regards, Danishka

3 5

[Tripleo] Support of Alma/Rocky Linux
by Anirudh Gupta 28 Sep '21

28 Sep '21

Hi Team As per the document, the supporting OS mentioned in the documentation is only Centos Stream and No Stream version (as of now) Since Centos has officially stopped releasing new versions and looking for EOL. I Just wanted to check if there is a possibility that we can install Tripleo on a specific version of Alma Linux or Rocky Linux Regards Anirudh Gupta

2 3

RE: Problems with Fluentd buffer [fluentd][kolla-ansible]
by Adam Tomas 28 Sep '21

28 Sep '21

Hi, I’ve turned off „central_logging” and now I have only several errors like below on all nodes (3-7 per day) 0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 04:49:32.728326628 +0200 chunk="5ccf11fdc0d6876abdef813211371285" error_class=RestClient::RequestTimeout error="408 Request Timeout” On compute node, which had errors like this : /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag=„openstack_python" I’ve restarted fluentd container on that node and now it’s not showing errors like this anymore. So what’s the problem? Performance of Monasca/Kibana/Elasticsearch? But the node (telemetry) on which this stack is running is not overloaded - 20-30% CPU (24 cores on that node) , 40% RAM (there is 64GB), a lot of space on SSD. Best regards Adam Tomas > Hi, > after some time of using newly deployed kolla-ansible I have fluentd errors: > > on all controllers, storage and network nodes: > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 01:40:46 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 01:40:47.847564807 +0200 chunk="5ccee7b1762157f0ae10dcbeb4e14fd8" error_class=RestClient::GatewayTimeout error="504 Gateway Timeout" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 04:49:31 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 04:49:32.728326628 +0200 chunk="5ccf11fdc0d6876abdef813211371285" error_class=RestClient::RequestTimeout error="408 Request Timeout” > > > on compute nodes: > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="openstack_python" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="kolla.var.log.kolla.monasca.agent-statsd.log" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"action"=>:throw_exception, "message"=>"failed to write data into buffer by buffer overflow action=:throw_exception"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"error"=>"#<Fluent::Plugin::Buffer::BufferOverflowError: buffer space has too many data>", "location"=>"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'", "tag"=>"openstack_python", "message"=>"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'\" tag=\"openstack_python\""} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"error"=>"#<Fluent::Plugin::Buffer::BufferOverflowError: buffer space has too many data>", "location"=>"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'", "tag"=>"kolla.var.log.kolla.monasca.agent-statsd.log", "message"=>"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'\" tag=\"kolla.var.log.kolla.monasca.agent-statsd.log\""} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="openstack_python" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="kolla.var.log.kolla.monasca.agent-forwarder.log" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"action"=>:throw_exception, "message"=>"failed to write data into buffer by buffer overflow action=:throw_exception"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data” > > and on monitoring node: > > 2021-09-27 14:40:50 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=84.05643947119825 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:42:53 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=122.65328024700284 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:44:15 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=82.32426812895574 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:46:05 +0200 [warn]: #0 Bulk sending messages to monasca-api threw exception exceptionew=#<RestClient::Exceptions::ReadTimeout: Timed out reading data from server> > 2021-09-27 14:46:05 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 14:46:06.042150289 +0200 chunk="5ccf89c9ec54b09e616f7167d5f93cc1" error_class=RestClient::Exceptions::ReadTimeout error="Timed out reading data from server" > 2021-09-27 14:46:05 +0200 [warn]: #0 suppressed same stacktrace > 2021-09-27 14:47:58 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=112.90601075813174 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:47:58 +0200 [warn]: #0 retry succeeded. chunk_id="5ccf89c9ec54b09e616f7167d5f93cc1" > 2021-09-27 14:50:16 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=136.24769522389397 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:52:27 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=129.86474119895138 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:54:08 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=100.71324555086903 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:56:31 +0200 [warn]: #0 Bulk sending messages to monasca-api threw exception exceptionew=#<RestClient::Exceptions::ReadTimeout: Timed out reading data from server> > >

2 1

key_repository does not exist or Keystone does not have sufficient permission to access it [kolla-ansible][keystone]
by Adam Tomas 27 Sep '21

27 Sep '21

Hi, some next problem with my kolla-ansible deployment. On controller nodes I have (several times per hour) error messages like below: ERROR keystone.common.fernet_utils [req-386e223e-5f08-4959-bd0b-3dbbfd3b534f 73c7a9a7175e46e9971181963267225b efa5d1ce80324476b4ce189686dda2d1 - default default] Either [None] key_repository does not exist or Keystone does not have sufficient permission to access it: /etc/keystone/credential-keys/ I’ve checked - such path doesn’t exist inside keystone containers. I found bug #1863643 (https://bugs.launchpad.net/kolla/+bug/1863643 <https://bugs.launchpad.net/kolla/+bug/1863643>) but I don’t see any progress on it nor information that victoria is not affected (so I assume it’s the same bug and Victoria is affected?). What should be done to get rid of this errors? Best regards Adam Tomas

1 0

[cinder] reminder: this week's meeting in video+IRC
by Brian Rosmaita 27 Sep '21

27 Sep '21

Quick reminder that this week's Cinder team meeting on Wednesday 29 September, being the final meeting of the month, will be held in both videoconference and IRC at the regularly scheduled time of 1400 UTC. These are the video meeting rules we've agreed to: * Everyone will keep IRC open during the meeting. * We'll take notes in IRC to leave a record similar to what we have for our regular IRC meetings. * Some people are more comfortable communicating in written English. So at any point, any attendee may request that the discussion of the current topic be conducted entirely in IRC. * The meeting will be recorded. connection info: https://bluejeans.com/3228528973 meeting agenda: https://etherpad.opendev.org/p/cinder-xena-meetings cheers, brian

1 0

[qa] Office Hour Sep 28th is cancelled
by Martin Kopec 27 Sep '21

27 Sep '21

Hi everyone, the next QA Office Hour on September 28th at 2pm UTC is cancelled due to a public holiday in my country and I won't be able to moderate it. Thank you for your understanding, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA

1 0

[OSSA-2021-002] Nova: Open Redirect in noVNC proxy (CVE-2021-3654)
by Jeremy Stanley 27 Sep '21

27 Sep '21

=========================================== OSSA-2021-002: Open Redirect in noVNC proxy =========================================== :Date: July 29, 2021 :CVE: CVE-2021-3654 Affects ~~~~~~~ - Nova: <21.2.3, >=22.0.0 <22.2.3, >=23.0.0 <23.0.3 Description ~~~~~~~~~~~ Swe Aung, Shahaan Ayyub, and Salman Khan with the Monash University Cyber Security team reported a vulnerability affecting Nova's noVNC proxying implementation which exposed access to a well-known redirect behavior in the Python standard library's http.server.SimpleHTTPRequestHandler and thus noVNC's WebSockifyRequestHandler which uses it. By convincing a user to follow a specially-crafted novncproxy URL, the user could be redirected to an unrelated site under control of the attacker in an attempt to convince them to divulge credentials or other sensitive data. All Nova deployments with novncproxy enabled are affected. Errata ~~~~~~ The initial fix did not take into account the possibility of bypass using exactly three slashes. This update provides a more thorough revised fix for the issue. The affected versions list has been updated to indicate versions expected to include the newer solution. Patches ~~~~~~~ - https://review.opendev.org/791807 (Train) - https://review.opendev.org/806629 (errata 1) (Train) - https://review.opendev.org/791806 (Ussuri) - https://review.opendev.org/806628 (errata 1) (Ussuri) - https://review.opendev.org/791805 (Victoria) - https://review.opendev.org/806626 (errata 1) (Victoria) - https://review.opendev.org/791577 (Wallaby) - https://review.opendev.org/805818 (errata 1) (Wallaby) - https://review.opendev.org/791297 (Xena) - https://review.opendev.org/805654 (errata 1) (Xena) Credits ~~~~~~~ - Swe Aung from Monash University Cyber Security team (CVE-2021-3654) - Shahaan Ayyub from Monash University Cyber Security team (CVE-2021-3654) - Salman Khan from Monash University Cyber Security team (CVE-2021-3654) References ~~~~~~~~~~ - https://launchpad.net/bugs/1927677 - https://bugs.python.org/issue32084 - http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3654 Notes ~~~~~ - The stable/train branch is under extended maintenance and will receive no new point releases, but a patch for it is provided as a courtesy. OSSA History ~~~~~~~~~~~~ - 2021-09-27 - Errata 1 - 2021-07-29 - Original Version -- Jeremy Stanley

1 0

Not registering Hypervisors
by Karera Tony 27 Sep '21

27 Sep '21

Hello Team, I have deployed Openstack Victoria using Kolla-ansible on Ubuntu 20.04 and ceph as the backend storage for Nova, Cinder and Glance. It finished with no error but it has failed to register any on the Compute Nodes under Hypervisors. Any idea on how to resolve this ? Regards Tony Karera

5 22

[neutron] upgrading port binding:profile.allocation value for port-resource-request-groups API extension
by Balazs Gibizer 27 Sep '21

27 Sep '21

Hi Neutrinos! We found a technical challenge during implementing the port-resource-request-groups API extension[1]. That extension changes the format of the port.resoruce_request as well as the format of the port.binding:profile.allocation. The former is a calculated field on the port so that is easy. However the bindig:profile is persisted in the database so data migration is needed. What is the canonical way to do such DB data translation in Neutron? Can we translate the data in place during alembic migration? Or should we do some kind of online data migration when the data is translated by neutron when it is read from the db? cheers, gibi [1] https://review.opendev.org/c/openstack/neutron/+/805637/5

2 6

Problems with Fluentd buffer [fluentd][kolla-ansible]
by Adam Tomas 27 Sep '21

27 Sep '21

Hi, after some time of using newly deployed kolla-ansible I have fluentd errors: on all controllers, storage and network nodes: /var/log/kolla/fluentd/fluentd.log:2021-09-27 01:40:46 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 01:40:47.847564807 +0200 chunk="5ccee7b1762157f0ae10dcbeb4e14fd8" error_class=RestClient::GatewayTimeout error="504 Gateway Timeout" /var/log/kolla/fluentd/fluentd.log:2021-09-27 04:49:31 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 04:49:32.728326628 +0200 chunk="5ccf11fdc0d6876abdef813211371285" error_class=RestClient::RequestTimeout error="408 Request Timeout” on compute nodes: /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="openstack_python" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="kolla.var.log.kolla.monasca.agent-statsd.log" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"action"=>:throw_exception, "message"=>"failed to write data into buffer by buffer overflow action=:throw_exception"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"error"=>"#<Fluent::Plugin::Buffer::BufferOverflowError: buffer space has too many data>", "location"=>"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'", "tag"=>"openstack_python", "message"=>"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'\" tag=\"openstack_python\""} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"error"=>"#<Fluent::Plugin::Buffer::BufferOverflowError: buffer space has too many data>", "location"=>"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'", "tag"=>"kolla.var.log.kolla.monasca.agent-statsd.log", "message"=>"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'\" tag=\"kolla.var.log.kolla.monasca.agent-statsd.log\""} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="openstack_python" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="kolla.var.log.kolla.monasca.agent-forwarder.log" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"action"=>:throw_exception, "message"=>"failed to write data into buffer by buffer overflow action=:throw_exception"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data” and on monitoring node: 2021-09-27 14:40:50 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=84.05643947119825 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:42:53 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=122.65328024700284 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:44:15 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=82.32426812895574 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:46:05 +0200 [warn]: #0 Bulk sending messages to monasca-api threw exception exceptionew=#<RestClient::Exceptions::ReadTimeout: Timed out reading data from server> 2021-09-27 14:46:05 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 14:46:06.042150289 +0200 chunk="5ccf89c9ec54b09e616f7167d5f93cc1" error_class=RestClient::Exceptions::ReadTimeout error="Timed out reading data from server" 2021-09-27 14:46:05 +0200 [warn]: #0 suppressed same stacktrace 2021-09-27 14:47:58 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=112.90601075813174 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:47:58 +0200 [warn]: #0 retry succeeded. chunk_id="5ccf89c9ec54b09e616f7167d5f93cc1" 2021-09-27 14:50:16 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=136.24769522389397 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:52:27 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=129.86474119895138 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:54:08 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=100.71324555086903 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:56:31 +0200 [warn]: #0 Bulk sending messages to monasca-api threw exception exceptionew=#<RestClient::Exceptions::ReadTimeout: Timed out reading data from server> 2021-09-27 14:56:31 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 14:56:32.140741991 +0200 chunk="5ccf89e1feb471b7ce25a26d37977baa" error_class=RestClient::Exceptions::ReadTimeout error="Timed out reading data from server" 2021-09-27 14:56:31 +0200 [warn]: #0 suppressed same stacktrace 2021-09-27 14:58:03 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=90.91785193886608 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:58:03 +0200 [warn]: #0 retry succeeded. chunk_id="5ccf89e1feb471b7ce25a26d37977baa" 2021-09-27 15:00:01 +0200 [info]: #0 detected rotation of /var/log/kolla/kafka/server.log; waiting 5 seconds 2021-09-27 15:00:01 +0200 [info]: #0 following tail of /var/log/kolla/kafka/server.log 2021-09-27 15:00:10 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=127.54797655600123 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 15:02:29 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=137.74220423400402 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 15:03:08 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=38.81670297612436 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 15:03:13 +0200 [info]: #0 detected rotation of /var/log/kolla/kafka/controller.log; waiting 5 seconds 2021-09-27 15:03:13 +0200 [info]: #0 following tail of /var/log/kolla/kafka/controller.log What seems to be the problem? Is there any way to run more than one fluentd worker in kolla? Best regards Adam Tomas

1 0

[keystone] Correct policies setup for System Administrators
by Jan Wasilewski 27 Sep '21

27 Sep '21

Hello, I am preparing policies configuration before an upgrade to the newer OpenStack release(Stein) and I would like to create a group of System Administrators to be able to get i.e. a list of all projects in the OpenStack cloud. I was following a description from this page [1] but it seems my admin user is able to get only a list of projects where it is directly added(i.e. with member role, reader role, or admin role). I am just wondering if we can list all of the OpenStack projects by System Administrator user without role reader added to every single project? To summarize what steps were done so far: - Original policy.json file which was used is here [2] - Only one option was changed so far: from: "identity:list_projects": "rule:cloud_admin or rule:admin_and_matching_domain_id", to: "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)", - Output for command: openstack role assignment list --system all --role member --role reader +----------------------------------+------+----------------------------------+---------+--------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +----------------------------------+------+----------------------------------+---------+--------+--------+-----------+ | e39e97c23bfe45d1a2f9689b6985f990 | | a0841b83f583477887219f27dd95477b | | | all | False | +----------------------------------+------+----------------------------------+---------+--------+--------+-----------+Shows only role reader, not role member, which is a bit strange if we compare with linked page above. But we have this in implied roles:openstack implied role list +----------------------------------+-----------------+----------------------------------+-------------------+ | Prior Role ID | Prior Role Name | Implied Role ID | Implied Role Name | +----------------------------------+-----------------+----------------------------------+-------------------+ | a3c7bb5d06884b048c1bfb4403b82b42 | admin | 3f20cb7be46346a8b2ba65c4684d50a3 | member | | a3c7bb5d06884b048c1bfb4403b82b42 | admin | 9fe2ff9ee4384b1894a90878d3e92bab | _member_ | | 3f20cb7be46346a8b2ba65c4684d50a3 | member | e39e97c23bfe45d1a2f9689b6985f990 | reader | +----------------------------------+-----------------+----------------------------------+-------------------+- Admin roles are grouped in a group ATM.Admin: openstack role assignment list --names --system all --role admin: +-------+---------------------+-------------------+---------+--------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +-------+---------------------+-------------------+---------+--------+--------+-----------+ | admin | | ATM.Admin@Default | | | all | False | | admin | admin@Default | | | | all | False | | admin | jwasilewski@Default | | | | all | False | +-------+---------------------+-------------------+---------+--------+--------+-----------+Just to be sure that IDs are linked, we can check it here:openstack role assignment list --system all --role admin +----------------------------------+----------------------------------+----------------------------------+---------+--------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +----------------------------------+----------------------------------+----------------------------------+---------+--------+--------+-----------+ | a3c7bb5d06884b048c1bfb4403b82b42 | | a0841b83f583477887219f27dd95477b | | | all | False | | a3c7bb5d06884b048c1bfb4403b82b42 | 19416fe5a2da45c88eb66c3aaf856c73 | | | | all | False | | a3c7bb5d06884b048c1bfb4403b82b42 | f42df418fd404d04b9bdabf2f1b49fd9 | | | | all | False | +----------------------------------+----------------------------------+----------------------------------+---------+--------+--------+-----------+ So by linking roles(implied roles): admin(a3c7bb5d06884b048c1bfb4403b82b42 ) -> member(3f20cb7be46346a8b2ba65c4684d50a3) -> reader(e39e97c23bfe45d1a2f9689b6985f990). Correlation is visible, based on that my user(jwasilewski) should retrieve a full project list, but it seems only three projects are visible where this user is an admin. I do not want to add my user as a reader to every single project to be able to list all of them. Is there a way how to make it or the only way is to add this role(reader) for a user to all projects? Thank you in advance for any suggestions. Best regards, Jan Wasilewski [1] https://docs.openstack.org/keystone/stein/admin/service-api-protection.html… -> https://docs.openstack.org/keystone/latest/admin/service-api-protection.htm… [2] https://paste.openstack.org/show/bq0HgyqouZF1vywKVkGn/

1 0

[neutron] CI update
by Slawek Kaplonski 27 Sep '21

27 Sep '21

Hi, Due to very urgent things which I had yesterday, I wasn't able to run Neutron CI meeting as usually. Fortunatelly we don't have many new issues in our CI. There is only one new issue in our scenario jobs which I wanted to discuss [1]. It's impacting Ironic gates but I noticed it also in the Neutron CI as well. See [2] or [3] for example. I'm not sure about Ironic jobs but in Neutron I saw it mostly (or only, I'm not sure) in the multinode jobs. [1] https://bugs.launchpad.net/neutron/+bug/1944201 [2] https:// e36beaa2ff297ebe7d5f-5944c3d62ed334b8cdf50b534c246731.ssl.cf5.rackcdn.com/ 805849/9/check/neutron-ovs-tempest-dvr-ha-multinode-full/f83fa96/compute1/ logs/screen-q-agt.txt [3] https://storage.bhs.cloud.ovh.net/v1/ AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_88f/803045/12/check/ neutron-ovs-tempest-slow/88f8bb7/job-output.txt -- Slawek Kaplonski Principal Software Engineer Red Hat

5 8

[neutron] Bug deputy report (week starting on 2021-09-20)
by Bernard Cafarelli 27 Sep '21

27 Sep '21

Hello neutrinos, here is the list of bugs reported for last week, overall all bugs have assignees or fixes in progress. Both critical bugs are merged and with pending xena backports (this week is the final RCs week) In details: Critical * neutron-openvswitch-agent crashes on start with firewall config of br-int - https://bugs.launchpad.net/neutron/+bug/1944201 First seen in Ironic jobs but we also see it in our jobs, fix by ralonsoh merged: https://review.opendev.org/c/openstack/neutron/+/810592 (pending xena backport) * neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle fails with Port already has an attached device - https://bugs.launchpad.net/tripleo/+bug/1943708 Previously reported as https://bugs.launchpad.net/neutron/+bug/1943679, fix with revert proposed by slaweq and merged: https://review.opendev.org/c/openstack/neutron/+/809550 (pending xena backport) High * Traffic leaked from dhcp port before vlan tag is applied - https://bugs.launchpad.net/neutron/+bug/1930414 Possible security issue that was under investigation and now public, assigned to rubasov * OVN does not update the network segment tag - https://bugs.launchpad.net/neutron/+bug/1944708 Assigned to lucasagomes, fix in review: https://review.opendev.org/c/openstack/neutron/+/810704 Medium * Fix OVN metadata agent functional tests for Chassis_Private - https://bugs.launchpad.net/neutron/+bug/1944460 Functional OVN tests failing when using newer OVN, fix merged: https://review.opendev.org/c/openstack/neutron/+/810309 * IPv6 slaac subnet creation causes FixedIpsSubnetsNotOnSameSegment error - https://bugs.launchpad.net/neutron/+bug/1944948 Error raised when creating an IPv6 slaac subnet with multisegment, assigned to elvira Incomplete * Instances with SRIOV ports loose access after failed live migrations - https://bugs.launchpad.net/neutron/+bug/1944619 SRIOV live migration issue, liuyulong gave some pointers -- Bernard Cafarelli

1 0

[release] Release countdown for week R-1, Sep 27 - Oct 1
by Herve Beraud 27 Sep '21

27 Sep '21

Development Focus ----------------- We are on the final mile of the Xena development cycle! Remember that the Xena final release will include the latest release candidate (for cycle-with-rc deliverables) or the latest intermediary release (for cycle-with-intermediary deliverables) available. September 30 is the deadline for final Xena release candidates as well as any last cycle-with-intermediary deliverables. We will then enter a quiet period until we tag the final release on October 6, 2021. Teams should be prioritizing fixing release-critical bugs, before that deadline. Otherwise it's time to start planning the Yoga development cycle, including discussing sessions content in preparation of the PTG on the week of October 18. Actions ------- Watch for any translation patches coming through on the stable/xena branch and merge them quickly. If you discover a release-critical issue, please make sure to fix it on the master branch first, then backport the bugfix to the stable/xena branch before triggering a new release. Please drop by #openstack-release with any questions or concerns about the upcoming release ! Upcoming Deadlines & Dates -------------------------- Final Xena release: October 6 Yoga PTG: October 18-22 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud

1 0

[neutron][ovn] support for stateless NAT for floating ip in ml2 ovn
by Moshe Levi 26 Sep '21

26 Sep '21

Hi all, OVN support stateless NAT operations [1] for use case of 1:1 mapped between inner and external ips, i.e dnat_and_snat rule. In openstack is the floating ip use-case. Looking on ml2 ovn support it seem that it only support floating ip with connection tracking. Can ml2 ovn support also the stateless NAT option? Is there concerns using stateless NAT? [1] - https://patchwork.ozlabs.org/project/openvswitch/cover/1570154179-14525-1-g…

3 8

[requirements] Please consider bumping docker to 5.0.2 in Xena u-c
by Pierre Riteau 26 Sep '21

26 Sep '21

Hello, Xena upper constraints currently contain docker 5.0.1, released on August 31. This version includes a critical issue [1] which was fixed by version 5.0.2 released the next day. Except for a docstring update this is the only change, according to git diff. Would it be possible to update Xena u-c to use docker 5.0.2? We currently have to work around it in Kayobe to avoid installing the buggy version [2]. Thanks, Pierre Riteau (priteau) [1] https://github.com/docker/docker-py/issues/2885 [2] https://review.opendev.org/c/openstack/kayobe/+/807345/9/ansible/kolla-targ…

3 2

[zun][requirements] request to blacklist docker 5.0.1 for X release
by Hongbin Lu 26 Sep '21

26 Sep '21

Hi, It appears that docker 5.0.1 is broken [1]. All the OpenStack projects that depends on docker-py (i.e. Zun [2]) are impacted. Would the requirements team consider blacklist docker 5.0.1 for X release? Please note that Zun must have such blacklist. Otherwise, the X release will be totally broken. [1] https://github.com/docker/docker-py/issues/2885 [2] https://bugs.launchpad.net/zun/+bug/1945088 Best regards, Hongbin

2 2

anyone running ovn with ovs+dpdk
by Satish Patel 25 Sep '21

25 Sep '21

Folks, Anyone running ovn with dpdk on ubuntu 20.04 (Focal) with ovs version 2.13.3 ? I am trying to do that but my ovs switch keep crashing as soon as i spin up vm and it creates socket file in /var/run/openvswitch and I am not able to understand what is wrong with it. (no generating crash dump also) Please advise or share your thoughts.

1 0

All Devstack Based Jobs Failing to Create Swap on master
by Clark Boylan 24 Sep '21

24 Sep '21

Hello everyone, At approximately 12:53 UTC September 24 (today) [0] Devstack's Xena branch was created. Due to an unexpected side effect in Devstack's Zuul configs this has caused all Devstack based jobs against master and possible stable/xena to RETRY_LIMIT and fail. The underlying issue you'll see in the jobs is that mkswap cannot create swap because it already exists [1]. This happens because Devstack's pre-run playbook is running twice [2] and the configure-swap role is not idempotent. The role is running twice because the Devstack base job is loading master and stable/xena's configs and composing them together which results in two pre-run playbooks being selected one for each branch. This happens because Devstack's stable/xena branch explicitly states that its configs apply to master and feature/r1 branches [3]. I've pushed up a change [4] to remove this Zuul configuration from the Xena branch which I expect will fix things for us. [0] https://zuul.opendev.org/t/openstack/build/ebe77aee0c104c9e9b917ff28a4731ee… [1] https://zuul.opendev.org/t/openstack/build/52fa83b18e1a476b80afc41617fe68f9… [2] https://zuul.opendev.org/t/openstack/build/52fa83b18e1a476b80afc41617fe68f9… [3] https://opendev.org/openstack/devstack/src/branch/stable/xena/.zuul.yaml#L1… [4] https://review.opendev.org/c/openstack/devstack/+/810947

2 2

[all][tc] What's happening in Technical Committee: summary 24th Sept, 21: Reading: 5 min
by Ghanshyam Mann 24 Sep '21

24 Sep '21

Hello Everyone, Here is this week's summary of the Technical Committee activities. 1. TC Meetings: ============ * TC this week IRC meeting held on Sept 23rd Thursday. * Most of the meeting discussions are summarized in the below sections (Completed or in-progress activities section). To know more details, you can check the complete logs @ https://meetings.opendev.org/meetings/tc/2021/tc.2021-09-23-15.00.log.html * We will have next week's meeting on Sept 30th, Thursday 15:00 UTC on IRC, feel free the topic in agenda [1] by Sept 29th. 2. What we completed this week: ========================= * Moved Cloud Research SIG to active state[2] 3. Activities In progress: ================== TC Tracker for Xena cycle ------------------------------ * TC is using the etherpad[3] for Xena cycle working item. We will be checking and updating the status biweekly on the same etherpad. * Current status is: 7 completed, 6 in-progress Open Reviews ----------------- * Five open reviews for ongoing activities[4]. Add project health check tool ----------------------------------- * We are automating the project health checks. Currently, every TC member is assigned a number of projects to keep checking the project health/ listening to their issues but this is not so successful so far. * Rico has proposed the code[5] to collect the contribution stats and we are in the process of documenting the usage and interpretation of those stats. Stable Core team process change --------------------------------------- * As discussed in Shanghai and Xena PTG, there are some concerns over the current stable core team process (especially now when there are very few global stable team members) and to improve the process, mnaser has proposed the draft resolution[6] . Feel free to provide early feedback if you have any. Removing stable release 'Unmaintained' phase ------------------------------------------------------- * 'Unmaintained' phase in the stable release process has been confusing than adding any value. * Elod has proposed the removal of the 'unmaintained' phase[7], feel free to reply if any objection otherwise we will merge the patch on Monday(27th Sept). Places for projects to spreading the word ------------------------------------------------- * Amy has started to list down all the places that projects can use to spread the word out about your project which will help to get additional contributors and users[8]. She also sent emails to PTLs. * Please make note of it and use such places/blogs to highlights/educate operators/users/community etc. TC tags analysis ------------------- * As discussed in the last PTG, TC is working on an analysis of the usefulness of the Tags framework[9] or what all tags can be cleaned up. * We are still waiting for the operator's response to the email on openstack-disscuss ML[10]. If you are an operator, please respond to the email and based on the feedback we will continue the discussion in PTG. Project updates ------------------- * Add the cinder-netapp charm to Openstack charms[11] * Retiring js-openstack-lib [12] * Retire puppet-freezer[13] Yoga release community-wide goal ----------------------------------------- * Please add the possible candidates in this etherpad [14]. * Current status: "Secure RBAC" is selected for Yoga cycle[15]. PTG planning ---------------- * We are collecting the PTG topics in etherpad[16], feel free to add any topic you would like to discuss. * We discussed the live stream of one of the TC PTG sessions like we did last time. Once we will have more topics in etherpad then we can select the appropriate one. Test support for TLS default: ---------------------------------- * Rico has started a separate email thread over testing with tls-proxy enabled[17], we encourage projects to participate in that testing and help to enable the tls-proxy in gate testing. 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[18]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15 UTC [19] 3. Office hours: The Technical Committee offers a weekly office hour every Tuesday at 0100 UTC [20] 4. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [2] https://review.opendev.org/c/openstack/governance-sigs/+/809887 [3] https://etherpad.opendev.org/p/tc-xena-tracker [4] https://review.opendev.org/q/projects:openstack/governance+status:open [5] https://review.opendev.org/c/openstack/governance/+/810037 [6] https://review.opendev.org/c/openstack/governance/+/810721 [7] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/02501… [8] https://review.opendev.org/c/openstack/project-team-guide/+/810766 [9] https://governance.openstack.org/tc/reference/tags/index.html [10] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/02480… [11] https://review.opendev.org/c/openstack/governance/+/809011 [12] https://review.opendev.org/c/openstack/governance/+/798540 [13] https://review.opendev.org/c/openstack/governance/+/807163 [14] https://etherpad.opendev.org/p/y-series-goals [15] https://review.opendev.org/c/openstack/governance/+/803783 [16] https://etherpad.opendev.org/p/tc-yoga-ptg [17] http://lists.openstack.org/pipermail/openstack-discuss/2021-June/023000.html [18] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [19] http://eavesdrop.openstack.org/#Technical_Committee_Meeting [20] http://eavesdrop.openstack.org/#Technical_Committee_Office_hours -gmann

1 0

[ops][neutron] Is it possible to "lock" a floating IP to an instance ?
by Massimo Sgaravatto 24 Sep '21

24 Sep '21

Hello I have the following use case: A user creates a VM and associates a floating IP to such instance Is in some way possible to prevent that the floating IP is disassociated from that instance by another user of the same project ? If it helps, the user owning the instance could be admin (but allowing only the admin user to manage floating IPs is not an option) Thanks, Massimo

2 2

[Sahara]Currently about the development of the Sahara community there are some points
by Juntingqiu Qiujunting (邱军婷) 24 Sep '21

24 Sep '21

Hi all: Currently about the development of the Sahara community there are some points as following: 1. About the schedule of the regular meeting of the Sahara project? What is your suggestion? How about the regular meeting time every Wednesday afternoon 15:00 to 16:30? 2. Regarding the Sahara project maintenance switch from StoryBoard to launchpad. https://storyboard.openstack.org/ https://blueprints.launchpad.net/openstack/ The reasons are as follows: 1. OpenSatck core projects are maintained on launchpad, such as nova, cinder, neutron, etc. 2. Most OpenStack contributors are used to working on launchpad. 3. Do you have any suggestions? If you think this is feasible, I will post this content in the Sahara community later. Thank you for your help. Thank you Fossen. --------------------------------- Fossen Qiu | 邱军婷 CBRD | 云计算与大数据研发部 T: 18249256272 E: qiujunting(a)inspur.com<mailto:qiujunting@inspur.com> 浪潮云海

3 2

Neutron FIP Qos with OVN Backend
by Ammad Syed 24 Sep '21

24 Sep '21

Hello, I am using Neutron wallaby with OVN backend. I am testing QoS FIP and it works fine. I have created a QoS policy and attached it with FIP, It works fine. Is it possible to set one of the QoS policy as default that each time I acquire a floating IP, the QoS should be attached to it by default ? Ammad

4 4

[devstack][requirements][stable/ussuri] New setuptools doesn't like suds-jurko?
by Neil Jerram 24 Sep '21

24 Sep '21

I use the stable/ussuri branches of devstack and tempest and have recently started seeing this stack.sh failure: Collecting suds-jurko===0.6 Downloading suds-jurko-0.6.tar.bz2 (143 kB) ERROR: Command errored out with exit status 1: command: /usr/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-wsiug8gg/suds-jurko/setup.py'"'"'; __file__='"'"'/tmp/pip-install-wsiug8gg/suds-jurko/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-h8s2x8w4 cwd: /tmp/pip-install-wsiug8gg/suds-jurko/ Complete output (1 lines): error in suds-jurko setup command: use_2to3 is invalid. IIUC that last error is from setuptools, specifically version 58.0.4. In previous runs with setuptools 57.4.0, this error did not occur. Does it mean that stable/ussuri global-requirements.txt for setuptools should have "<58" ? I haven't investigated why suds-jurko is pulled in, but I guess it's from the reference Neutron code - as opposed to something that my particular testing has added - and so this could affect anyone using stable/ussuri devstack. Best wishes, Neil -- Neil Jerram Principal Software Engineer Tigera neil(a)tigera.io | @neiljerram <http://twitter.com/neiljerram> Follow Tigera: Blog <https://blog.tigera.io/> | Twitter <https://twitter.com/tigeraio> | LinkedIn <https://www.linkedin.com/company/tigera/> Leader in Kubernetes Security and Observability <https://tigera.io/>

2 2

[release][qa][requirements][i18n] time to finalize stable branch setup
by Előd Illés 24 Sep '21

24 Sep '21

Hi, All the projects that are enabled in devstack by default have been branched, so it's time to - [requirements]: create branch for 'requirements' to be able to lift the requirements freeze - [qa]: do the necessary QA preparation patches ( https://wiki.openstack.org/wiki/QA/releases <https://wiki.openstack.org/wiki/QA/releases> ) - [i18n]: update the translation tools for the new stable series So I ask the PTLs of these teams to do the above necessary preparation. Thanks, Előd

1 0

[ironic] Yoga PTG
by Iury Gregory 23 Sep '21

23 Sep '21

Hello Ironicers! We are approaching the time of the next PTG (it's in October but we need to make some decisions). Our yoga etherpad is available[1], feel free to add any topics we shall discuss. In the ethercalc[2] you can find the slots we already booked, we may add more slots for APAC if someone is willing to run the meeting and discussions. Don't forget to register! [3]. [1] https://etherpad.opendev.org/p/ironic-yoga-ptg [2] https://ethercalc.openstack.org/8tum5yl1bx43 [3] https://openinfra-ptg.eventbrite.com -- *Att[]'sIury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Part of the ironic-core and puppet-manager-core team in OpenStack* *Software Engineer at Red Hat Czech* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory(a)gmail.com <iurygregory(a)gmail.com>*

1 1

[neutron] Drivers meeting - Friday 24.09.2021 - cancelled
by Lajos Katona 23 Sep '21

23 Sep '21

Hi Neutron Team, Due to lack of agenda, let's cancel tomorrow's drivers meeting. Let's concentrate on the ongoing release and see You on the meeting next week :-) Lajos Katona (lajoskatona)

1 0