December 2021 - openstack-discuss

[neutron][CI] How to reduce number of rechecks - brainstorming
by Slawek Kaplonski 07 Dec '21

07 Dec '21

Hi, Recently I spent some time to check how many rechecks we need in Neutron to get patch merged and I compared it to some other OpenStack projects (see [1] for details). TL;DR - results aren't good for us and I think we really need to do something with that. Of course "easiest" thing to say is that we should fix issues which we are hitting in the CI to make jobs more stable. But it's not that easy. We are struggling with those jobs for very long time. We have CI related meeting every week and we are fixing what we can there. Unfortunately there is still bunch of issues which we can't fix so far because they are intermittent and hard to reproduce locally or in some cases the issues aren't realy related to the Neutron or there are new bugs which we need to investigate and fix :) So this is never ending battle for us. The problem is that we have to test various backends, drivers, etc. so as a result we have many jobs running on each patch - excluding UT, pep8 and docs jobs we have around 19 jobs in check and 14 jobs in gate queue. In the past we made a lot of improvements, like e.g. we improved irrelevant files lists for jobs to run less jobs on some of the patches, together with QA team we did "integrated-networking" template to run only Neutron and Nova related scenario tests in the Neutron queues, we removed and consolidated some of the jobs (there is still one patch in progress for that but it should just remove around 2 jobs from the check queue). All of that are good improvements but still not enough to make our CI really stable :/ Because of all of that, I would like to ask community about any other ideas how we can improve that. If You have any ideas, please send it in this email thread or reach out to me directly on irc. We want to discuss about them in the next video CI meeting which will be on November 30th. If You would have any idea and would like to join that discussion, You are more than welcome in that meeting of course :) [1] http://lists.openstack.org/pipermail/openstack-discuss/2021-November/ 025759.html -- Slawek Kaplonski Principal Software Engineer Red Hat

11 19

[ops]RabbitMQ High Availability
by DHilsbos＠performair.com 07 Dec '21

07 Dec '21

All; In the time I've been part of this mailing list, the subject of RabbitMQ high availability has come up several times, and each time specific recommendations for both Rabbit and Open Stack are provided. I remember it being an A or B kind of recommendation (i.e. configure Rabbit like A1, and Open Stack like A2, OR configure Rabbit like B1, and Open Stack like B2). Unfortunately, I can't find the previous threads on this topic. Does anyone have this information, that they would care to share with me? Thank you, Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

3 3

[kolla-ansible] Default masakari configuration should evacuate instances across nodes in wallaby?
by Rodrigo Lima 07 Dec '21

07 Dec '21

I'm working on upgrading an openstack farm from victoria to wallaby. After successful upgrade, I would like to enable hacluster and masakari to test HA between failing compute nodes. Everything seems to be running (pacemaker with the remote nodes OK, corosync without errors, masakari-monitors detects the 2 compute nodes online), but... when I simulate the failure of a node with shutdown, the failure and notification appears in the hostmonitor log, but the instances that were on the failed node don't evacuate, and I couldn't find documentation that explains how to do this specific configuration, if necessary. Does anyone have any ideas?

3 5

[dev][panko] deprecation doubts
by Soares Sarto, Ricardo 06 Dec '21

06 Dec '21

Hi Everyone Now that Panko has been deprecated, where are the events stored? What is the evolution strategy for OpenStack Telemetry? What is a current list of data generated by core openstack services (nova, neutron, keystone, glance, cinder, heat, etc.) to ceilometer ? (e.g. this link is from ocata … where is the current one ? https://docs.openstack.org/ocata/admin-guide/telemetry-data-collection.html ) Thank you!

1 0

OpenInfra Live - December 9, 2021 at 9am CT
by Ashlee Ferguson 06 Dec '21

06 Dec '21

Hi everyone, This week’s OpenInfra Live episode is brought to you by the OpenStack community. This new episode of the “Large Scale OpenStack” show will discuss operators' tricks and tools. In a live and direct discussion, our guests will reveal tricks and homegrown tools that they use in the trenches to keep their OpenStack clusters ticking like clockwork. Episode: Large Scale OpenStack: Operators’ tricks and tools Date and time: December 9, 2021 at 9am CT (1500 UTC) You can watch us live on: YouTube: https://www.youtube.com/watch?v=F_9KKAQE4fc LinkedIn: https://www.linkedin.com/feed/update/urn:li:ugcPost:6873686260419055616/ Facebook: https://www.facebook.com/104139126308032/posts/4596356560419577/ WeChat: recording will be posted on OpenStack WeChat after the live stream Speakers: Shatadru Bandyopadhyay (Workday) Thierry Carrez (OpenInfra Foundation) Thomas Goirand (Infomaniak) Axel Jacquet (Infomaniak) Gene Kuo (LINE) Belmiro Moreira (CERN) Arnaud Morin (OVHCloud) Adrien Pensart (OVHCloud) Have an idea for a future episode? Share it now at ideas.openinfra.live. Thanks! Ashlee Ashlee Ferguson Carlson Community & Events Open Infrastructure Foundation

1 0

[cinder] midcycle R-17 followup meeting thursday
by Brian Rosmaita 06 Dec '21

06 Dec '21

The poll results are in and our midcycle followup meeting will be held: DATE: Thursday, 9 December 2021 TIME: 1400-1430 UTC LOCATION: https://bluejeans.com/3228528973 The meeting will be recorded. The midcycle etherpad is here: https://etherpad.opendev.org/p/cinder-yoga-midcycles (Followup info starts at line 34, unless some joker has added a bunch of stuff at the top of the etherpad.) The primary topic is Gorka's proposal to revamp the quota system. There are links to the proposed spec and some POC patches on the etherpad. It's worth taking a look in advance so that we can have a short and productive meeting! cheers, brian

1 0

Re: [ops] [kolla] RabbitMQ High Availability
by Bogdan Dobrelya 06 Dec '21

06 Dec '21

> I read this with great interest because we are seeing this issue. Questions: > > 1. We are running kola-ansible Train, and our RMQ version is 3.7.23. Should we be upgrading our Train clusters to use 3.8.x? > 2. Document [2] recommends policy '^(?!(amq\.)|(.*_fanout_)|(reply_)).*'. I don't see this in our ansible playbooks, nor in any of the config files in the RMQ container. What would this look like in Ansible, and what should the resulting container config look like? > 3. It appears that we are not setting "amqp_durable_queues = True". What does this setting look like in Ansible, and what file does it go into? Note that even having rabbit HA policies adjusted like that and its HA replication factor [0] decreased (e.g. to a 2), there still might be high churn caused by a large enough number of replicated durable RPC topic queues. And that might cripple the cloud down with the incurred I/O overhead because a durable queue requires all messages in it to be persisted to a disk (for all the messaging cluster replicas) before they are ack'ed by the broker. Given that said, Oslo messaging would likely require a more granular control for topic exchanges and the durable queues flag - to tell it to declare as durable only the most critical paths of a service. A single config setting and a single control exchange per a service might be not enough. There are also race conditions with durable queues enabled, like [1]. A solution could be where each service declare its own dedicated control exchange with its own configuration. Finally, openstack components should add perhaps a *.next CI job to test it with durable queues, like [2] [0] https://www.rabbitmq.com/ha.html#replication-factor [1] https://zuul.opendev.org/t/openstack/build/aa514dd788f34cc1be3800e6d7dba0e8… [2] https://review.opendev.org/c/openstack/nova/+/820523 > > Does anyone have a sample set of RMQ config files that they can share? > > It looks like my Outlook has ruined the link; reposting: > [2] https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit -- Best regards, Bogdan Dobrelya, Irc #bogdando

1 0

[cinder][OpenstackAnsible] Xena Cycle-Trailing Release Deadline
by Előd Illés 06 Dec '21

06 Dec '21

Hello teams with trailing projects, Next week is the Xena cycle-trailing release deadline [1], and all projects following the cycle-trailing release model must release their Xena deliverables by 16 December, 2021. The following trailing projects haven't been released yet for Xena: Cinder team's deliverables: - cinderlib OSA team's deliverables: - openstack-ansible-roles - openstack-ansible Patches were generated 2 weeks ago [2], feel free to update those if they are not up to date anymore. This is just a friendly reminder to allow you to release these projects in time. Do not hesitate to ping us if you have any questions or concerns. [1] https://releases.openstack.org/yoga/schedule.html#y-cycle-trail [2] https://review.opendev.org/q/topic:trailing-xena Előd Illés irc: elodilles

1 0

[neutron] Bug deputy report for week of November 29
by Lajos Katona 06 Dec '21

06 Dec '21

Hi Neutron Team I was bug deputy in neutron last week. Summary ========== 3 bugs need more attention, one is related to db upgrade (upgrade from Rocky to Stein), one bug is an issue between Nova and Neutron and related to network-vif-plug event, and one seems to be related to IPv6 Metadata. If you have time please check them and help solve them and check the reviews I linked. Needs attention ================ * "neutron-db-manage upgrade" is interrupted with error about neutron.portuplinkstatuspropagation ( https://bugs.launchpad.net/neutron/+bug/1952675 ) * VirtualInterfaceCreateException due to "Timeout waiting for [('network-vif-plugged'..." (https://bugs.launchpad.net/neutron/+bug/1944779 ) * DHCP agent fails to fully configure DHCP namespaces because of duplicate address detected (https://bugs.launchpad.net/neutron/+bug/1953165 ) ** Seems to be an issue with IPv6 metadata. Bugs with assignees =================== * Neurton linuxbridge agent does not create VXLAN interface ( https://bugs.launchpad.net/neutron/+bug/1952611 ) ** Fix proposed by mnaser to oslo.privsep: https://review.opendev.org/c/openstack/oslo.privsep/+/819996 * Segment updates may cause unnecessary overload ( https://bugs.launchpad.net/neutron/+bug/1952730 ) ** fix from rubasov: https://review.opendev.org/c/openstack/neutron/+/819777 * Cannot setup IPv6 overlay network in xena ( https://bugs.launchpad.net/neutron/+bug/1952897 ) ** https://review.opendev.org/c/openstack/neutron/+/820376 * Allow modification of max retries in OVSNeutronAgent ( https://bugs.launchpad.net/neutron/+bug/1952898 ) ** https://review.opendev.org/c/openstack/neutron/+/819816 * Gratuitous ARPs are not sent during master transition ( https://bugs.launchpad.net/neutron/+bug/1952907 ): ** Possible fix from Liu: https://review.opendev.org/c/openstack/neutron/+/712474 * On IPv6 overlay networks linuxbridge vxlans are created always on loopback device (https://bugs.launchpad.net/neutron/+bug/1953139 ): ** https://review.opendev.org/c/openstack/neutron/+/820484 RFEs ====== * [RFE] Unify quota engine API ( https://bugs.launchpad.net/neutron/+bug/1953170 ) ** We discussed the need for this rfe on last Friday's drivers meeting: https://meetings.opendev.org/meetings/neutron_drivers/2021/neutron_drivers.… Regards Lajos Katona (lajoskatona)

1 0

[kolla] Plan to deprecate binary and unify on single distrubition
by Michał Nasiadka 06 Dec '21

06 Dec '21

Hello Koalas, On the PTG we have discussed two topics: 1) Deprecate and drop binary type of Kolla images 2) Use a common base (single Linux distribution) for Kolla images This is a call for feedback - for people that have not been attending the PTG. What this essentially mean for consumers: 1) In Yoga cycle we will deprecate binary type of Kolla images, and in Z cycle those will be dropped. 2) We are not going to support CentOS Stream 9 (cs9) as a base operating system, and the source type build will rely on CentOS Stream 8 in Z release. 3) Beginning from A release Kolla will build only Debian source images - but Kolla-Ansible will still support deployment of those images on CentOS/Ubuntu/Debian Host operating systems (and Rocky Linux to be added in Yoga to that mix). Justification: The Kolla project team is limited in numbers, therefore supporting current broad mix of operating systems (especially with CentOS Stream 9 ,,on the way’’) is a significant maintenance burden. By dropping binary type of images - users would be running more tested images (since Kolla/Kolla-Ansible CI runs source images jobs as voting). In Xena we’ve already changed the default image type Kolla-Ansible uses to source. We also feel that using a unified base OS for Kolla container images is a way to remove some of the maintenance burden (including CI cycles and Request for feedback: If any of those changes is a no go from your perspective - we’d like to hear your opinions. Best regards, Michal Nasiadka

9 17