March 2021 - openstack-discuss - lists.openstack.org

[cinder] Review of tiny patch to add Ceph RBD fast-diff to cinder-backup
by Christian Rohmann 07 Apr '21

07 Apr '21

Hey everyone, I wrote a tiny patch to add the Ceph RDB feature of fast-diff to backups created by cinder-backup: * https://review.opendev.org/c/openstack/cinder/+/766856/ Could someone please take a peek and let me know of this is sufficient to be merged? Thanks and with kind regards, Christian

3 12

[docs] Project guides in PDF format
by Peter Matulis 07 Apr '21

07 Apr '21

Hello, I understand that there was a push [1] to make PDFs available for download from the published pages on docs.openstack.org for the various guides (Sphinx projects). Is there any steam left in this initiative? The next best thing is to have something at the repository level that people can work with. Currently, there are a limited number of guides that can be converted to PDF from the openstack-manuals repository. What is the suggested implementation for individual projects? Thanks, Peter Matulis [1]: https://etherpad.opendev.org/p/train-pdf-support-goal

5 11

[qa][hacking] Proposing new core reviewers
by Martin Kopec 06 Apr '21

06 Apr '21

Hi all, I'd like to propose Sorin Sbarnea (IRC: zbr) and Radosław Piliszek (IRC: yoctozepto) to hacking core. They both are doing a great upstream work among multiple different projects and volunteered to help us with maintenance of hacking project as well. You can vote/feedback in this email thread. If no objection by 6th of April, we will add them to the list. Regards, -- Martin Kopec

3 3

[all] Gate resources and performance
by Dan Smith 06 Apr '21

06 Apr '21

Hi all, I have become increasingly concerned with CI performance lately, and have been raising those concerns with various people. Most specifically, I'm worried about our turnaround time or "time to get a result", which has been creeping up lately. Right after the beginning of the year, we had a really bad week where the turnaround time was well over 24 hours. That means if you submit a patch on Tuesday afternoon, you might not get a test result until Thursday. That is, IMHO, a real problem and massively hurts our ability to quickly merge priority fixes as well as just general velocity and morale. If people won't review my code until they see a +1 from Zuul, and that is two days after I submitted it, that's bad. Things have gotten a little better since that week, due in part to getting past a rush of new year submissions (we think) and also due to some job trimming in various places (thanks Neutron!). However, things are still not great. Being in almost the last timezone of the day, the queue is usually so full when I wake up that it's quite often I don't get to see a result before I stop working that day. I would like to ask that projects review their jobs for places where they can cut out redundancy, as well as turn their eyes towards optimizations that can be made. I've been looking at both Nova and Glance jobs and have found some things I think we can do less of. I also wanted to get an idea of who is "using too much" in the way of resources, so I've been working on trying to characterize the weight of the jobs we run for a project, based on the number of worker nodes required to run all the jobs, as well as the wall clock time of how long we tie those up. The results are interesting, I think, and may help us to identify where we see some gains. The idea here is to figure out[1] how many "node hours" it takes to run all the normal jobs on a Nova patch compared to, say, a Neutron one. If the jobs were totally serialized, this is the number of hours a single computer (of the size of a CI worker) would take to do all that work. If the number is 24 hours, that means a single computer could only check *one* patch in a day, running around the clock. I chose the top five projects in terms of usage[2] to report here, as they represent 70% of the total amount of resources consumed. The next five only add up to 13%, so the "top five" seems like a good target group. Here are the results, in order of total consumption: Project % of total Node Hours Nodes ------------------------------------------ 1. TripleO 38% 31 hours 20 2. Neutron 13% 38 hours 32 3. Nova 9% 21 hours 25 4. Kolla 5% 12 hours 18 5. OSA 5% 22 hours 17 What that means is that a single computer (of the size of a CI worker) couldn't even process the jobs required to run on a single patch for Neutron or TripleO in a 24-hour period. Now, we have lots of workers in the gate, of course, but there is also other potential overhead involved in that parallelism, like waiting for nodes to be available for dependent jobs. And of course, we'd like to be able to check more than patch per day. Most projects have smaller gate job sets than check, but assuming they are equivalent, a Neutron patch from submission to commit would undergo 76 hours of testing, not including revisions and not including rechecks. That's an enormous amount of time and resource for a single patch! Now, obviously nobody wants to run fewer tests on patches before they land, and I'm not really suggesting that we take that approach necessarily. However, I think there are probably a lot of places that we can cut down the amount of *work* we do. Some ways to do this are: 1. Evaluate whether or not you need to run all of tempest on two configurations of a devstack on each patch. Maybe having a stripped-down tempest (like just smoke) to run on unique configs, or even specific tests. 2. Revisit your "irrelevant_files" lists to see where you might be able to avoid running heavy jobs on patches that only touch something small. 3. Consider moving some jobs to the experimental queue and run them on-demand for patches that touch particular subsystems or affect particular configurations. 4. Consider some periodic testing for things that maybe don't need to run on every single patch. 5. Re-examine tests that take a long time to run to see if something can be done to make them more efficient. 6. Consider performance improvements in the actual server projects, which also benefits the users. If you're a project that is not in the top ten then your job configuration probably doesn't matter that much, since your usage is dwarfed by the heavy projects. If the heavy projects would consider making changes to decrease their workload, even small gains have the ability to multiply into noticeable improvement. The higher you are on the above list, the more impact a small change will have on the overall picture. Also, thanks to Neutron and TripleO, both of which have already addressed this in some respect, and have other changes on the horizon. Thanks for listening! --Dan 1: https://gist.github.com/kk7ds/5edbfacb2a341bb18df8f8f32d01b37c 2; http://paste.openstack.org/show/C4pwUpdgwUDrpW6V6vnC/

12 36

[kolla][glance] Few questions about images.
by Eddie Yen 06 Apr '21

06 Apr '21

Hi everyone, I want to ask about the image operating permission & Windows images issue we met since we can't find any answers on the internet. 1. Now we're still using Rocky with Ceph as storage. Sometimes we need to re-pack the image on the Openstack. We used to save as snapshot (re-pack by Nova ephemeral VM) or upload to image (re-pack by volume), then set snapshot/image's visibility to the public. But since Rocky, we can't do this anymore because when we try to set public, Horizon always shows "not enough permission" error. The workaround we're using for now is creating a nova snapshot after re-pack, download the snapshot, then upload the snapshot again as public. But it's utterly wasting time if the images are huge. So we want to know how to unleash this limitation can let us just change snapshot to public at least. 2. Openstack uses virtio as a network device by default, so we always install a virtio driver when packing Windows images. As the network performance issue in GSO/TSO enablement, we also need to disable them in device properties. But since Windows 10 2004 build (my thought) device properties always reset these settings after Sysprep. We found there's a workaround [1] to solve this issue, but may not work sometimes. Is there a better way to solve this issue? Many thanks, Eddie. [1] PersistAllDeviceInstalls | Microsoft Docs <https://docs.microsoft.com/en-us/windows-hardware/customize/desktop/unatten…>

1 1

launch VM on volume vs. image
by Tony Liu 03 Apr '21

03 Apr '21

Hi, With Ceph as the backend storage, launching a VM on volume takes much longer than launching on image. Why is that? Could anyone elaborate the high level workflow for those two cases? Thanks! Tony

3 9

[ironic] Review Jams
by Julia Kreger 01 Apr '21

01 Apr '21

In the Ironic team's recent mid-cycle call, we discussed the need to return to occasionally having review jams in order to help streamline the review process. In other words, get eyes on a change in parallel and be able to discuss the change. The goal is to help get people on the same page in terms of what and why. Be on hand to answer questions or back-fill context. This is to hopefully avoid the more iterative back and forth nature of code review, which can draw out a long chain of patches. As always, the goal is not perfection, but forward movement especially for complex changes. We've established two time windows that will hopefully not to be too hard for some contributors to make it to. It doesn't need to be everyone, but it would help for at least some people whom actively review or want to actively participate in reviewing, or whom are even interested in a feature to join us for our meeting. I've added an entry on to our wiki page to cover this, with the current agenda and anticipated review jam topic schedule. The tl;dr is we will use meetpad[1] and meet on Mondays at 2 PM UTC and Tuesdays at 6 PM UTC. The hope is to to enable some overlap of reviewers. If people are interested in other times, please bring this up in the weekly meeting or on the mailing list. I'm not sending out calendar invites for this. Yet. :) See everyone next week! -Julia [0]: https://wiki.openstack.org/wiki/Meetings/Ironic#Review_Jams [1]: https://meetpad.opendev.org/ironic

3 2

How to customize the xml used in libvirt from GUI/opestack command line?
by Evan Zhao 01 Apr '21

01 Apr '21

Hi there, I googled this question and found two major answers: 1. ssh to the compute node and use `virsh dumpxml` and `virsh undefine/define`, etc. 2. edit nova/virt/libvrit/config.py directly. However, it's trivial to ssh to each node and do the modification, and I prefer not to touch the nova source code, is there any better ways to achieve this? I expect to edit the namespace of a certain element and append an additional element <qemu:command> to the xml file. Any information will be appreciated.

2 1

[cinder][nova][requirements] RFE requested for os-brick
by Brian Rosmaita 01 Apr '21

01 Apr '21

Hello Requirements Team, The Cinder team recently became aware of a potential data-loss bug [0] that has been fixed in os-brick master [1] and backported to os-brick stable/wallaby [2]. We've proposed a release of os-brick 4.4.0 from stable/wallaby [3] and are petitioning for an RFE to include 4.4.0 in the wallaby release. We have three jobs running tempest with os-brick source in master that have passed with [1]: os-brick-src-devstack-plugin-ceph [4], os-brick-src-tempest-lvm-lio-barbican [5],and os-brick-src-tempest-nfs [6]. The difference between os-brick master (at the time the tests were run) and stable/wallaby since the 4.3.0 tag is as follows: master: d4205bd 3 days ago iSCSI: Fix flushing after multipath cfg change (Gorka Eguileor) 0e63fe8 2 weeks ago Merge "RBD: catch read exceptions prior to modifying offset" (Zuul) 28545c7 4 months ago RBD: catch read exceptions prior to modifying offset (Jon Bernard) 99b2c60 2 weeks ago Merge "Dropping explicit unicode literal" (Zuul) 7cfdb76 6 weeks ago Dropping explicit unicode literal (tushargite96) 9afa1a0 3 weeks ago Add Python3 xena unit tests (OpenStack Release Bot) ab57392 3 weeks ago Update master for stable/wallaby (OpenStack Release Bot) 91a1cca 3 weeks ago (tag: 4.3.0) Merge "NVMeOF connector driver connection information compatibility fix" (Zuul) stable/wallaby: f86944b 3 days ago Add release note prelude for os-brick 4.4.0 (Brian Rosmaita) c70d70b 3 days ago iSCSI: Fix flushing after multipath cfg change (Gorka Eguileor) 6649b8d 3 weeks ago Update TOX_CONSTRAINTS_FILE for stable/wallaby (OpenStack Release Bot) f3f93dc 3 weeks ago Update .gitreview for stable/wallaby (OpenStack Release Bot) 91a1cca 3 weeks ago (tag: 4.3.0) Merge "NVMeOF connector driver connection information compatibility fix" (Zuul) This gives us very high confidence that the results of the tests run against master also apply to stable/wallaby at f86944b. Thank you for considering this request. (I've included Nova here because the bug occurs when the configuration option that enables multipath connections on a compute is changed while volumes are attached, so if this RFE is approved, nova might want to raise the minimum version of os-brick in wallaby to 4.4.0.) [0] https://launchpad.net/bugs/1921381 [1] https://review.opendev.org/c/openstack/os-brick/+/782992 [2] https://review.opendev.org/c/openstack/os-brick/+/783207 [3] https://review.opendev.org/c/openstack/releases/+/783641 [4] https://zuul.opendev.org/t/openstack/build/30a103668e4c4a8cb6f1ef907ef3edcb [5] https://zuul.opendev.org/t/openstack/build/bb11eef737d34c41bb4a52f8433850b0 [6] https://zuul.opendev.org/t/openstack/build/3ad3359ca712432d9ef4261d72c787fa

4 9

[tc] Technical Committee weekly meeting
by Ghanshyam Mann 31 Mar '21

31 Mar '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for April 1st at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, March 31st, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

[ironic] PTL on vacation - Week of April 12th
by Julia Kreger 31 Mar '21

31 Mar '21

Greetings folks! I'm going to take a little time off before we get to the Project Teams Gathering, so aside from the two calls I've committed myself to during that week, I'm going to be sitting on a mountain enjoying a change of scenery. If someone will volunteer to run the weekly meeting during the week of the 12th, I will be greatly appreciative. I'm hoping to have a rough PTG schedule up in advance of that meeting as well as the topic list refined, so it will just be links in the weekly agenda. Thanks, -Julia

2 1

[neutron] oslo.privsep migration in Neutron
by Rodolfo Alonso Hernandez 31 Mar '21

31 Mar '21

Hello Neutrinos: During the last cycles we have been migrating the Neutron code from oslo.rootwrap to oslo.privsep. Those efforts are aimed at reaching the goal defined in [1] and are tracked in [2]. At this point, starting Xena developing cycle, we can state that we have migrated all short lived commands from oslo.rootwrap to oslo.privsep or to a native implementation (that could also use oslo.privsep to elevate the permissions if needed). The problem are the daemons or services (long lived processes) that Neutron spawns using "ProcessManager"; this is why "ProcessManager.enable" is the only code calling "utils.execute" without "privsep_exec" parameter. Those process cannot be executed using oslo.privsep because the privsep root daemon has a limited number of executing threads. The remaining processes are [3]. Although we didn't reach the Completion Criteria defined in [1], that is remove the oslo.rootwrap dependency, I think we don't have an alternative to run those services and we should keep rootwrap for them. If there are no objections, once [3] is merged we can consider that Neutron (not other Stadium projects) finished the efforts on [1]. Please, any feedback is always welcome. Regards. [1]https://review.opendev.org/c/openstack/governance/+/718177 [2]https://storyboard.openstack.org/#!/story/2007686 [3] https://review.opendev.org/c/openstack/neutron/+/778444/2/etc/neutron/rootw…

5 6

[ironic] How to move nodes from a 'clean failed' state into 'Available'
by Igal Katzir 31 Mar '21

31 Mar '21

Hello Team, I had a situation where my undercloud-node had a problem with it’s disk and has disconnected from overcloud. I couldn’t restore the undercloud controller and ended up re-installing it (running 'openstack undercloud install’). The installation ended successfully but now I’m in a situation where Cleanup of the overcloud deployed nodes fails: (undercloud) [stack@interop010 ~]$ openstack baremetal node list +--------------------------------------+------------+---------------+-------------+--------------------+-------------+ | UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance | +--------------------------------------+------------+---------------+-------------+--------------------+-------------+ | 97b9a603-f64f-47c1-9fb4-6c68a5b38ff6 | interop025 | None | power on | clean failed | True | | 4b02703a-f765-4ebb-85ed-75e88b4cbea5 | interop026 | None | power on | clean failed | True | +--------------------------------------+------------+---------------+-------------+--------------------+-------------+ I’ve tried to move node to available state but cannot: (undercloud) [stack@interop010 ~]$ openstack baremetal node provide 97b9a603-f64f-47c1-9fb4-6c68a5b38ff6 The requested action "provide" can not be performed on node "97b9a603-f64f-47c1-9fb4-6c68a5b38ff6" while it is in state "clean failed". (HTTP 400) My question is: How do I make the nodes available again? as the deployment of overcloud fails with: ERROR due to "Message: No valid host was found. , Code: 500” Thanks, Igal

3 9

[ironic] PTG and topics oh my
by Julia Kreger 31 Mar '21

31 Mar '21

As everyone hopefully knows by now, the PTG[0] is coming up. We've had an etherpad[1] posted for about a month. During this time, the etherpad has collected 22 topics. I'm 99.95% sure there is some topic overlap, but what we need to make a schedule... or at least begin to form an agenda which requires us understanding what community members perceive as priorities or important topics. So if everyone over the next couple days could do the following two tasks, it would help tremendously for consensus building. 1) Add any missing topics you feel to be important 2) Add +1's to topics you feel are important for yourself OR the community. As always, pressing topics and topics which have the most votes will tend to have more time allocated. -Julia [0]: https://www.eventbrite.com/e/project-teams-gathering-april-2021-tickets-143… [1]: https://etherpad.opendev.org/p/ironic-xena-ptg

1 0

How to debug silent live migration errors
by Linebarger, John 31 Mar '21

31 Mar '21

How would I debug silent (or mostly silent) live migration errors? We're using the Stein release of Canonical's Charmed OpenStack. I have configured it for live migration per the instructions at this link: https://docs.openstack.org/nova/pike/admin/configuring-migrations.html#sect… Specifically: 1. I did not specify vncserver_listen=0.0.0.0 in nova.conf because we are not running VNC on our instances 2. instances_path is /var/lib/nova/instances on all compute nodes 3. I believe that MAAS is "the sole provider of DHCP and DNS for the network hosting the MAAS cluster", per https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/late… 4. Identical authorized_keys files are present on all compute nodes with keys from all compute nodes by default 5. I manually configured the firewalls on all compute nodes to allow libvirt to communicate between compute hosts with: sudo ufw allow 49152:49261/tcp 6. The following settings are specified in nova.conf on each compute node: live_migration_downtime = 500 live_migration_downtime_steps = 10 live_migration_downtime_delay = 75 live_migration_permit_post_copy=true Here's what happens when I try to Live Migrate from the Horizon Dashboard: 1. As admin, in the Admin --> Instances menu, I select the dropdown arrow to the right of the instance. Live Migrate Instance appears (but in black, unlike Migrate Instance, which appears in red). I select Live Migrate Instance, and whether or not I Automatically schedule new host or manually select a new host the Task column says "Migrating" and then it stops and reverts to None. The server never changes. The Action Log shows the live migration request but the Message column is blank. 2. I do the very same thing but this time select Disk Over Commit. Same results. Migrating reverts back to None and the server never changes. 3. I do the very same thing but this time select Block Migration. This time I do get an error: "Failed to live migrate instance to host 'AUTO_SCHEDULE'". And this time the Action Log has "Error" in the Message column. Same behavior with the CLI. For example, this CLI command below completes silently, yet the server for the instance never changes. john@vm-dev-john:~/bin$ openstack server migrate <instanceID> --live <newServerName> [Silent failure] john@vm-dev-john:~/bin$ openstack server show <instancID> [Still running on original server] Note that I *can* successfully Migrate, both using the Horizon Dashboard and the CLI. What fails is Live Migration. I just have no idea why, and no error is displayed in the Action Log for the instance. For reference, the instance is an m1.small with 2GB of RAM, 1 VCPU, and a 20GB Cinder disk volume attached on /dev/vda. Any and all debugging ideas would be most welcome. Without logs I am simply guessing in the dark at this point. Thanks! Enjoy! John M. Linebarger, PhD, MBA Principal Member of Technical Staff Sandia National Laboratories (Office) 505-845-8282 (Cell) 505-681-4879 [cid:image002.jpg@01D72535.958BCAE0]<http://www.sandia.gov/>[AWS Certified Solutions Architect - Professional]<https://www.youracclaim.com/badges/24fe4e43-2f72-4ecc-a11a-783d21dada0f>[AWS Certified Solutions Architect - Associate]<https://www.youracclaim.com/badges/707a5f6f-d919-4daa-bbc4-81b5a779a6cd>[AWS Certified Developer - Associate]<https://www.youracclaim.com/badges/b52ac093-5415-4493-85c3-b77e055211f5>[cid:image003.png@01D72531.072F13F0]<https://www.youracclaim.com/badges/d6e5a453-0e61-4dbe-9716-6af51f364710/pub…>[cid:image005.png@01D72535.958BCAE0]<https://www.youracclaim.com/badges/035d4967-f77d-476e-915b-1061bb789ec3/pub…>

3 3

[cinder] Bug deputy report for week of 2021-03-31
by Sofia Enriquez 31 Mar '21

31 Mar '21

Hello, This is a bug report from 2021-03-16 to 2021-03-24. You're welcome to join the next Cinder Bug Meeting later today. Weekly on Wednesday at 1500 UTC in #openstack-cinder Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Critical: - https://bugs.launchpad.net/os-brick/+bug/1921381 "iSCSI: Flushing issues when multipath config has changed". Stable/ussuri backport assigned to Luigi Toscano (ltoscano) High: - Medium: - https://bugs.launchpad.net/cinder/+bug/1922013 "IBM SVF driver: Add vols to a group (consitent_group_replication enabled) fails sometimes if rccg and rcrel are in different states" Unassigned. Low: - https://bugs.launchpad.net/cinder/+bug/1921584: "backup/drivers/s3.py uses vendored requests from botocore". Unassigned. Incomplete: - Regards Sofi -- L. Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[kolla] does someone uses AArch64 images?
by Marcin Juszkiewicz 31 Mar '21

31 Mar '21

AArch64 support in Kolla(-Ansible) project is present since Ocata (iirc). For most of that time we supported all three distributions: CentOS, Debian and Ubuntu. # Distribution coverage CentOS has binary (with packages from RDO project) and source images. We supported version 7, 8 and now we are using CentOS Stream 8 in Wallaby. Debian is source only as there never was any source of up-to-date OpenStack packages (Debian OpenStack team builds only x86-64 packages and they do it after release). Ubuntu has both binary (packages from UCA) and source images. # My interest I take care of Debian/source ones as we use them at Linaro in our developer cloud setups. It often means backporting of fixes when we upgrade from one previous release to something newer. # External repos problem The problem in all cases is external repositories and their lack of support for AArch64 architecture. ## RabbitMQ Biggest problem is with RabbitMQ and Erlang. For x86-64 RMQ developers build packages for CentOS/Debian/Ubuntu. We use them. On AArch64 it differ: - CentOS uses erlang from CentOS distribution - Debian uses erlang from my Linaro OBS repository (where I do builds of backport of Debian 'testing' package). - Ubuntu uses erlang from Ubuntu distribution Ubuntu one is too old - 22.2 while current rabbitmq needs 22.3 or newer. I do not plan to work on getting it working. # Final question I would like to know does anyone use AArch64 images built with Kolla. Please respond which branches you use, distribution used as host OS and as images OS.

3 9

RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller
by Braden, Albert 31 Mar '21

31 Mar '21

Centos7: {rabbit,"RabbitMQ","3.7.24"}, "Erlang/OTP 22 [erts-10.7.2.8] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:128] [hipe]\n"}, Centos8: {rabbit,"RabbitMQ","3.7.28"}, "Erlang/OTP 22 [erts-10.7.2.8] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:128] [hipe]\n"}, When I deploy the first Centos8 controller, RMQ comes up with all 3 nodes active and seems to be working fine until I shut down the 2nd controller. The only hint of trouble when I replace the 1st node is this error message the first time I run the deployment: https://paste.ubuntu.com/p/h9HWdfwmrK/ and the crash dump that appears on control2: crash dump log: https://paste.ubuntu.com/p/MpZ8SwTJ2T/ First 1500 lines of the dump: https://paste.ubuntu.com/p/xkCyp2B8j8/ If I wait for a few minutes then RMQ recovers on control2 and the 2nd run of the deployment seems to work, and there is no trouble until I shut down control1. -----Original Message----- From: Mark Goddard <mark(a)stackhpc.com> Sent: Wednesday, March 31, 2021 4:14 AM To: Braden, Albert <C-Albert.Braden(a)charter.com> Cc: openstack-discuss(a)lists.openstack.org Subject: [EXTERNAL] Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. On Tue, 30 Mar 2021 at 13:41, Braden, Albert <C-Albert.Braden(a)charter.com> wrote: > > I’ve created a heat stack and installed Openstack Train to test the Centos7->8 upgrade following the document here: > > > > https://docs.openstack.org/kolla-ansible/train/user/centos8.html#migrating-… > > > > I used the instructions here to successfully remove and replace control0 with a Centos8 box > > > > https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hos… > > > > After this my RMQ admin page shows all 3 nodes up, including the new control0. The name of the cluster is rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-2 /]# rabbitmqctl cluster_status > > Cluster status of node rabbit@chrnc-void-testupgrade-control-2 ... > > [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', > > 'rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}]}, > > {running_nodes,['rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}, > > {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, > > {partitions,[]}, > > {alarms,[{'rabbit@chrnc-void-testupgrade-control-0-replace',[]}, > > {'rabbit@chrnc-void-testupgrade-control-1',[]}, > > {'rabbit@chrnc-void-testupgrade-control-2',[]}]}] > > > > After that I create a new VM to verify that the cluster is still working, and then perform the same procedure on control1. When I shut down services on control1, the ansible playbook finishes successfully: > > > > kolla-ansible -i ../multinode stop --yes-i-really-really-mean-it --limit control1 > > … > > control1 : ok=45 changed=22 unreachable=0 failed=0 skipped=105 rescued=0 ignored=0 > > > > After this my RMQ admin page stops responding. When I check RMQ on the new control0 and the existing control2, the container is still up but RMQ is not running: > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status > > Error: this command requires the 'rabbit' app to be running on the target node. Start it with 'rabbitmqctl start_app'. > > > > If I start it on control0 and control2, then the cluster seems normal and the admin page starts working again, and cluster status looks normal: > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status > > Cluster status of node rabbit@chrnc-void-testupgrade-control-0-replace ... > > [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', > > 'rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}]}, > > {running_nodes,['rabbit@chrnc-void-testupgrade-control-2', > > 'rabbit@chrnc-void-testupgrade-control-0-replace']}, > > {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, > > {partitions,[]}, > > {alarms,[{'rabbit@chrnc-void-testupgrade-control-2',[]}, > > {'rabbit@chrnc-void-testupgrade-control-0-replace',[]}]}] > > > > But my hypervisors are down: > > > > (openstack) [root@chrnc-void-testupgrade-build kolla-ansible]# ohll > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB | > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > | 3 | chrnc-void-testupgrade-compute-2.dev.chtrse.com | QEMU | 172.16.2.106 | down | 5 | 8 | 2560 | 30719 | > > | 6 | chrnc-void-testupgrade-compute-0.dev.chtrse.com | QEMU | 172.16.2.31 | down | 5 | 8 | 2560 | 30719 | > > | 9 | chrnc-void-testupgrade-compute-1.dev.chtrse.com | QEMU | 172.16.0.30 | down | 5 | 8 | 2560 | 30719 | > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > > > When I look at the nova-compute.log on a compute node, I see RMQ failures every 10 seconds: > > > > 172.16.2.31 compute0 > > 2021-03-30 03:07:54.893 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out > > 2021-03-30 03:07:55.905 7 INFO oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] Reconnected to AMQP server on 172.16.1.132:5672 via [amqp] client with port 56422. > > 2021-03-30 03:08:05.915 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out > > > > In the RMQ logs I see this every 10 seconds: > > > > 172.16.1.132 control2 > > [root@chrnc-void-testupgrade-control-2 ~]# tail -f /var/log/kolla/rabbitmq/rabbit\(a)chrnc-void-testupgrade-control-2.log |grep 172.16.2.31 > > 2021-03-30 03:07:54.895 [warning] <0.13247.35> closing AMQP connection <0.13247.35> (172.16.2.31:56420 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): > > client unexpectedly closed TCP connection > > 2021-03-30 03:07:55.901 [info] <0.15288.35> accepting AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) > > 2021-03-30 03:07:55.903 [info] <0.15288.35> Connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) has a client-provided name: nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e > > 2021-03-30 03:07:55.904 [info] <0.15288.35> connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e): user 'openstack' authenticated and granted access to vhost '/' > > 2021-03-30 03:08:05.916 [warning] <0.15288.35> closing AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): > > > > Why does RMQ fail when I shut down the 2nd controller after successfully replacing the first one? Hi Albert, Could you share the versions of RabbitMQ and erlang in both versions of the container? When initially testing this setup, I think we had 3.7.24 on both sides. Perhaps the CentOS 8 version has moved on sufficiently to become incompatible? Mark > > > > I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails. > > > > The contents of this e-mail message and > any attachments are intended solely for the > addressee(s) and may contain confidential > and/or legally privileged information. If you > are not the intended recipient of this message > or if this message has been addressed to you > in error, please immediately alert the sender > by reply e-mail and then delete this message > and any attachments. If you are not the > intended recipient, you are notified that > any use, dissemination, distribution, copying, > or storage of this message or any attachment > is strictly prohibited. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

1 0

[qa][tempest] Announcing scenario.manager stable interface
by Martin Kopec 31 Mar '21

31 Mar '21

Hi all, we would like to announce that tempest.scenario.manager has been declared stable in Tempest 27.0.0 and it's ready to be consumed by tempest plugins. All related patches were tracked under 'tempest-scenario-manager-stable' blueprint [1] and the whole effort was tracked in [2]. A little history: Some time ago, tempest/scenario/manager.py got copied to most of the plugins and therefore, it diverged - every plugin's copy had slight differences. In the latest release, we pushed changes to unify the manager's methods and improved their APIs in order to have them easier to consume. I would also like to thank Soniya Vyas for an exceptional job moving this effort forward. [1] https://review.opendev.org/q/topic:bp/tempest-scenario-manager-stable [2] https://etherpad.opendev.org/p/tempest-scenario-manager Regards, -- Martin Kopec

5 4

[Swift] swift client querying capabilites fails and may affect Horizon's container management
by Stefan Kelber 31 Mar '21

31 Mar '21

Hello, i have a Kolla-Ansible AIO stack i added SWIFT to after the initial installation. CLI management of containers work on CLI, but Horizon is troubled: Clicking "container" throws "Error: Unable to fetch the policy details", and no horizon based container management is possible therefore. In an effort to find the root problem, i stumbled upon the following, which i assume to be related: it looks to me like there are issues with keystone authentication of SWIFT client (3.10.1) when attempting to fetch capabilities on OS Victoria Kolla-Ansible, but *only* when querying capabilities: <EXAMPLES> ## 1 (swift-container-server)[root@testhost ~]# swift --os-auth-url http://10.10.10.82:35357/v3 --auth-version 3 --os-project-name admin --os-project-domain-name Default --os-username admin --os-user-domain-name Default --os-password T-STRING-s6 stat Account: AUTH_4334f282aa3a498aac95d2cf9aa9fa91 Containers: 1 Objects: 2 Bytes: 0 Containers in policy "policy-0": 1 Objects in policy "policy-0": 2 Bytes in policy "policy-0": 0 Content-Type: text/plain; charset=utf-8 X-Timestamp: 1616964437.15820 Accept-Ranges: bytes X-Account-Project-Domain-Id: default Vary: Accept X-Trans-Id: tx2638af9bd98646ce92dfe-00606342a1 X-Openstack-Request-Id: tx2638af9bd98646ce92dfe-00606342a1 Transfer-Encoding: chunked ## 2 (swift-container-server)[root@testhost ~]# swift --os-auth-url http://10.10.10.82:35357/v3 --auth-version 3 --os-project-name admin --os-project-domain-name Default --os-username admin --os-user-domain-name Default --os-password T-STRING-s6 list mycontainer ## 3 (swift-container-server)[root@testhost ~]# swift --os-auth-url http://10.10.10.82:35357/v3 --auth-version 3 --os-project-name admin --os-project-domain-name Default --os-username admin --os-user-domain-name Default --os-password T-STRING-s6 capabilities Capabilities GET failed: http://10.10.10.82:8080/info 401 Unauthorized [first 60 chars of response] b'{"error": {"code": 401, "title": "Unauthorized", "message": ' Failed Transaction ID: txeb57fccc09e54e3fa2fa5-00606342af </EXAMPLES> Am i on the wrong track? Best Stefan Here is the entire debug output of the failing transaction: (swift-container-server)[root@testhost ~]# swift --os-auth-url http://10.10.10.82:35357/v3 --auth-version 3 --os-project-name admin --os-project-domain-name Default --os-username admin --os-user-domain-name Default --os-password T-STRING-s6 capabilities --debug DEBUG:keystoneclient.auth.identity.v3.base:Making authentication request to http://10.10.10.82:35357/v3/auth/tokens DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 10.10.10.82:35357 DEBUG:urllib3.connectionpool:http://10.10.10.82:35357 "POST /v3/auth/tokens HTTP/1.1" 201 7299 DEBUG:keystoneclient.auth.identity.v3.base:{"token": {"methods": ["password"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "12940206c0a1464f96a47fe74d19c893", "name": "admin", "password_expires_at": null}, "audit_ids": ["WjAnZyn5TaGufxZGqnvW3Q"], "expires_at": "2021-03-31T14:22:13.000000Z", "issued_at": "2021-03-30T14:22:13.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "4334f282aa3a498aac95d2cf9aa9fa91", "name": "admin"}, "is_domain": false, "roles": [{"id": "b740e2667f034d02b6af2114ec06e20a", "name": "admin"}, {"id": "ceb68a7214c74a969cd83b9146e1fbda", "name": "heat_stack_owner"}, {"id": "6b2ae10793c34fb287dd994a8dd2e8a1", "name": "member"}, {"id": "edb76a75b2fc4af79bf09e38e07be5f1", "name": "reader"}], "catalog": [{"endpoints": [{"id": "05e0b6b67f094b64b1caa9cf78dd34b2", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8780", "region": "RegionOne"}, {"id": "9575e60e3a034ff282bbeda2287b3d01", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8780", "region": "RegionOne"}, {"id": "f5fb2fffde5d4b88b230397560ac6952", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8780", "region": "RegionOne"}], "id": "16edbecc93484568a2f5b46db9909293", "type": "placement", "name": "placement"}, {"endpoints": [{"id": "09275022c3de4405adb6bb25071a49ba", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:9292", "region": "RegionOne"}, {"id": "3a6fa026c7c9422f8cb013492f5ba0e0", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:9292", "region": "RegionOne"}, {"id": "a4fa0d097eed4c3b97ff4ed0f27096d6", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:9292", "region": "RegionOne"}], "id": "2a71f9f773734edb8ea6b711c844b334", "type": "image", "name": "glance"}, {"endpoints": [{"id": "a20fbce4737540ce91039175c7409811", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8774/v2.1", "region": "RegionOne"}, {"id": "b16bad7fdf1342329a59e231cf56851d", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8774/v2.1", "region": "RegionOne"}, {"id": "ce27e7239706458c80e42a68a13383b2", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8774/v2.1", "region": "RegionOne"}], "id": "2c56db642dde46a2ab2f766dbd0ba9c9", "type": "compute", "name": "nova"}, {"endpoints": [{"id": "8c183a6cd8244b2cb1630a662c3a292e", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8004/v1/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "958487dfae194f69b5f56f0b6e901163", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8004/v1/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "a8e0db3b2e1c49a99ed97ec18a64ece6", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8004/v1/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}], "id": "5428e2837ccf413886140c0603867ee3", "type": "orchestration", "name": "heat"}, {"endpoints": [{"id": "2e9a6a2fc78340b7bdc18041dec79a86", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8774/v2/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "894ab56c6c7140ebaac8e7ec502e9b03", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8774/v2/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "8e86543bfeb547d78cf707100d0ef7f0", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8774/v2/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}], "id": "5639441ae48441ed9c8163fc537f0c1a", "type": "compute_legacy", "name": "nova_legacy"}, {"endpoints": [{"id": "32092c9ac4634ece9d89edc2b146af54", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:9696", "region": "RegionOne"}, {"id": "dad19d0e0d8e420caa5b1276d82917e4", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:9696", "region": "RegionOne"}, {"id": "f085e59dae874005b8e6fb1df55db5d5", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:9696", "region": "RegionOne"}], "id": "56e63ef1e3b24fbca6a2efb7394eb77b", "type": "network", "name": "neutron"}, {"endpoints": [{"id": "0ccb4189837f482aac1ee67a0f5b69b7", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8000/v1", "region": "RegionOne"}, {"id": "be4fab7e4ae04786817702817ea09c34", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8000/v1", "region": "RegionOne"}, {"id": "eb8ccdc41ab94aaa9567e29e728e9109", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8000/v1", "region": "RegionOne"}], "id": "83cee8b745d74b1d9a3bc1e37d3e21bf", "type": "cloudformation", "name": "heat-cfn"}, {"endpoints": [{"id": "917c36b07fae4b278a2f63d3886933c3", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:5000", "region": "RegionOne"}, {"id": "9d40ecfea4a04327b64e5751ed8043ce", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:5000", "region": "RegionOne"}, {"id": "d4941b09a2cb40648f3b019bb1653120", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:35357", "region": "RegionOne"}], "id": "94dce13398fd42e4805ae1a7d9976e78", "type": "identity", "name": "keystone"}, {"endpoints": [{"id": "10297840af2b4370a2f63908aea3ff9c", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8776/v2/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "a1a617f8996d439b8de091a4f15a88ae", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8776/v2/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "a930062cb2da497bbeb727c9fec6bba0", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8776/v2/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}], "id": "d7f1cd74b2294126ac939b7d705369a3", "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"id": "1bb18181af9b4dd28bbf16491adb16fe", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8080/v1/AUTH_4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "686b82631d4d40b1b62ed1a7cb8fb1cf", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8080/v1", "region": "RegionOne"}, {"id": "e0cc80d07c6246668ca9cce9d736af62", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8080/v1/AUTH_4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}], "id": "eabcc4d34b3c4ce499025816e2302f34", "type": "object-store", "name": "swift"}, {"endpoints": [{"id": "4c2b1dbe8d3544c9add475d63b0492d4", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8776/v3/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "77634a04b3644d0eb2abab41f3325a1c", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8776/v3/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "7f11d1c28644413e8cfe5ff19b91ebd2", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8776/v3/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}], "id": "fda50e1a62564367bca385a64088334d", "type": "volumev3", "name": "cinderv3"}]}} DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 10.10.10.82:8080 DEBUG:urllib3.connectionpool:http://10.10.10.82:8080 "GET /info HTTP/1.1" 401 114 INFO:swiftclient:REQ: curl -i http://10.10.10.82:8080/info -X GET -H "Accept-Encoding: gzip" INFO:swiftclient:RESP STATUS: 401 Unauthorized INFO:swiftclient:RESP HEADERS: {'Content-Type': 'application/json', 'Content-Length': '114', 'WWW-Authenticate': 'Keystone uri="http://10.10.10.82:5000"', 'X-Trans-Id': 'txac041bfff30342729c2dd-0060633415', 'X-Openstack-Request-Id': 'txac041bfff30342729c2dd-0060633415', 'Date': 'Tue, 30 Mar 2021 14:22:13 GMT'} INFO:swiftclient:RESP BODY: b'{"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}' Capabilities GET failed: http://10.10.10.82:8080/info 401 Unauthorized [first 60 chars of response] b'{"error": {"code": 401, "title": "Unauthorized", "message": ' Failed Transaction ID: txac041bfff30342729c2dd-0060633415

1 1

[kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller
by Braden, Albert 31 Mar '21

31 Mar '21

I've created a heat stack and installed Openstack Train to test the Centos7->8 upgrade following the document here: https://docs.openstack.org/kolla-ansible/train/user/centos8.html#migrating-… I used the instructions here to successfully remove and replace control0 with a Centos8 box https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hos… After this my RMQ admin page shows all 3 nodes up, including the new control0. The name of the cluster is rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com<mailto:rabbit@chrnc-void-testupgrade-control-0.dev.chtrse.com> (rabbitmq)[root@chrnc-void-testupgrade-control-2 /]# rabbitmqctl cluster_status Cluster status of node rabbit@chrnc-void-testupgrade-control-2 ... [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', 'rabbit@chrnc-void-testupgrade-control-0-replace', 'rabbit@chrnc-void-testupgrade-control-1', 'rabbit@chrnc-void-testupgrade-control-2']}]}, {running_nodes,['rabbit@chrnc-void-testupgrade-control-0-replace', 'rabbit@chrnc-void-testupgrade-control-1', 'rabbit@chrnc-void-testupgrade-control-2']}, {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, {partitions,[]}, {alarms,[{'rabbit@chrnc-void-testupgrade-control-0-replace',[]}, {'rabbit@chrnc-void-testupgrade-control-1',[]}, {'rabbit@chrnc-void-testupgrade-control-2',[]}]}] After that I create a new VM to verify that the cluster is still working, and then perform the same procedure on control1. When I shut down services on control1, the ansible playbook finishes successfully: kolla-ansible -i ../multinode stop --yes-i-really-really-mean-it --limit control1 ... control1 : ok=45 changed=22 unreachable=0 failed=0 skipped=105 rescued=0 ignored=0 After this my RMQ admin page stops responding. When I check RMQ on the new control0 and the existing control2, the container is still up but RMQ is not running: (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status Error: this command requires the 'rabbit' app to be running on the target node. Start it with 'rabbitmqctl start_app'. If I start it on control0 and control2, then the cluster seems normal and the admin page starts working again, and cluster status looks normal: (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status Cluster status of node rabbit@chrnc-void-testupgrade-control-0-replace ... [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', 'rabbit@chrnc-void-testupgrade-control-0-replace', 'rabbit@chrnc-void-testupgrade-control-1', 'rabbit@chrnc-void-testupgrade-control-2']}]}, {running_nodes,['rabbit@chrnc-void-testupgrade-control-2', 'rabbit@chrnc-void-testupgrade-control-0-replace']}, {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, {partitions,[]}, {alarms,[{'rabbit@chrnc-void-testupgrade-control-2',[]}, {'rabbit@chrnc-void-testupgrade-control-0-replace',[]}]}] But my hypervisors are down: (openstack) [root@chrnc-void-testupgrade-build kolla-ansible]# ohll +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB | +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ | 3 | chrnc-void-testupgrade-compute-2.dev.chtrse.com | QEMU | 172.16.2.106 | down | 5 | 8 | 2560 | 30719 | | 6 | chrnc-void-testupgrade-compute-0.dev.chtrse.com | QEMU | 172.16.2.31 | down | 5 | 8 | 2560 | 30719 | | 9 | chrnc-void-testupgrade-compute-1.dev.chtrse.com | QEMU | 172.16.0.30 | down | 5 | 8 | 2560 | 30719 | +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ When I look at the nova-compute.log on a compute node, I see RMQ failures every 10 seconds: 172.16.2.31 compute0 2021-03-30 03:07:54.893 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out 2021-03-30 03:07:55.905 7 INFO oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] Reconnected to AMQP server on 172.16.1.132:5672 via [amqp] client with port 56422. 2021-03-30 03:08:05.915 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out In the RMQ logs I see this every 10 seconds: 172.16.1.132 control2 [root@chrnc-void-testupgrade-control-2 ~]# tail -f /var/log/kolla/rabbitmq/rabbit\(a)chrnc-void-testupgrade-control-2.log |grep 172.16.2.31 2021-03-30 03:07:54.895 [warning] <0.13247.35> closing AMQP connection <0.13247.35> (172.16.2.31:56420 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): client unexpectedly closed TCP connection 2021-03-30 03:07:55.901 [info] <0.15288.35> accepting AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) 2021-03-30 03:07:55.903 [info] <0.15288.35> Connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) has a client-provided name: nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e 2021-03-30 03:07:55.904 [info] <0.15288.35> connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e): user 'openstack' authenticated and granted access to vhost '/' 2021-03-30 03:08:05.916 [warning] <0.15288.35> closing AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): Why does RMQ fail when I shut down the 2nd controller after successfully replacing the first one? I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

2 1

Networking-midonet deprecated
by Slawek Kaplonski 31 Mar '21

31 Mar '21

Hi, As it was discussed and decided in the other thread [1] due to lack of maintainers and not good shape of the project, networking-midonet was deprecated and is not part of the official Neutron stadium. Almost all required patches from [2] are already merged. Also patch [3] is merged already so there will be no official Wallaby release of the networking-midonet project. Last stable, supported release is Victroria. Project can be revived in the x/ namespace if there will be someone who wants to maintain it. [1] http://lists.openstack.org/pipermail/openstack-discuss/2021-March/021400.ht… [2] https://review.opendev.org/q/topic:%22deprecate-networking-midonet%22+(stat… [3] https://review.opendev.org/c/openstack/releases/+/781713 -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

How to customize the xml used in libvirt from GUI/opestack command line?
by Evan Zhao 30 Mar '21

30 Mar '21

Hi there, I googled this question and found two major answers: 1. ssh to the compute node and use `virsh dumpxml` and `virsh undefine/define`, etc. 2. edit nova/virt/libvrit/config.py directly. However, it's trivial to ssh to each node and do the modification, and I prefer not to touch the nova source code, is there any better ways to achieve this? I expect to edit the namespace of a certain element and append an additional element <qemu:command> to the xml file. Any information will be appreciated.

1 0

[security sig] April meeting cancelled
by Gage Hugo 30 Mar '21

30 Mar '21

Hey team, The security sig meeting for April 1st is cancelled, we will meet again next month at the usual time. Thanks

1 0

[security sig] PTG Timeslot
by Gage Hugo 30 Mar '21

30 Mar '21

Hello, The security SIG has a spot at the PTG from 1500 to 1700 UTC on April 19th. The agenda[0] is up for anyone to add if they have something to discuss. Hope to see you all there. [0] https://etherpad.opendev.org/p/security-sig-ptg-xena

1 0