April 2021 - openstack-discuss - lists.openstack.org

[Interop][Refstack] this Friday meeting
by Kanevsky, Arkady 09 Apr '21

09 Apr '21

Team, This Friday is Good Friday and some people have a day off. Should we cancel this week meeting? Please, respond so we can see if we will have quorum. Thanks, Arkady Arkady Kanevsky, Ph.D. SP Chief Technologist & DE Dell Technologies office of CTO Dell Inc. One Dell Way, MS PS2-91 Round Rock, TX 78682, USA Phone: 512 7204955

5 6

[neutron][all] Tempest jobs running on rocky and queens branches are broken
by Slawek Kaplonski 09 Apr '21

09 Apr '21

Hi, I noticed it mostly in the neutron jobs but it seems that it's true also for other projects for jobs which still runs on Ubuntu 16.04. I Neutron case those are all jobs on stable/rocky and stable/queens branches. Due to [1] those jobs will end up with POST_FAILURE. So please don't recheck Your patches if You have such errors until that bug will be fixed. I think that gmann has or is working on fix for that. [1] https://bugs.launchpad.net/devstack/+bug/1923042 -- Slawek Kaplonski Principal Software Engineer Red Hat

2 1

Re: [Release-job-failures] Release of openstack/eslint-config-openstack for ref refs/tags/4.1.0 failed
by Herve Beraud 09 Apr '21

09 Apr '21

FYI Looks similar to that story: - http://lists.openstack.org/pipermail/openstack-discuss/2021-March/021002.ht… - http://lists.openstack.org/pipermail/openstack-discuss/2021-March/021217.ht… I proposed a patch to move to nodejs10 all our projects that depend on nodejs: https://review.opendev.org/c/openstack/project-config/+/785353 When this patch will be merged I think that this job could be reenqueued. Le jeu. 8 avr. 2021 à 11:10, <zuul(a)openstack.org> a écrit : > Build failed. > > - release-openstack-javascript > https://zuul.opendev.org/t/openstack/build/4062ea0df4e74565b9f8b443e550c0fd > : RETRY_LIMIT in 3m 53s > - announce-release https://zuul.opendev.org/t/openstack/build/None : > SKIPPED > - openstack-upload-github-mirror > https://zuul.opendev.org/t/openstack/build/2e9b658e70f340818cafefa88c5044e2 > : SUCCESS in 41s > > _______________________________________________ > Release-job-failures mailing list > Release-job-failures(a)lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/release-job-failures > -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

2 3

[ironic] APAC-Europe SPUC time?
by Dmitry Tantsur 09 Apr '21

09 Apr '21

Hi folks! The initial SPUC datetime was for 10am UTC, which was 11am for us in central Europe, now is supposed to be 12pm. On one hand, I find it more convenient to have SPUC at 11am still, on the other - I have German classes at this time for a few months starting mid-April. What do you think? Should we keep it in UTC, i.e. 12pm for us in Europe? Will that work for you Jacob? Dmitry -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

3 2

[telemetry][cyborg][heat][monasca][tacker][keystone][release] Last minute RC to land fixes
by Herve Beraud 09 Apr '21

09 Apr '21

Hello teams listed above, We identified fixes and significant changes in your repos so we proposed last minute RC to allow you to release them before the final release. Your teams patches are available here: https://review.opendev.org/q/topic:%22wallaby-final-rc%22 Deadline is today, please validate them ASAP to have a chance to see these fixes released. Patches without response from PTLs/liaisons will be abandoned. After this point final release for RC projects will be started. Notice that RC changes should be on stable/wallaby and not on master, all projects are now branched so your master branches are now for Xena purpose. Thank you for your understanding. -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

2 1

[neutron] Neutron, nftables support and other fantastic beasts
by Rodolfo Alonso Hernandez 09 Apr '21

09 Apr '21

Hello Neutrinos: During Wallaby I've been working on enabling "nftables" support in Neutron. The goal was to use the new Netfilter framework replacing the legacy tools ("iptables", "ip6tables", "arptables" and "ebtables"). Because each namespace has its own Netfilter process, isolated from other namespaces, the migration process could be segmented in several tasks: dnat, fip, router, dhcp, metadata, Linux Bridge FW and OVS hybrid FW (I think I'm not missing anything here). When swapping to the new "nftables" framework, we can use the legacy API tools provided. Those tools provide a smooth transition to the new tooling (we found some differences that are now solved). That means we can keep the current code while using "nftables". Please, read [3] before reading the next paragraph, explaining the three "Netfilter" available framework alternatives. I started creating a "nft" (the "nftables" native binary) parser [1] to implement a NFtablesManager class, same as IPtablesManager. But soon I found that the transition to the new API is not that easy. This is not only a matter of creating the equivalent rule in the "nft" API but considering how those rules are handled in "nftables". Other problems found when using the new "nft" API: - The "--checksum-fill" command used in OVN metadata and DHCP namespace has no equivalent in "nft". That means old DHCP servers incorrectly calculating the packet checksum or DKDP environments won't work correctly. - "ipset" tool, used to group IP addresses and reduce the LB FW rule size, can be converted into a "map" [3]. The problem is this is only understood by the new API, not the "nftables" binaries using the legacy API. In a nutshell, what is the current status? We support (a) legacy tools and (b) "nftables" binaries with legacy API. This is the list of patches enabling the second option: - https://review.opendev.org/c/openstack/neutron/+/784913: this problem was affecting LB FW when "ipset" was disabled (merged). - https://review.opendev.org/c/openstack/neutron/+/785177: reorder the "ebtables" rules and prevent execution error 4 with empty chains. - https://review.opendev.org/c/openstack/neutron/+/785144: this patch, on top of the other two, creates two new neutron-tempest-plugin CI jobs, based on "linuxbridge" and "openvswitch-iptables_hybrid", to test the execution with the new binaries. - https://review.opendev.org/c/openstack/neutron/+/775413: this patch tests what is implemented in the previous one but testing those jobs in the "check" queue (it is a DNM patch just for testing). About the third option, to support the native "nft" API, I don't know if now we have the resources (time) and the need for that. This could be discussed again in the next PTG and in this mail too. Regards. [1]https://review.opendev.org/c/openstack/neutron/+/759874 [2] https://review.opendev.org/c/openstack/neutron/+/785137/3/doc/source/admin/… [3] https://review.opendev.org/c/openstack/neutron/+/775413/10/neutron/agent/li…

1 0

RE: [kolla] RHEL based container image (Tony Liu)
by CHALANSONNET Stéphane (Acoss) 09 Apr '21

09 Apr '21

Hello, You need an active subscription RHOSP for doing that , but Kolla was not supported by Redhat unfortunely ... Stéphane Chalansonnet -----Message d'origine----- De : openstack-discuss-request(a)lists.openstack.org <openstack-discuss-request(a)lists.openstack.org> Envoyé : jeudi 8 avril 2021 22:04 À : openstack-discuss(a)lists.openstack.org Objet : openstack-discuss Digest, Vol 30, Issue 56 Send openstack-discuss mailing list submissions to openstack-discuss(a)lists.openstack.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss or, via email, send a message with subject or body 'help' to openstack-discuss-request(a)lists.openstack.org You can reach the person managing the list at openstack-discuss-owner(a)lists.openstack.org When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..." Today's Topics: 1. Re: [Release-job-failures] Release of openstack/eslint-config-openstack for ref refs/tags/4.1.0 failed (Jeremy Stanley) 2. Re: [zun][kuryr][neutron] Missing vxlan ports in br-tun for Zun containers? (Jason Anderson) 3. [edge][cinder][manila][swift][tripleo] Storage at the edge discussions at the PTG (Ildiko Vancsa) 4. Re: [edge][cinder][manila][swift][tripleo] Storage at the edge discussions at the PTG (John Fulton) 5. [kolla] RHEL based container image (Tony Liu) ---------------------------------------------------------------------- Message: 1 Date: Thu, 8 Apr 2021 16:50:37 +0000 From: Jeremy Stanley <fungi(a)yuggoth.org> To: openstack-discuss(a)lists.openstack.org Subject: Re: [Release-job-failures] Release of openstack/eslint-config-openstack for ref refs/tags/4.1.0 failed Message-ID: <20210408165036.6tkcfwwix5ms3ig4(a)yuggoth.org> Content-Type: text/plain; charset="utf-8" On 2021-04-08 11:54:16 +0200 (+0200), Herve Beraud wrote: [...] > I proposed a patch to move to nodejs10 all our projects that depend on > nodejs: > > https://review.opendev.org/c/openstack/project-config/+/785353 > > When this patch will be merged I think that this job could be reenqueued. I reenqueued the tag, but release-openstack-javascript failed on a different problem. NPM complains that there's already a eslint-config-openstack 4.0.1 published which can't be overwritten, but the tag is for 4.1.0... someone should probably update the version parameter in eslint-config-openstack's package.json file, which means it'll need another release tagged anyway (4.1.1?). -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210408…> ------------------------------ Message: 2 Date: Thu, 8 Apr 2021 17:00:19 +0000 From: Jason Anderson <jasonanderson(a)uchicago.edu> To: openstack-discuss <openstack-discuss(a)lists.openstack.org> Subject: Re: [zun][kuryr][neutron] Missing vxlan ports in br-tun for Zun containers? Message-ID: <0A70CB1A-35AA-4782-8BF3-496080E47341(a)uchicago.edu> Content-Type: text/plain; charset="utf-8" As usual, “rubber ducking” the openstack-discuss list yielded fruit. It turns out that I didn’t have the l2population mechanism driver enabled. I thought this was optional for some reason. It looks like enabling this and restarting the neutorn-openvswitch-agent has fixed connectivity! /Jason > On Apr 8, 2021, at 11:20 AM, Jason Anderson <jasonanderson(a)uchicago.edu> wrote: > > Hello stackers, > > I’m interested in using zun to launch containers and assign floating IPs via neutron to those containers. I am deploying zun, kuryr-libnetwork, and neutron with kolla-ansible on the Train release. I’ve configured neutron with one physical network and I’d like to use a VXLAN overlay for tenant networks. > > What works: > - I can launch containers on a neutron tenant network, they start successfully, they get an IP and can reach each other if they’re co-located on a single host. > - I can create all my neutron networks, routers, subnets, without (obvious) errors. > - I can update security groups on the container and see the iptables rules updated appropriately. > - I can directly create Docker networks using the kuryr driver/type. > > What doesn’t work: > - I can’t see any vxlan ports on the br-tun OVS bridge > - I can’t access the exposed container ports from the control/network > node via the router netns > - Because of that, I can’t assign floating IPs because NAT effectively > won’t work to reach the containers > > The fact that there are no ports on br-tun is supicious, but I’m not sure how this is supposed to work. I don’t see anything weird in neutron-openvswitch-agent logs but those logs are quite noisy and I’m not sure what to look for. > > Has anybody deployed such a setup / are there limitations I should know about? > > Thank you! > > > Jason Anderson > > DevOps Lead, Chameleon > > --- > > Department of Computer Science, University of Chicago Mathematics and > Computer Science, Argonne National Laboratory > jasonanderson(a)uchicago.edu > ------------------------------ Message: 3 Date: Thu, 8 Apr 2021 19:18:54 +0200 From: Ildiko Vancsa <ildiko.vancsa(a)gmail.com> To: OpenStack Discuss <openstack-discuss(a)lists.openstack.org> Subject: [edge][cinder][manila][swift][tripleo] Storage at the edge discussions at the PTG Message-ID: <0A7B1EBD-0715-43ED-B388-A2011D437DD0(a)gmail.com> Content-Type: text/plain; charset=utf-8 Hi, I’m reaching out to draw your attention to the Edge Computing Group sessions on the PTG in less than two weeks. We are still formalizing our agenda, but we have storage identified as one of the topics that the working group would like to discuss. It would be great to have the session also as a continuation to earlier discussions that we had on previous PTGs with relevant OpenStack project contributors. We have a few cross-community sessions scheduled already, but we still have some flexibility in our agenda to schedule this topic so the most people who are interested in participating can join. Our available options are: * Monday (April 19) between 1400 UTC and 1500 UTC * Tuesday (April) between 1400 UTC and 1600 UTC __Please let me know if you or your project would like to participate and if you have a time slot difference from the above.__ Thanks and Best Regards, Ildikó (IRC ildikov on Freenode) ------------------------------ Message: 4 Date: Thu, 8 Apr 2021 13:39:45 -0400 From: John Fulton <johfulto(a)redhat.com> To: Ildiko Vancsa <ildiko.vancsa(a)gmail.com> Cc: OpenStack Discuss <openstack-discuss(a)lists.openstack.org> Subject: Re: [edge][cinder][manila][swift][tripleo] Storage at the edge discussions at the PTG Message-ID: <CAE66OMCQWZFB+7SM1amzRr1Vq3pOm=fEgtQTuHLh0=0PPXy-sQ(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" On Thu, Apr 8, 2021 at 1:21 PM Ildiko Vancsa <ildiko.vancsa(a)gmail.com> wrote: > Hi, > > I’m reaching out to draw your attention to the Edge Computing Group > sessions on the PTG in less than two weeks. > > We are still formalizing our agenda, but we have storage identified as > one of the topics that the working group would like to discuss. It > would be great to have the session also as a continuation to earlier > discussions that we had on previous PTGs with relevant OpenStack project contributors. > > We have a few cross-community sessions scheduled already, but we still > have some flexibility in our agenda to schedule this topic so the most > people who are interested in participating can join. Our available > options > are: > > * Monday (April 19) between 1400 UTC and 1500 UTC > * Tuesday (April) between 1400 UTC and 1600 UTC > I'm not available Monday but could join Tuesday. I'd be curious to hear what others are doing with Storage on the Edge and could share some info on how TripleO does it. John > > __Please let me know if you or your project would like to participate > and if you have a time slot difference from the above.__ > > Thanks and Best Regards, > Ildikó > (IRC ildikov on Freenode) > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210408…> ------------------------------ Message: 5 Date: Thu, 8 Apr 2021 20:04:04 +0000 From: Tony Liu <tonyliu0592(a)hotmail.com> To: "openstack-discuss(a)lists.openstack.org" <openstack-discuss(a)lists.openstack.org> Subject: [kolla] RHEL based container image Message-ID: <MWHPR08MB2382486467C9EE24247052B9BD749(a)MWHPR08MB2382.namprd08.prod.outlook.com> Content-Type: text/plain; charset="us-ascii" Hi, Given [1], RHEL based container is supported on RHEL 8 by Kolla. Where can I get RHEL based container images? I see CentOS and Ubuntu based images on docker hub, but can't find RHEL based images. [1] https://docs.openstack.org/kolla-ansible/ussuri/user/support-matrix.html Thanks! Tony ------------------------------ Subject: Digest Footer _______________________________________________ openstack-discuss mailing list openstack-discuss(a)lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss ------------------------------ End of openstack-discuss Digest, Vol 30, Issue 56 *************************************************

3 2

[zun][kuryr][neutron] Missing vxlan ports in br-tun for Zun containers?
by Jason Anderson 08 Apr '21

08 Apr '21

Hello stackers, I’m interested in using zun to launch containers and assign floating IPs via neutron to those containers. I am deploying zun, kuryr-libnetwork, and neutron with kolla-ansible on the Train release. I’ve configured neutron with one physical network and I’d like to use a VXLAN overlay for tenant networks. What works: - I can launch containers on a neutron tenant network, they start successfully, they get an IP and can reach each other if they’re co-located on a single host. - I can create all my neutron networks, routers, subnets, without (obvious) errors. - I can update security groups on the container and see the iptables rules updated appropriately. - I can directly create Docker networks using the kuryr driver/type. What doesn’t work: - I can’t see any vxlan ports on the br-tun OVS bridge - I can’t access the exposed container ports from the control/network node via the router netns - Because of that, I can’t assign floating IPs because NAT effectively won’t work to reach the containers The fact that there are no ports on br-tun is supicious, but I’m not sure how this is supposed to work. I don’t see anything weird in neutron-openvswitch-agent logs but those logs are quite noisy and I’m not sure what to look for. Has anybody deployed such a setup / are there limitations I should know about? Thank you! Jason Anderson DevOps Lead, Chameleon --- Department of Computer Science, University of Chicago Mathematics and Computer Science, Argonne National Laboratory jasonanderson(a)uchicago.edu

2 2

Re: [glance][openstack-ansible] Snapshots disappear during saving
by Oliver Wenz 08 Apr '21

08 Apr '21

Hi Dmitriy, > I'm wondering if you see also stack trace in keystone logs? Running 'journalctl' on the keystone container, I don't see any tracebacks. Or is there a specific service I should check? Kind regards, Oliver

2 1

Between openstack and offline IDC, a large intranet and a large second-tier network in the world improve the feasibility of data migration and disaster recovery.
by 赵陈洲 08 Apr '21

08 Apr '21

1 0

migration maybe a iaas service for openstack
by 赵陈洲 08 Apr '21

08 Apr '21

1 0

[Group-based-policy] not able to create the policies
by Mantesh Patil 08 Apr '21

08 Apr '21

Hi, I am deploying devstack Ussuri with GBP (stable/ussuri) on ubuntu 18.04 LTS(Updated the packages), after installation I am able to create the policies as mentioned in the wiki <https://wiki.openstack.org/wiki/GroupBasedPolicy/InstallDevstack>. But after creation, I am not able to list the policies. I am using the following command to list the policies and it is giving a warning "/usr/local/lib/python3.6/dist-packages/keystoneauth1/adapter.py:235: UserWarning: Using keystoneclient sessions has been deprecated. Please update your software to use keystoneauth1. warnings.warn('Using keystoneclient sessions has been deprecated. '" Command1: source admin-openrc.sh *Command2: gbp group-create web * and also getting the following error while creating the group [image: image.png] Command3: *gbp group-list -c name -c tenant_id -f value* Please give the information that how can I get a list of group policies using CLI and new authentication.

1 0

[all][elections][tc] TC Vacancy Special Election Voting Kickoff
by Jeremy Stanley 08 Apr '21

08 Apr '21

TC Vacancy Special Election Nomination period is now over. The four already elected TC members for this term are listed as candidates in the special election, but will not appear on any resulting poll as they have already been officially elected. Only new candidates who are not the four elected TC members for this term will appear on a subsequent poll for the TC vacancy special election. The poll for the TC Vacancy Special Election is now open and will remain open until Apr 15, 2021 23:45 UTC. We are selecting 1 additional TC member, please rank all candidates in your order of preference. You are eligible to vote if you are a Foundation individual member[1] that also has committed to one of the official project teams' deliverable repositories[2] over the Apr 24, 2020 00:00 UTC - Mar 08, 2021 00:00 UTC timeframe (Victoria to Wallaby) or if you are one of the extra-atcs.[3] Please note that in order to confirm contributors are foundation members, the preferred address in Gerrit must also be included in the addresses for the corresponding member profile. What to do if you don't see the email and have a commit in at least one of the official deliverables[2]: * check the trash or spam folder of your gerrit Preferred Email address[4], in case it went into trash or spam * wait a bit and check again, in case your email server is a bit slow * find the sha of at least one commit from an official deliverable repo[2] and email the election officials[1]. If we can confirm that you are entitled to vote, we will add you to the voters list and you will be emailed a ballot. Our democratic process is important to the health of OpenStack, please exercise your right to vote. Candidate statements/platforms can be found linked to Candidate names[6]. Happy voting! Thank you, [1] https://www.openstack.org/community/members/ [2] https://opendev.org/openstack/governance/src/commit/892c4f3a851428cf41bab57… [3] Look for the extra-atcs element in [2] [4] Sign into review.openstack.org: Go to Settings > Contact Information. Look at the email listed as your preferred email. That is where the ballot has been sent. [5] https://governance.openstack.org/election/#election-officials [6] https://governance.openstack.org/election/#xena-tc-candidates -- Jeremy Stanley on behalf of the OpenStack Technical Elections Officials

1 0

[first contact] [SIG] PTG Planning!
by Kendall Nelson 08 Apr '21

08 Apr '21

Hello! I didn't think we would need a ton of time and I tried to pick a time to balance everyone's timezones so we have an hour at 22 UTC on Monday in the Austin room. We have an etherpad that was autogenerated: https://etherpad.opendev.org/p/apr2021-ptg-first-contact Please add topics if you have them and your name if you plan to join us! -Kendall Nelson (diablo_rojo)

1 0

[puppet] Proposing Alan Bishop (abishop) for puppet-cinder core and puppet-glance core
by Takashi Kajinami 08 Apr '21

08 Apr '21

Hello, I'd like to propose Alan Bishop (abishop) for the core team of puppet-cinder and puppet-glance. Alan has been actively involved in these 2 modules for a few years and has implemented some nice features like multiple backend support in glance, cinder s3 backup driver and etc, which expanded adoption of puppet-openstack. He has also provided good reviews on patches for these 2 repos based on his understanding about our code, puppet and serverspec. He is an active contributor to cinder and has deep knowledge about it. In addition He is also a core review in TripleO, which consumes our puppet modules, and mainly covers storage components like cinder and glance, so he is familiar with the way how these two components are deployed and configured. I believe adding him to our board helps us improve our review of these two modules. I'll wait for one week to hear any feedback from other core reviewers. Thank you, Takashi

6 6

[victoria][magnum]fedora-atomic-27 image
by Luke Camilleri 08 Apr '21

08 Apr '21

We have insatlled magnum following the installation guide here https://docs.openstack.org/magnum/victoria/install/install-rdo.html and the process was quite smooth but we have been having some issues with the deployment of the clusters. The image being used as per the documentation is https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-27-2… Our first issue was that podman was being used even if we specified the use_podman=false (since the image above did not include podman) but this was resulting in a timeout and the cluster would fail to deploy. We have then installed podman in the image and the cluster progressed a bit further /+ echo 'WARNING Attempt 60: Trying to install kubectl. Sleeping 5s'// //+ sleep 5s// //+ ssh -F /srv/magnum/.ssh/config root@localhost '/usr/bin/podman run --entrypoint /bin/bash --name install-kubectl --net host --privileged --rm --user root --volume /srv/magnum/bin:/host/srv/magnum/bin k8s.gcr.io/hyperkube:v1.15.7 -c '\''cp /usr/local/bin/kubectl /host/srv/magnum/bin/kubectl'\'''// //bash: /usr/bin/podman: No such file or directory// //ERROR Unable to install kubectl. Abort.// //+ i=61// //+ '[' 61 -gt 60 ']'// //+ echo 'ERROR Unable to install kubectl. Abort.'// //+ exit 1/ The cluster is now failing here at "kube_cluster_deploy" and when checking the logs on the master node we noticed the following in the log files: /Starting to run kube-apiserver-to-kubelet-role// //Waiting for Kubernetes API...// //+ echo 'Waiting for Kubernetes API...'// //++ curl --silent http://127.0.0.1:8080/healthz// //+ '[' ok = '' ']'// //+ sleep 5/ This is because the kubernetes API server is not installed either. I have noticed some scripts that should handle the installation but I would like to know if anyone here has had similar issues with a clean Victoria installation. Also should we have to install any packages in the fedora atomic image file or should the installation requirements be part of the stack? Thanks in advance for any asistance

4 9

[baremetal-sig][ironic] Tue Apr 13, 2021, 2pm UTC: Secure RBAC in Ironic
by Arne Wiebalck 08 Apr '21

08 Apr '21

Dear all, The Bare Metal SIG will meet next week Tue Apr 13, 2021, at 2pm UTC on zoom. There will be two main points on the agenda: - A "topic-of-the-day" presentation by Julia Kreger (TheJulia) on 'Secure RBAC in Ironic' and a - PTG pre-discussion on a potential integration of Ironic with Kea DHCP. As usual, all details on https://etherpad.opendev.org/p/bare-metal-sig Everyone is welcome! Cheers, Arne

1 0

[cinder] final reviews for RC-2
by Brian Rosmaita 08 Apr '21

08 Apr '21

We have 3 patches that need review/revision/approval as soon as possible before we release RC-2 tomorrow (Thursday 8 April). All 3 are updates to the release notes: Release note for mTLS support cinder->glance - https://review.opendev.org/c/openstack/cinder/+/783964 Release note about the cgroups v1 situation - https://review.opendev.org/c/openstack/cinder/+/784179 Add known issue note about RBD encrypted volumes - https://review.opendev.org/c/openstack/cinder/+/785235 Please review and leave comments as soon as you can.

2 1

[neutron] Drivers meeting agenda - 09.04.2021
by Slawek Kaplonski 08 Apr '21

08 Apr '21

Hi, Agenda for the tomorrow's drivers meeting is at [1]. We have 2 RFEs to discuss: https://bugs.launchpad.net/neutron/+bug/1922237 - [RFE][QoS] Add minimum guaranteed packet rate QoS rule https://bugs.launchpad.net/neutron/+bug/1921461 - [RFE] Enhancement to Neutron BGPaaS to directly support Neutron Routers & bgp-peering from such routers over internal & external Neutron Networks [1] https://wiki.openstack.org/wiki/Meetings/NeutronDrivers#Agenda -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[Keystone] Managing keystone tokens in high availability environment
by Taha Adel 07 Apr '21

07 Apr '21

Hello Engineers and Developers, I'm currently deploying a three-nodes openstack controller cluster, controller-01, controller-02, anc controller-03. I have installed the keystone service on the three controllers and generated fernet keys on one node and distributed the keys to the other nodes of the cluster. Hence, I have configured an HAProxy in front of them that would distribute the incoming requests over them. The issue is, when I try to access the keystone endpoint from using the VIP of the loadbalancer, the service works ONLY on the node that I have generated the keys on, and it doesn't work on the nodes that got the keys by distribution. the error message I have got is *"INTERNAL SERVER ERROR (500)"* In other words, the node that had* keystone-manage fernet_setup *command ran on it, it can run the service properly, but the others can't. Is the way of replicating the key incorrect? is there any other way? Thanks in advance

2 1

[all][tc] Technical Committee next weekly meeting on April 8th at 1500 UTC.
by Ghanshyam Mann 07 Apr '21

07 Apr '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for April 8th at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, April 7th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

[all] vPTG April 2021 Registration & Schedule
by Kendall Nelson 07 Apr '21

07 Apr '21

Hello everyone! The April 2021 Project Teams Gathering is right around the corner! The official schedule has now been posted on the PTG website [1], the PTGbot will be up to date by the end of the week [2], and we have also attached it to this email. Please double check your rooms! We did a little consolidation and shifting while maintaining the times you signed up for. Friendly reminder, if you have not already registered, please do so [3]. It is important that we get everyone to register for the event as this is how we will contact you about tooling information/passwords and other event details. Please let us know if you have any questions! Cheers, - The Kendalls (diablo_rojo & wendallkaters) [1] PTG Website www.openstack.org/ptg [2] PTGbot: http://ptg.openstack.org/ptg.html [3] PTG Registration: https://april2021-ptg.eventbrite.com

1 0

[tripleo][DIB] Second core review/w+ requested on DIB patch to unblock tripleo promotions
by Ronelle Landy 07 Apr '21

07 Apr '21

Hi diskimage-builder cores, OVB jobs have been failing on centos-8 based releases promotion jobs since the move to container-tools:3.0. https://review.opendev.org/c/openstack/diskimage-builder/+/785138 - "Make DIB_DNF_MODULE_STREAMS part of yum element" and https://review.opendev.org/c/openstack/tripleo-ci/+/785087 - "Use dib_dnf_module_streams for enabling modules" patches were added to address the failures. These patches were tested in https://review.rdoproject.org/r/c/testproject/+/33138 - see passing OVB job https://review.rdoproject.org/zuul/build/6582fb8afa2a44c6a806bb2545a9fadf. Thanks to Carlos for reviewing the DIB patch. We are looking for a second core review and w+ on this patch to clear the promotion lines. Thank you, tripleo CI ruck/rovers

1 0

[ptg] Secure RBAC and Policy Xena PTG sessoins
by Lance Bragstad 07 Apr '21

07 Apr '21

Hey all, Several projects are working through RBAC overhauls and naturally sessions are cropping up for the PTG. I tried bouncing around to various policy sessions during the Wallaby PTG, but I didn't plan things out very well. As a result, I missed sessions, had duplicate conversations with multiple groups, and ended up being more reactive than I'd like. To prevent that, Ghanshyam and I have condensed all the policy/RBAC sessions we know about in a single etherpad [0]. I know most projects are still firming up their schedules, but I've written down the session times that we know of and organized them chronologically. My hope is that this will help us group similar discussions and reach broader consensus on topics easier and quicker. For example, keystone and nova have a cross-project session on Thursday to discuss how nova should handle consuming system-scoped tokens for project-specific operations. This topic certainly isn't exclusive to nova. It'll impact just about every other service and approaching it consistently will be huge for end users and operators. Another good example of this would be the glance refactor to integrate system-scope support we're going to talk about on Wednesday (cinder and barbican are potentially facing very similar refactors). Each session in the etherpad [0] has topics, so if a topic sounds relevant to your service, please feel free to drop into those discussions. A rough outline is that: - Monday we're going to focus on QA and general policy problems (e.g., converting tempest to use system-scope, the JSON->YAML community goal, overall status from Wallaby, etc) - Tuesday we're going to find ways to adopt system-scope in cinder - Wednesday we're going to work through system-scope adoption, the meta definitions API, and test coverage in glance - Thursday we're going to discuss what the experience should be like for operators using system-scoped tokens to do project-specific operations with nova (e.g., rebooting instances) I'm contemplating hosting a 30 minute recap session on Friday that attempts to summarize everything from the week regarding RBAC discussions. If that sounds useful, I'll ask Kristi if I can use one of the keystone sessions for that recap. I know, this feels like a lot of focus for one thing and I appreciate everyone's help working through this stuff. But, I'm hopeful that better organization throughout the PTG week will result in less confusion about what we plan to do in Xena with RBAC so we can deliver something useful to users and operators. Thanks, Lance [0] https://etherpad.opendev.org/p/policy-popup-xena-ptg

2 2

[largescale-sig] Next meeting: April 7, 15utc
by Thierry Carrez 07 Apr '21

07 Apr '21

Hi everyone, Our next Large Scale SIG meeting will be this Wednesday in #openstack-meeting-3 on IRC, at 15UTC. You can doublecheck how it translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20210407T15 A number of topics have already been added to the agenda, including discussing our next video meetings and PTG participation. Feel free to add other topics to our agenda at: https://etherpad.openstack.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

Re: [Release-job-failures] Pre-release of openstack/neutron for ref refs/tags/18.0.0.0rc2 failed
by Thierry Carrez 07 Apr '21

07 Apr '21

zuul(a)openstack.org wrote: > Build failed. > > - openstack-upload-github-mirror https://zuul.opendev.org/t/openstack/build/4b31dfd25e244e38a346e097c993c519 : SUCCESS in 54s > - release-openstack-python https://zuul.opendev.org/t/openstack/build/309d80630c8e4086b52a0d83062cd431 : SUCCESS in 4m 43s > - announce-release https://zuul.opendev.org/t/openstack/build/4bde576261db4943a1dd6610aa52f6be : POST_FAILURE in 8m 33s > - propose-update-constraints https://zuul.opendev.org/t/openstack/build/18de7b1f60044445a06db64da42bbcc3 : SUCCESS in 5m 20s We are missing logs, but it looks like the job actually succeeded at announcing the release: http://lists.openstack.org/pipermail/release-announce/2021-April/011022.html So this can be safely ignored. -- Thierry Carrez (ttx)

2 1

[neutron][nova] Port binding fails when creating an instance
by Maxime d'Estienne 07 Apr '21

07 Apr '21

Hello, I spent a lot of time troubleshooting my issue, which I described here : https://serverfault.com/questions/1058969/cannot-create-an-instance-due-to-… To summarize, when I want to create an instance, binding fails on compute node, the dhcp agent seems to give an ip to the VM but I have an error. I don't know where to dig, besides what I have done. Thanks a lot for your help !

2 5

[Neutron] How to provide internet access to tier 2 instance
by Bernd Bausch 07 Apr '21

07 Apr '21

I have a pretty standard single-server Victoria Devstack, where I created this network topology: public private backend | | | | /-------\ |-- I1 |- I2 |--|Router1|--| | | \-------/ | | | | /-------\ | | |--|Router2|--| | | \-------/ | | | | I1 and I2 are instances. My question: Is it possible to give I2 access to the external world to install software and download files? I don't need access **to** I2 **from** the external world. My unsuccessful attempt: After adding a static default route via Router1 to Router2, I can ping the internet from Router2's namespace, but not from I2. My guess is that Router1 ignores traffic from networks that are not attached to it. I don't have enough experience to understand the netfilter rules in Router1's namespace, and in any case, rather than tweaking them I need a supported method to give I2 internet access, or the confirmation that it is not possible. Thanks much for any insights and suggestions. Bernd

2 1

[cinder] Bug deputy report for week of 2021-04-07
by Sofia Enriquez 07 Apr '21

07 Apr '21

Hello, This is a bug report from 2021-03-31 to 2021-04-07. You're welcome to join the next Cinder Bug Meeting later today. Weekly on Wednesday at 1500 UTC in #openstack-cinder Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Critical:- High: - Medium: - https://bugs.launchpad.net/cinder/+bug/1922408 `` Create volume from snapshot will lose encrypted head when source volume is encrypted in RBD''. Assigned to haixin. - https://bugs.launchpad.net/cinder/+bug/1922013 '' IBM SVF driver: GMCV Add vols to a group fails even if rcrel and rccg are in the same state". Unassigned. Low: - https://bugs.launchpad.net/cinder/+bug/1922255 "Dell PowerVault PVMEISCSIDriver driver cannot manage volumes". Unassigned. - https://bugs.launchpad.net/python-cinderclient/+bug/1922749 "Top-level client doesn't support auth v3". Unassigned. Undecided: - https://bugs.launchpad.net/devstack-plugin-ceph/+bug/1921897 "fast-diff in default feature set breaks 'rbd map'". Assigned to Eric Harney. Regards, Sofi -- L. Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[keystone]improvments in mapping models/support for JWT tokens
by pregusia 07 Apr '21

07 Apr '21

Hello members! Please direct your attention to some keystone modyfications - to be more precisly two of them: (1) extension to mapping engine in order to support multiple projects and assigning project by id (2) extension to authorization mechanisms - add JWT token support Ad (1): Currently mapping engine can map multiple projects for user (as stated in https://docs.openstack.org/keystone/pike/advanced-topics/federation/mapping…) but this support lacks from (a) dynamic mapping projects from assertion (for ex. projects ids are in assertion) and (b) mapping engine cannot map to list from assertion (if in asertion we have "some_field": [ id1, id2, ..., idN ] - then using this field by substitution - eg "{1}" wont map this to whole needed structure of projects in mapping). In patch I extend mapping schema by adding new field "projects_spec" of type string. This field contains string formated like project1ID:role1:role2:..:roleN,project2ID:role1:...:roleN,... and this is mapped into proper structure { "projects": [ { "id": .., "roles": [ ... ] }, ... ] } this allows the identity provider to supply permission-like informations about what projects user should have access to. The implementation is not ideal, and some work shoud be do to make it more user-friendly (for ex. configuration options for auto-creating projects by ID, and options for allowing user to log-in if project-by-id not exists etc) This is only proof of concept and question if this direction is proper according to keystone development. Ad (2) This patch adds new authorization protocol - jwt_token. It allows to access endpoint '/v3/OS-FEDERATION/identity_providers/{IDP_NAME}/protocols/jwt_token/auth' (and obtain keystone auth token) using JWT token in Authorization header field. Header value is checked against proper formating and JWT token is extracted from it. Then, token is validated against public key (RS256/RS512 algorithms only supported), against issuer and expiration date and others. Then, if this end with success, payload from token is supplied to mapping engine, when new user (and according to (1), it's permission to projects for ex.) is created and given access to proper projects/roles. Happy reviewing! pregusia

2 2

[cinder] Review of tiny patch to add Ceph RBD fast-diff to cinder-backup
by Christian Rohmann 07 Apr '21

07 Apr '21

Hey everyone, I wrote a tiny patch to add the Ceph RDB feature of fast-diff to backups created by cinder-backup: * https://review.opendev.org/c/openstack/cinder/+/766856/ Could someone please take a peek and let me know of this is sufficient to be merged? Thanks and with kind regards, Christian

3 12

[victoria]oslo_privsep.daemon.FailedToDropPrivileges
by 朝阳未烈 07 Apr '21

07 Apr '21

Hi, everyone: I tried to build an instance on the compute node but failed. I am sure that every necessary connection has been built. And I found the same error information on the controller node and the compute node , in /var/log/neutron/linuxbride-agent.log That is information: INFO neutron.common.config [-] Logging enabled! 2021-04-07 11:30:52.866 2182 INFO neutron.common.config [-] /usr/bin/neutron-linuxbridge-agent version 17.1.0 2021-04-07 11:30:52.867 2182 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Interface mappings: {'provider': 'ens160'} 2021-04-07 11:30:52.867 2182 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Bridge mappings: {} 2021-04-07 11:30:52.868 2182 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/neutron/neutron-dist.conf', '--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/linuxbridge_agent.ini', '--config-dir', '/etc/neutron/conf.d/neutron-linuxbridge-agent', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmpm5d0ytiv/privsep.sock'] 2021-04-07 11:30:53.346 2182 CRITICAL oslo.privsep.daemon [-] privsep helper command exited non-zero (1) 2021-04-07 11:30:53.346 2182 CRITICAL neutron [-] Unhandled error: oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1) 2021-04-07 11:30:53.346 2182 ERROR neutron Traceback (most recent call last): 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/bin/neutron-linuxbridge-agent", line 10, in <module> 2021-04-07 11:30:53.346 2182 ERROR neutron     sys.exit(main()) 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/neutron/cmd/eventlet/plugins/linuxbridge_neutron_agent.py", line 28, in main 2021-04-07 11:30:53.346 2182 ERROR neutron     agent_main.main() 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 1052, in main 2021-04-07 11:30:53.346 2182 ERROR neutron     manager = LinuxBridgeManager(bridge_mappings, interface_mappings) 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 79, in __init__ 2021-04-07 11:30:53.346 2182 ERROR neutron     self.validate_interface_mappings() 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 94, in validate_interface_mappings 2021-04-07 11:30:53.346 2182 ERROR neutron     if not ip_lib.device_exists(interface): 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 748, in device_exists 2021-04-07 11:30:53.346 2182 ERROR neutron     return IPDevice(device_name, namespace=namespace).exists() 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 328, in exists 2021-04-07 11:30:53.346 2182 ERROR neutron     return privileged.interface_exists(self.name, self.namespace) 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 246, in _wrap 2021-04-07 11:30:53.346 2182 ERROR neutron     self.start() 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 258, in start 2021-04-07 11:30:53.346 2182 ERROR neutron     channel = daemon.RootwrapClientChannel(context=self) 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 367, in __init__ 2021-04-07 11:30:53.346 2182 ERROR neutron     raise FailedToDropPrivileges(msg) 2021-04-07 11:30:53.346 2182 ERROR neutron oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1) 2021-04-07 11:30:53.346 2182 ERROR neutron     And it is the configuration in /etc/sudoer.d/neutron below:   Defaults:neutron !requiretty neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf * neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf     I googled for the solution but they didn’t matter. How can I solve this problem? Thanks for your advicement!

2 1

[docs] Project guides in PDF format
by Peter Matulis 07 Apr '21

07 Apr '21

Hello, I understand that there was a push [1] to make PDFs available for download from the published pages on docs.openstack.org for the various guides (Sphinx projects). Is there any steam left in this initiative? The next best thing is to have something at the repository level that people can work with. Currently, there are a limited number of guides that can be converted to PDF from the openstack-manuals repository. What is the suggested implementation for individual projects? Thanks, Peter Matulis [1]: https://etherpad.opendev.org/p/train-pdf-support-goal

5 11

Re: [glance][openstack-ansible] Snapshots disappear during saving
by Oliver Wenz 06 Apr '21

06 Apr '21

Following the suggestion from https://docs.openstack.org/swift/latest/overview_auth.html I set 'log_level=debug' for authtoken. Now I'm seeing more errors in the glance-api logs: Apr 06 15:07:40 infra1-glance-container-99614ac2 glance-wsgi-api[1837]: 2021-04-06 15:07:38.197 1837 WARNING keystonemiddleware.auth_token [req-ad8d0db9-b630-4237-9fff-d7ff282155d2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Identity response: {"error": {"code": 500, "title": "Internal Server Error", "message": "An unexpected error prevented the server from fulfilling your request."}}: keystoneauth1.exceptions.http.InternalServerError: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-55767c13-a450-4f81-90ab-5a7af6b3f672) Apr 06 15:07:46 infra1-glance-container-99614ac2 uwsgi[1822]: DAMN ! worker 13 (pid: 1836) died, killed by signal 9 :( trying respawn ... Here's the full log: http://paste.openstack.org/show/804208/ The options you suggested @Dmitriy are still there in user_variables.yml: glance_glance_api_conf_overrides: keystone_authtoken: service_token_roles_required: True service_token_roles: service Kind regards, Oliver

2 1

[dev][nova] Problem about vm migration compatibility check
by Guoyi Tu 06 Apr '21

06 Apr '21

hi there, In my test environment, i created a vm and configured the cpu with host-model, when I migrate the vm to another host with the same cpu, it failed the migration compatibility check which complains the cpu definition of domain is incompatible with target host cpu. As we know, when the domain configured as above starts, the host-model cpu definition will automatically converted to custom cpu model and with some addtional features that the KVM supported, these addtional features may contains features that the host doesn't support. In the code, the compatibility of the target host is check by calling compareCPU()(libvirt API). The compareCPU() can only recongnize the features probed by cpuid instruction on the host, but it may not recognize the features of cpu definition of domain xml (virsh dumpxml domainname) when the domain running. So the compatibility check will fail when KVM support one or more features which is considerd as disabled by the cpuid instuction. I think we should call compareHypervisorCPU() or something like that (supported by libvirt since v4.4.0) instead of compareCPU() to check the migration compatibility. My test environment is as follow: host cpu: Cascadelake libvirt-6.9 qemu-5.0 host-model cpu: <mode name='host-model' supported='yes'> <model fallback='forbid'>Cascadelake-Server</model> <vendor>Intel</vendor> <feature policy='require' name='ss'/> <feature policy='require' name='hypervisor'/> <feature policy='require' name='tsc_adjust'/> <feature policy='require' name='umip'/> <feature policy='require' name='pku'/> <feature policy='require' name='md-clear'/> <feature policy='require' name='stibp'/> <feature policy='require' name='arch-capabilities'/> <feature policy='require' name='xsaves'/> <feature policy='require' name='invtsc'/> <feature policy='require' name='rdctl-no'/> <feature policy='require' name='ibrs-all'/> <feature policy='require' name='skip-l1dfl-vmentry'/> <feature policy='require' name='mds-no'/> <feature policy='require' name='pschange-mc-no'/> <feature policy='disable' name='hle'/> <feature policy='disable' name='rtm'/> </mode> The hypervisor, umip, pschange-mc-no features block the compatibility check -- Best Regards, Guoyi Tu

2 1

How to modify staging-ovirt parameters
by Gianluca Cecchi 06 Apr '21

06 Apr '21

Hello, I have a test lab with Queens on CentOS 7 deployed with TripleO, that I use sometimes to mimic an OSP 13 environment. The Openstack nodes are oVirt VMS. I moved these VMs from one oVirt infrastructure to another one. The Openstack environment starts and operates without problems (3 controllers, 2 computes and 3 cephs), but I don't know how to modify the staging-ovirt parameters. At oVirt side I re-created as before a user ostackpm@internal with sufficient power and I also set the same password. But the ip address of the engine cannot be the same. When I deployed the environment I put all the parameters inside the instackenv.json file to do introspection and deploy. Where are they put on undercloud? Is there a way to update them? Perhaps some database entries or other kind of repos where I can update the ip? Thanks in advance, Gianluca

2 2

Unable to retrieve ceph docker image for Queens
by Gianluca Cecchi 06 Apr '21

06 Apr '21

Hello, I'm trying to extend an old small test lab Queens environment with dedicated Ceph Storage nodes. I would like to add some disks to the storage nodes and so I customize my original templates/ceph-config.yaml adding them under devices section: parameter_defaults: CephAnsibleDisksConfig: osd_scenario: lvm osd_objectstore: bluestore devices: - /dev/sda - /dev/sdb - /dev/sdc . . . When I run the overcloud deploy I get this in ceph-install-workflow.log 2021-04-06 21:24:57,955 p=32618 u=mistral | TASK [ceph-docker-common : pulling docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64 image] *** 2021-04-06 21:24:57,955 p=32618 u=mistral | Tuesday 06 April 2021 21:24:57 +0200 (0:00:00.153) 0:06:17.978 ********* 2021-04-06 21:25:13,206 p=32618 u=mistral | FAILED - RETRYING: pulling docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64 image (3 retries left). . . . 2021-04-06 21:26:03,715 p=32618 u=mistral | FAILED - RETRYING: pulling docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64 image (1 retries left). 2021-04-06 21:26:28,839 p=32618 u=mistral | fatal: [172.23.0.232]: FAILED! => {"attempts": 3, "changed": false, "cmd": ["timeout", "300s", "docker", "pull", "docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64"], "delta": "0:00:15.022274", "end": "2021-04-06 21:25:48.701980", "msg": "non-zero return code", "rc": 1, "start": "2021-04-06 21:25:33.679706", "stderr": "Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)", "stderr_lines": ["Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"], "stdout": "Trying to pull repository docker.io/ceph/daemon ... ", "stdout_lines": ["Trying to pull repository docker.io/ceph/daemon ... "]} So it seems the docker images are not there any more. My existing ceph nodes are using that image: [heat-admin@ostack-ceph0 ~]$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7ad4aeb3f4cb docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64 "/entrypoint.sh" 3 weeks ago Up 3 weeks ceph-osd-0 0dc9a9889283 docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64 "/entrypoint.sh" 3 weeks ago Up 3 weeks ceph-osd-3 8d543016cce7 docker.io/tripleoqueens/centos-binary-cron:current-tripleo "dumb-init --singl..." 11 months ago Up 3 weeks logrotate_crond [heat-admin@ostack-ceph0 ~]$ Any way or suggestion to manage this? Thanks, Gianluca

1 0

[qa][gate][stable] stable stein|train py2 devstack jobs are broken
by Ghanshyam Mann 06 Apr '21

06 Apr '21

Hello Everyone, During fixing the grenade issue on stable/train[1], there is another issue that came up for py2 devstack based jobs. I have logged the bug on devstack side as of now - https://bugs.launchpad.net/devstack/+bug/1922736 Let's wait for this to fix before you recheck on failing patches. [1] http://lists.openstack.org/pipermail/openstack-discuss/2021-April/021597.ht… -gmann

3 2

[all] Octavia LoadBalancer Error
by Adekunbi Adewojo 06 Apr '21

06 Apr '21

Hi there, I recently deployed a load balancer on our openstack private cloud. I used this manual - https://docs.openstack.org/octavia/latest/user/guides/basic-cookbook.html to create the load balancer. However, after creating and trying to access it, it returns an error message saying "No server is available to handle this request". Also on the dashboard, "Operating status" shows offline but "provisioning status" shows active. I have two web applications as members of the load balancer and I can individually access those web applications. Could someone please point me in the right direction? Thanks.

3 3

all Octavia LoadBalancer
by Adekunbi Adewojo 06 Apr '21

06 Apr '21

Hi there, I recently deployed a load balancer on our openstack private cloud. I used this manual - https://docs.openstack.org/octavia/latest/user/guides/basic-cookbook.html to create the load balancer. However, after creating and trying to access it, it returns an error message saying "No server is available to handle this request". Also on the dashboard, "Operating status" shows offline but "provisioning status" shows active. I have two web applications as members of the load balancer and I can individually access those web applications. Could someone please point me in the right direction? Thanks.

3 3

[tripleo] master and victoria promotions delayed
by Wesley Hayutin 06 Apr '21

06 Apr '21

0/ Hey just want to communicate that promotions for master and victoria are being delayed due to promotion-blockers. If you have a merged patch in a repo that is not gated by tripleo jobs you will NOT be able to use that patch until we promote. http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?org… Please review the following bugs to better understand what is blocking. https://bugs.launchpad.net/tripleo/+bugs?field.tag=promotion-blocker&orderb… I'll note, any tempest test results logged in bugs are most likely already skipped and not blocking upstream via https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/… Your focus should be on deployment failures, not tempest failures at this time. Thank you!

1 0

[qa][hacking] Proposing new core reviewers
by Martin Kopec 06 Apr '21

06 Apr '21

Hi all, I'd like to propose Sorin Sbarnea (IRC: zbr) and Radosław Piliszek (IRC: yoctozepto) to hacking core. They both are doing a great upstream work among multiple different projects and volunteered to help us with maintenance of hacking project as well. You can vote/feedback in this email thread. If no objection by 6th of April, we will add them to the list. Regards, -- Martin Kopec

3 3

[neutron] bug deputy report (3/29-4/4)
by Akihiro Motoki 06 Apr '21

06 Apr '21

Hi, I was a bug deputy last week and here is a report. Last week is relatively quite. Please check my report. ## Unassigned * OpenStack Metadata API and OVN in Neutron https://bugs.launchpad.net/neutron/+bug/1921809 It would be nice if OVN folks follow it more while haleyb replied. ## Medium, Assigned * [LB] Linux Bridge iptables firewall does not work without "ipset" https://bugs.launchpad.net/neutron/+bug/1922127 Assigned to ralonsoh ## Fix Released * Strings in tags field is limited to 60 chars https://bugs.launchpad.net/neutron/+bug/1921713 ## Almost RFE * allow using tap device on netdev enabled host https://bugs.launchpad.net/neutron/+bug/1922222 I requested the bug author to provide more information on the background ## RFE * [QoS] Add minimum guaranteed packet rate QoS rule https://bugs.launchpad.net/neutron/+bug/1922237 * [RFE] BFD for BGP Dynamic Routing https://bugs.launchpad.net/neutron/+bug/1922716

1 0

[cinder][horizon] Ussuri: Horizon shows error for managed volume(s)
by Eugen Block 06 Apr '21

06 Apr '21

Hi *, I'm struggling with a dashboard error that seems to be wrong, I can't seem to find any existing reports but I could have simply missed it so please point me to any existing reports about that. Anyway, a user imported two volumes from Ceph via 'cinder manage' in our Train version of OpenStack, we upgraded last week to Ussuri. One of those two volumes is bootable and the instance is up and running with both volumes (extended volume group). When clicking on the instance details the Horizon dashboard shows a red window saying "Failed to get attached volume." The dashboard log says: [Tue Apr 06 09:29:56.197238 2021] [wsgi:error] [pid 10140] [remote IP:59482] WARNING horizon.exceptions Recoverable error: volume_image_metadata But I can see both volumes in the details page, so the message seems incorrect. I searched the logs for this error message from before the upgrade but couldn't find any. So this seems to be new in Ussuri, I assume. Does anyone have the same experience with managed volumes? Thanks and best regards, Eugen

1 0

[victoria][neutron][horizon]l3-agent+port-forwarding
by Luke Camilleri 06 Apr '21

06 Apr '21

Hello everyone, I have enable the L3 extension for port-forwarding and can succesfully port-forward traffic after assigning an additional floating IP to the project. I would like to know if it is possible to enable the corresponding horizon functionality for this extension (port-forwarding) please? Regards

3 4

RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller
by Braden, Albert 06 Apr '21

06 Apr '21

Sorry that was a typo. Stopping RMQ during the removal of the *second* controller is what causes the problem. Is there a way to tell Centos 8 Train to use RMQ 3.7.24 instead of 3.7.28? -----Original Message----- From: Braden, Albert Sent: Thursday, April 1, 2021 9:34 AM To: 'openstack-discuss(a)lists.openstack.org' <openstack-discuss(a)lists.openstack.org> Subject: RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller I did some experimenting and it looks like stopping RMQ during the removal of the first controller is what causes the problem. After deploying the first controller, stopping the RMQ container on any controller including the new centos8 controller will cause the entire cluster to stop. This crash dump appears on the controllers that stopped in sympathy: https://paste.ubuntu.com/p/ZDgFgKtQTB/ This appears in the RMQ log: https://paste.ubuntu.com/p/5D2Qjv3H8c/ -----Original Message----- From: Braden, Albert Sent: Wednesday, March 31, 2021 8:31 AM To: openstack-discuss(a)lists.openstack.org Subject: RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller Centos7: {rabbit,"RabbitMQ","3.7.24"}, "Erlang/OTP 22 [erts-10.7.2.8] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:128] [hipe]\n"}, Centos8: {rabbit,"RabbitMQ","3.7.28"}, "Erlang/OTP 22 [erts-10.7.2.8] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:128] [hipe]\n"}, When I deploy the first Centos8 controller, RMQ comes up with all 3 nodes active and seems to be working fine until I shut down the 2nd controller. The only hint of trouble when I replace the 1st node is this error message the first time I run the deployment: https://paste.ubuntu.com/p/h9HWdfwmrK/ and the crash dump that appears on control2: crash dump log: https://paste.ubuntu.com/p/MpZ8SwTJ2T/ First 1500 lines of the dump: https://paste.ubuntu.com/p/xkCyp2B8j8/ If I wait for a few minutes then RMQ recovers on control2 and the 2nd run of the deployment seems to work, and there is no trouble until I shut down control1. -----Original Message----- From: Mark Goddard <mark(a)stackhpc.com> Sent: Wednesday, March 31, 2021 4:14 AM To: Braden, Albert <C-Albert.Braden(a)charter.com> Cc: openstack-discuss(a)lists.openstack.org Subject: [EXTERNAL] Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. On Tue, 30 Mar 2021 at 13:41, Braden, Albert <C-Albert.Braden(a)charter.com> wrote: > > I’ve created a heat stack and installed Openstack Train to test the Centos7->8 upgrade following the document here: > > > > https://docs.openstack.org/kolla-ansible/train/user/centos8.html#migrating-… > > > > I used the instructions here to successfully remove and replace control0 with a Centos8 box > > > > https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hos… > > > > After this my RMQ admin page shows all 3 nodes up, including the new control0. The name of the cluster is rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-2 /]# rabbitmqctl cluster_status > > Cluster status of node rabbit@chrnc-void-testupgrade-control-2 ... > > [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', > > 'rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}]}, > > {running_nodes,['rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}, > > {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, > > {partitions,[]}, > > {alarms,[{'rabbit@chrnc-void-testupgrade-control-0-replace',[]}, > > {'rabbit@chrnc-void-testupgrade-control-1',[]}, > > {'rabbit@chrnc-void-testupgrade-control-2',[]}]}] > > > > After that I create a new VM to verify that the cluster is still working, and then perform the same procedure on control1. When I shut down services on control1, the ansible playbook finishes successfully: > > > > kolla-ansible -i ../multinode stop --yes-i-really-really-mean-it --limit control1 > > … > > control1 : ok=45 changed=22 unreachable=0 failed=0 skipped=105 rescued=0 ignored=0 > > > > After this my RMQ admin page stops responding. When I check RMQ on the new control0 and the existing control2, the container is still up but RMQ is not running: > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status > > Error: this command requires the 'rabbit' app to be running on the target node. Start it with 'rabbitmqctl start_app'. > > > > If I start it on control0 and control2, then the cluster seems normal and the admin page starts working again, and cluster status looks normal: > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status > > Cluster status of node rabbit@chrnc-void-testupgrade-control-0-replace ... > > [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', > > 'rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}]}, > > {running_nodes,['rabbit@chrnc-void-testupgrade-control-2', > > 'rabbit@chrnc-void-testupgrade-control-0-replace']}, > > {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, > > {partitions,[]}, > > {alarms,[{'rabbit@chrnc-void-testupgrade-control-2',[]}, > > {'rabbit@chrnc-void-testupgrade-control-0-replace',[]}]}] > > > > But my hypervisors are down: > > > > (openstack) [root@chrnc-void-testupgrade-build kolla-ansible]# ohll > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB | > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > | 3 | chrnc-void-testupgrade-compute-2.dev.chtrse.com | QEMU | 172.16.2.106 | down | 5 | 8 | 2560 | 30719 | > > | 6 | chrnc-void-testupgrade-compute-0.dev.chtrse.com | QEMU | 172.16.2.31 | down | 5 | 8 | 2560 | 30719 | > > | 9 | chrnc-void-testupgrade-compute-1.dev.chtrse.com | QEMU | 172.16.0.30 | down | 5 | 8 | 2560 | 30719 | > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > > > When I look at the nova-compute.log on a compute node, I see RMQ failures every 10 seconds: > > > > 172.16.2.31 compute0 > > 2021-03-30 03:07:54.893 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out > > 2021-03-30 03:07:55.905 7 INFO oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] Reconnected to AMQP server on 172.16.1.132:5672 via [amqp] client with port 56422. > > 2021-03-30 03:08:05.915 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out > > > > In the RMQ logs I see this every 10 seconds: > > > > 172.16.1.132 control2 > > [root@chrnc-void-testupgrade-control-2 ~]# tail -f /var/log/kolla/rabbitmq/rabbit\(a)chrnc-void-testupgrade-control-2.log |grep 172.16.2.31 > > 2021-03-30 03:07:54.895 [warning] <0.13247.35> closing AMQP connection <0.13247.35> (172.16.2.31:56420 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): > > client unexpectedly closed TCP connection > > 2021-03-30 03:07:55.901 [info] <0.15288.35> accepting AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) > > 2021-03-30 03:07:55.903 [info] <0.15288.35> Connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) has a client-provided name: nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e > > 2021-03-30 03:07:55.904 [info] <0.15288.35> connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e): user 'openstack' authenticated and granted access to vhost '/' > > 2021-03-30 03:08:05.916 [warning] <0.15288.35> closing AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): > > > > Why does RMQ fail when I shut down the 2nd controller after successfully replacing the first one? Hi Albert, Could you share the versions of RabbitMQ and erlang in both versions of the container? When initially testing this setup, I think we had 3.7.24 on both sides. Perhaps the CentOS 8 version has moved on sufficiently to become incompatible? Mark > > > > I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails. > > > > The contents of this e-mail message and > any attachments are intended solely for the > addressee(s) and may contain confidential > and/or legally privileged information. If you > are not the intended recipient of this message > or if this message has been addressed to you > in error, please immediately alert the sender > by reply e-mail and then delete this message > and any attachments. If you are not the > intended recipient, you are notified that > any use, dissemination, distribution, copying, > or storage of this message or any attachment > is strictly prohibited. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

2 1

[docs][release] Creating Xena's landing pages
by Herve Beraud 06 Apr '21

06 Apr '21

Hello Docs team, This is a friendly reminder from the release team, I think that it should be safe for you to apply your process to create the new release series landing pages for docs.openstack.org. All stable branches are now created. If you want you can do the work before the final release date to avoid having to synchronize with the release team on that day. Let us know if you have any questions. Cheers -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

2 1

[all] Gate resources and performance
by Dan Smith 06 Apr '21

06 Apr '21

Hi all, I have become increasingly concerned with CI performance lately, and have been raising those concerns with various people. Most specifically, I'm worried about our turnaround time or "time to get a result", which has been creeping up lately. Right after the beginning of the year, we had a really bad week where the turnaround time was well over 24 hours. That means if you submit a patch on Tuesday afternoon, you might not get a test result until Thursday. That is, IMHO, a real problem and massively hurts our ability to quickly merge priority fixes as well as just general velocity and morale. If people won't review my code until they see a +1 from Zuul, and that is two days after I submitted it, that's bad. Things have gotten a little better since that week, due in part to getting past a rush of new year submissions (we think) and also due to some job trimming in various places (thanks Neutron!). However, things are still not great. Being in almost the last timezone of the day, the queue is usually so full when I wake up that it's quite often I don't get to see a result before I stop working that day. I would like to ask that projects review their jobs for places where they can cut out redundancy, as well as turn their eyes towards optimizations that can be made. I've been looking at both Nova and Glance jobs and have found some things I think we can do less of. I also wanted to get an idea of who is "using too much" in the way of resources, so I've been working on trying to characterize the weight of the jobs we run for a project, based on the number of worker nodes required to run all the jobs, as well as the wall clock time of how long we tie those up. The results are interesting, I think, and may help us to identify where we see some gains. The idea here is to figure out[1] how many "node hours" it takes to run all the normal jobs on a Nova patch compared to, say, a Neutron one. If the jobs were totally serialized, this is the number of hours a single computer (of the size of a CI worker) would take to do all that work. If the number is 24 hours, that means a single computer could only check *one* patch in a day, running around the clock. I chose the top five projects in terms of usage[2] to report here, as they represent 70% of the total amount of resources consumed. The next five only add up to 13%, so the "top five" seems like a good target group. Here are the results, in order of total consumption: Project % of total Node Hours Nodes ------------------------------------------ 1. TripleO 38% 31 hours 20 2. Neutron 13% 38 hours 32 3. Nova 9% 21 hours 25 4. Kolla 5% 12 hours 18 5. OSA 5% 22 hours 17 What that means is that a single computer (of the size of a CI worker) couldn't even process the jobs required to run on a single patch for Neutron or TripleO in a 24-hour period. Now, we have lots of workers in the gate, of course, but there is also other potential overhead involved in that parallelism, like waiting for nodes to be available for dependent jobs. And of course, we'd like to be able to check more than patch per day. Most projects have smaller gate job sets than check, but assuming they are equivalent, a Neutron patch from submission to commit would undergo 76 hours of testing, not including revisions and not including rechecks. That's an enormous amount of time and resource for a single patch! Now, obviously nobody wants to run fewer tests on patches before they land, and I'm not really suggesting that we take that approach necessarily. However, I think there are probably a lot of places that we can cut down the amount of *work* we do. Some ways to do this are: 1. Evaluate whether or not you need to run all of tempest on two configurations of a devstack on each patch. Maybe having a stripped-down tempest (like just smoke) to run on unique configs, or even specific tests. 2. Revisit your "irrelevant_files" lists to see where you might be able to avoid running heavy jobs on patches that only touch something small. 3. Consider moving some jobs to the experimental queue and run them on-demand for patches that touch particular subsystems or affect particular configurations. 4. Consider some periodic testing for things that maybe don't need to run on every single patch. 5. Re-examine tests that take a long time to run to see if something can be done to make them more efficient. 6. Consider performance improvements in the actual server projects, which also benefits the users. If you're a project that is not in the top ten then your job configuration probably doesn't matter that much, since your usage is dwarfed by the heavy projects. If the heavy projects would consider making changes to decrease their workload, even small gains have the ability to multiply into noticeable improvement. The higher you are on the above list, the more impact a small change will have on the overall picture. Also, thanks to Neutron and TripleO, both of which have already addressed this in some respect, and have other changes on the horizon. Thanks for listening! --Dan 1: https://gist.github.com/kk7ds/5edbfacb2a341bb18df8f8f32d01b37c 2; http://paste.openstack.org/show/C4pwUpdgwUDrpW6V6vnC/

12 36

[keystone][horizon][Victoria] scope-based policy problems
by Adam Tomas 06 Apr '21

06 Apr '21

Hi, I have some questions about horizon and keystone policies : Im trying to achieve "domain_admin" role with the ability to add/remove/update projects in particular domain and add/update/remove users in the same domain (and of course be able to see instances, networks, etc. in this domain). As described here http://lists.openstack.org/pipermail/openstack-discuss/2021-March/021105.ht… <http://lists.openstack.org/pipermail/openstack-discuss/2021-March/021105.ht…> : "Horizon does not support the system-scoped token yet as of Victoria and coming Wallaby.” So there is no way to write json/not scope-based policy for horizon and scope-based policy for keystone, because it will not work due to lack of scope information in horizon’s token? So the question is how the policies should look like? Is it possible at all to achieve such „domain admin” role? How in different way allow one user to add/remove/update projects and add/update/remove users? Another thing is, that if I use something like this in horizon/keystone policy: "identity:list_users_in_group": "rule:admin_required or (role:domain_admin and domain_id:%(domain_id)s)” then (besides of that domain users) there is also admin account in the list (so I assume admin „belongs” to all domains) - how to prevent newly created domain_admin from seeing admin account and making changes to that account? It really holds up my whole project, can you help mi guys? Best regards Adam

1 0

[kolla][glance] Few questions about images.
by Eddie Yen 06 Apr '21

06 Apr '21

Hi everyone, I want to ask about the image operating permission & Windows images issue we met since we can't find any answers on the internet. 1. Now we're still using Rocky with Ceph as storage. Sometimes we need to re-pack the image on the Openstack. We used to save as snapshot (re-pack by Nova ephemeral VM) or upload to image (re-pack by volume), then set snapshot/image's visibility to the public. But since Rocky, we can't do this anymore because when we try to set public, Horizon always shows "not enough permission" error. The workaround we're using for now is creating a nova snapshot after re-pack, download the snapshot, then upload the snapshot again as public. But it's utterly wasting time if the images are huge. So we want to know how to unleash this limitation can let us just change snapshot to public at least. 2. Openstack uses virtio as a network device by default, so we always install a virtio driver when packing Windows images. As the network performance issue in GSO/TSO enablement, we also need to disable them in device properties. But since Windows 10 2004 build (my thought) device properties always reset these settings after Sysprep. We found there's a workaround [1] to solve this issue, but may not work sometimes. Is there a better way to solve this issue? Many thanks, Eddie. [1] PersistAllDeviceInstalls | Microsoft Docs <https://docs.microsoft.com/en-us/windows-hardware/customize/desktop/unatten…>

1 1

RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller
by Braden, Albert 06 Apr '21

06 Apr '21

It looks like the problem may be caused by incompatible versions of RMQ. How can I work around that? -----Original Message----- From: Braden, Albert Sent: Friday, April 2, 2021 8:34 AM To: 'openstack-discuss(a)lists.openstack.org' <openstack-discuss(a)lists.openstack.org> Subject: RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller I opened a bug for this issue: https://bugs.launchpad.net/kolla-ansible/+bug/1922269 -----Original Message----- From: Braden, Albert Sent: Thursday, April 1, 2021 11:34 AM To: 'openstack-discuss(a)lists.openstack.org' <openstack-discuss(a)lists.openstack.org> Subject: RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller Sorry that was a typo. Stopping RMQ during the removal of the *second* controller is what causes the problem. Is there a way to tell Centos 8 Train to use RMQ 3.7.24 instead of 3.7.28? -----Original Message----- From: Braden, Albert Sent: Thursday, April 1, 2021 9:34 AM To: 'openstack-discuss(a)lists.openstack.org' <openstack-discuss(a)lists.openstack.org> Subject: RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller I did some experimenting and it looks like stopping RMQ during the removal of the first controller is what causes the problem. After deploying the first controller, stopping the RMQ container on any controller including the new centos8 controller will cause the entire cluster to stop. This crash dump appears on the controllers that stopped in sympathy: https://paste.ubuntu.com/p/ZDgFgKtQTB/ This appears in the RMQ log: https://paste.ubuntu.com/p/5D2Qjv3H8c/ -----Original Message----- From: Braden, Albert Sent: Wednesday, March 31, 2021 8:31 AM To: openstack-discuss(a)lists.openstack.org Subject: RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller Centos7: {rabbit,"RabbitMQ","3.7.24"}, "Erlang/OTP 22 [erts-10.7.2.8] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:128] [hipe]\n"}, Centos8: {rabbit,"RabbitMQ","3.7.28"}, "Erlang/OTP 22 [erts-10.7.2.8] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:128] [hipe]\n"}, When I deploy the first Centos8 controller, RMQ comes up with all 3 nodes active and seems to be working fine until I shut down the 2nd controller. The only hint of trouble when I replace the 1st node is this error message the first time I run the deployment: https://paste.ubuntu.com/p/h9HWdfwmrK/ and the crash dump that appears on control2: crash dump log: https://paste.ubuntu.com/p/MpZ8SwTJ2T/ First 1500 lines of the dump: https://paste.ubuntu.com/p/xkCyp2B8j8/ If I wait for a few minutes then RMQ recovers on control2 and the 2nd run of the deployment seems to work, and there is no trouble until I shut down control1. -----Original Message----- From: Mark Goddard <mark(a)stackhpc.com> Sent: Wednesday, March 31, 2021 4:14 AM To: Braden, Albert <C-Albert.Braden(a)charter.com> Cc: openstack-discuss(a)lists.openstack.org Subject: [EXTERNAL] Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. On Tue, 30 Mar 2021 at 13:41, Braden, Albert <C-Albert.Braden(a)charter.com> wrote: > > I’ve created a heat stack and installed Openstack Train to test the Centos7->8 upgrade following the document here: > > > > https://docs.openstack.org/kolla-ansible/train/user/centos8.html#migrating-… > > > > I used the instructions here to successfully remove and replace control0 with a Centos8 box > > > > https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hos… > > > > After this my RMQ admin page shows all 3 nodes up, including the new control0. The name of the cluster is rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-2 /]# rabbitmqctl cluster_status > > Cluster status of node rabbit@chrnc-void-testupgrade-control-2 ... > > [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', > > 'rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}]}, > > {running_nodes,['rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}, > > {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, > > {partitions,[]}, > > {alarms,[{'rabbit@chrnc-void-testupgrade-control-0-replace',[]}, > > {'rabbit@chrnc-void-testupgrade-control-1',[]}, > > {'rabbit@chrnc-void-testupgrade-control-2',[]}]}] > > > > After that I create a new VM to verify that the cluster is still working, and then perform the same procedure on control1. When I shut down services on control1, the ansible playbook finishes successfully: > > > > kolla-ansible -i ../multinode stop --yes-i-really-really-mean-it --limit control1 > > … > > control1 : ok=45 changed=22 unreachable=0 failed=0 skipped=105 rescued=0 ignored=0 > > > > After this my RMQ admin page stops responding. When I check RMQ on the new control0 and the existing control2, the container is still up but RMQ is not running: > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status > > Error: this command requires the 'rabbit' app to be running on the target node. Start it with 'rabbitmqctl start_app'. > > > > If I start it on control0 and control2, then the cluster seems normal and the admin page starts working again, and cluster status looks normal: > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status > > Cluster status of node rabbit@chrnc-void-testupgrade-control-0-replace ... > > [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', > > 'rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}]}, > > {running_nodes,['rabbit@chrnc-void-testupgrade-control-2', > > 'rabbit@chrnc-void-testupgrade-control-0-replace']}, > > {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, > > {partitions,[]}, > > {alarms,[{'rabbit@chrnc-void-testupgrade-control-2',[]}, > > {'rabbit@chrnc-void-testupgrade-control-0-replace',[]}]}] > > > > But my hypervisors are down: > > > > (openstack) [root@chrnc-void-testupgrade-build kolla-ansible]# ohll > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB | > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > | 3 | chrnc-void-testupgrade-compute-2.dev.chtrse.com | QEMU | 172.16.2.106 | down | 5 | 8 | 2560 | 30719 | > > | 6 | chrnc-void-testupgrade-compute-0.dev.chtrse.com | QEMU | 172.16.2.31 | down | 5 | 8 | 2560 | 30719 | > > | 9 | chrnc-void-testupgrade-compute-1.dev.chtrse.com | QEMU | 172.16.0.30 | down | 5 | 8 | 2560 | 30719 | > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > > > When I look at the nova-compute.log on a compute node, I see RMQ failures every 10 seconds: > > > > 172.16.2.31 compute0 > > 2021-03-30 03:07:54.893 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out > > 2021-03-30 03:07:55.905 7 INFO oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] Reconnected to AMQP server on 172.16.1.132:5672 via [amqp] client with port 56422. > > 2021-03-30 03:08:05.915 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out > > > > In the RMQ logs I see this every 10 seconds: > > > > 172.16.1.132 control2 > > [root@chrnc-void-testupgrade-control-2 ~]# tail -f /var/log/kolla/rabbitmq/rabbit\(a)chrnc-void-testupgrade-control-2.log |grep 172.16.2.31 > > 2021-03-30 03:07:54.895 [warning] <0.13247.35> closing AMQP connection <0.13247.35> (172.16.2.31:56420 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): > > client unexpectedly closed TCP connection > > 2021-03-30 03:07:55.901 [info] <0.15288.35> accepting AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) > > 2021-03-30 03:07:55.903 [info] <0.15288.35> Connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) has a client-provided name: nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e > > 2021-03-30 03:07:55.904 [info] <0.15288.35> connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e): user 'openstack' authenticated and granted access to vhost '/' > > 2021-03-30 03:08:05.916 [warning] <0.15288.35> closing AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): > > > > Why does RMQ fail when I shut down the 2nd controller after successfully replacing the first one? Hi Albert, Could you share the versions of RabbitMQ and erlang in both versions of the container? When initially testing this setup, I think we had 3.7.24 on both sides. Perhaps the CentOS 8 version has moved on sufficiently to become incompatible? Mark > > > > I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails. > > > > The contents of this e-mail message and > any attachments are intended solely for the > addressee(s) and may contain confidential > and/or legally privileged information. If you > are not the intended recipient of this message > or if this message has been addressed to you > in error, please immediately alert the sender > by reply e-mail and then delete this message > and any attachments. If you are not the > intended recipient, you are notified that > any use, dissemination, distribution, copying, > or storage of this message or any attachment > is strictly prohibited. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

2 1

[openstack-helm] Meeting cancelled
by Gage Hugo 06 Apr '21

06 Apr '21

Hey team, Since there are no agenda items [0] for the IRC meeting today April 6th, the meeting is cancelled. Our next IRC meeting will be April 13th. Thanks [0] https://etherpad.opendev.org/p/openstack-helm-weekly-meeting

1 0

[all][elections][tc] TC Vacancy Special Election Nominations end this week
by Jeremy Stanley 05 Apr '21

05 Apr '21

Just a reminder, nominations for one vacant OpenStack TC (Technical Committee) position are only open for three more days, until Apr 08, 2021 23:45 UTC. All nominations must be submitted as a text file to the openstack/election repository as explained at https://governance.openstack.org/election/#how-to-submit-a-candidacy Please make sure to follow the candidacy file naming convention: candidates/xena/<Project_Name or "TC">/<email_address> (for example, "candidates/xena/TC/stacker(a)example.org"). The name of the file should match an email address for your current OpenStack Foundation Individual Membership. Take this opportunity to ensure that your OSF member profile contains current information: https://www.openstack.org/profile/ Any OpenStack Foundation Individual Member can propose their candidacy for the vacant seat on the Technical Committee. This TC vacancy special election will be held from Apr 8, 2021 23:45 UTC through to Apr 15, 2021 23:45 UTC. The electorate for the TC election are the OpenStack Foundation Individual Members who have a code contribution to one of the official teams over the Victoria to Wallaby timeframe, Apr 24, 2020 00:00 UTC - Mar 08, 2021 00:00 UTC, as well as any Extra ATCs who are acknowledged by the TC. Note that the contribution qualifying period for this special election is being kept the same as what would have been used for the original TC election. The four already elected TC members for this term are listed as candidates in the special election, but will not appear on any resulting poll as they have already been officially elected. Only new candidates in addition to the four elected TC members for this term will appear on a subsequent poll for the TC vacancy special election. Please find below the timeline: nomination starts @ Mar 25, 2021 23:45 UTC nomination ends @ Apr 08, 2021 23:45 UTC elections start @ Apr 08, 2021 23:45 UTC elections end @ Apr 15, 2021 23:45 UTC Shortly after election officials approve candidates, they will be listed on the https://governance.openstack.org/election/ page. If you have any questions please be sure to either ask them on the mailing list or to the elections officials: https://governance.openstack.org/election/#election-officials -- Jeremy Stanley on behalf of the OpenStack Technical Elections Officials

1 0

[tripleo] centos-stream-8 and container-tools 3.0
by Wesley Hayutin 05 Apr '21

05 Apr '21

FYI.. The container-tools 3.0 module has been published by the centos team. We're seeing it land in: http://dashboard-ci.tripleo.org/d/jwDYSidGz/rpm-dependency-pipeline?viewPan… We will be moving ALL the upstream centos-stream-8 jobs to use container-tools 3.0 now. https://review.opendev.org/c/openstack/tripleo-quickstart/+/784770 https://review.opendev.org/c/openstack/tripleo-quickstart/+/784768 0/ happy monday

1 0

[TripleO][ussuri] undercloud install # fails on heat launch
by Ruslanas Gžibovskis 05 Apr '21

05 Apr '21

Hi all, I am trying to understand why undercloud install does not pass the heat step. I feel that it is related to undercloud.conf file, something wrong there? Or some special char which python do not want to understand? How to enable heat-launcher more verbose output? as last line is starting engine and that's it... VERY HELPFULLL!!! AND IT CANNOT CONNECT?! when I launch it manuallly it is running for long time... ?!>!???>?>?>?><>?!>?!!>!?!!>?!!?>!?! AAAAAA PAIN!!!!!! openstack undercloud install last lines: http://paste.openstack.org/show/RCUVMaHa4XwXwRM6IS67/ undercloud.conf: https://raw.githubusercontent.com/qw3r3wq/OSP-ussuri/master/undercloud-v3/u… heat.log: http://paste.openstack.org/show/C5yP3iO8VDk7ugLOHTo6/ -- Ruslanas Gžibovskis +370 6030 7030

2 4

can't build a instance successfully
by 朝阳未烈 04 Apr '21

04 Apr '21

Folks,   I am deploying OpenStack manually and have completed minimal development of the Ussuri. My controller node can find my compute node and confirm there are compute hosts in the database with the instruction:   ” openstack compute service list --service nova-compute”   But when I want to create an instance on the compute node, the status of the compute node just remains “build”. And I try to look for faults from the “/var/log/nova/nova-compute.log” of the compute node:   2021-04-03 00:59:43.379 1432 INFO os_vif [req-2ece8c1c-a96f-4d91-b704-5598c1166016 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] Successfully unplugged vif VIFBridge(active=True,address=fa:16:3e:55:a5:65,bridge_name='brq3169e77c-99',has_traffic_filtering=True,id=28248ef5-6ad6-44bf-b2ce-3fa7ac2371ef,network=Network(3169e77c-9945-454f-9562-6e9a55e1adce),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tap28248ef5-6a') 2021-04-03 00:59:43.380 1432 INFO nova.virt.libvirt.driver [req-2ece8c1c-a96f-4d91-b704-5598c1166016 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] [instance: 3de9dac7-6704-4df9-bd81-cfdadf3ba59a] Deletion of /var/lib/nova/instances/3de9dac7-6704-4df9-bd81-cfdadf3ba59a_del complete 2021-04-03 00:59:43.426 1432 INFO nova.compute.manager [req-2ece8c1c-a96f-4d91-b704-5598c1166016 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] [instance: 3de9dac7-6704-4df9-bd81-cfdadf3ba59a] Took 0.06 seconds to destroy the instance on the hypervisor. 2021-04-03 00:59:43.980 1432 INFO nova.compute.manager [-] [instance: 3de9dac7-6704-4df9-bd81-cfdadf3ba59a] Took 0.55 seconds to deallocate network for instance. 2021-04-03 00:59:44.139 1432 INFO nova.scheduler.client.report [req-2ece8c1c-a96f-4d91-b704-5598c1166016 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] Deleted allocation for instance 3de9dac7-6704-4df9-bd81-cfdadf3ba59a 2021-04-03 01:00:00.763 1432 INFO nova.compute.claims [req-b1aed986-ee9a-46ee-8408-d65a9357f430 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] [instance: 0f53309f-2196-4215-873b-ba05fb99de57] Claim successful on node compute1 2021-04-03 01:00:00.991 1432 INFO nova.virt.libvirt.driver [req-b1aed986-ee9a-46ee-8408-d65a9357f430 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] [instance: 0f53309f-2196-4215-873b-ba05fb99de57] Creating image 2021-04-03 01:00:01.421 1432 INFO oslo.privsep.daemon [req-b1aed986-ee9a-46ee-8408-d65a9357f430 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/nova/nova-dist.conf', '--config-file', '/etc/nova/nova.conf', '--privsep_context', 'nova.privsep.sys_admin_pctxt', '--privsep_sock_path', '/tmp/tmpqsoglfcp/privsep.sock'] 2021-04-03 01:00:02.056 1432 INFO oslo.privsep.daemon [req-b1aed986-ee9a-46ee-8408-d65a9357f430 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] Spawned new privsep daemon via rootwrap 2021-04-03 01:00:01.986 1861 INFO oslo.privsep.daemon [-] privsep daemon starting 2021-04-03 01:00:01.989 1861 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2021-04-03 01:00:01.991 1861 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_CHOWN|CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_FOWNER|CAP_NET_ADMIN|CAP_SYS_ADMIN/CAP_CHOWN|CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_FOWNER|CAP_NET_ADMIN|CAP_SYS_ADMIN/none 2021-04-03 01:00:01.991 1861 INFO oslo.privsep.daemon [-] privsep daemon running as pid 1861 2021-04-03 01:00:02.716 1432 INFO oslo.privsep.daemon [req-b1aed986-ee9a-46ee-8408-d65a9357f430 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/nova/nova-dist.conf', '--config-file', '/etc/nova/nova.conf', '--privsep_context', 'vif_plug_linux_bridge.privsep.vif_plug', '--privsep_sock_path', '/tmp/tmp50wp4hs_/privsep.sock'] 2021-04-03 01:00:03.404 1432 INFO oslo.privsep.daemon [req-b1aed986-ee9a-46ee-8408-d65a9357f430 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] Spawned new privsep daemon via rootwrap 2021-04-03 01:00:03.324 1890 INFO oslo.privsep.daemon [-] privsep daemon starting 2021-04-03 01:00:03.326 1890 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2021-04-03 01:00:03.328 1890 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_NET_ADMIN/CAP_NET_ADMIN/none 2021-04-03 01:00:03.328 1890 INFO oslo.privsep.daemon [-] privsep daemon running as pid 1890 2021-04-03 01:00:03.620 1432 INFO os_vif [req-b1aed986-ee9a-46ee-8408-d65a9357f430 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] Successfully plugged vif VIFBridge(active=False,address=fa:16:3e:e6:b4:89,bridge_name='brq3169e77c-99',has_traffic_filtering=True,id=c18f70ac-31ac-40eb-90f0-72c4e559f712,network=Network(3169e77c-9945-454f-9562-6e9a55e1adce),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tapc18f70ac-31') 2021-04-03 01:00:04.177 1432 INFO nova.compute.manager [-] [instance: 0f53309f-2196-4215-873b-ba05fb99de57] VM 已开始 (Lifecycle Event) 2021-04-03 01:00:04.202 1432 INFO nova.compute.manager [req-f136de80-ad6d-497a-b190-4181d739c8ba - - - - -] [instance: 0f53309f-2196-4215-873b-ba05fb99de57] VM 已暂停 (Lifecycle Event) 2021-04-03 01:00:04.246 1432 INFO nova.compute.manager [req-f136de80-ad6d-497a-b190-4181d739c8ba - - - - -] [instance: 0f53309f-2196-4215-873b-ba05fb99de57] During sync_power_state the instance has a pending task (spawning). Skip. 2021-04-03 01:00:36.581 1432 INFO nova.compute.manager [req-eb71729b-0263-4577-8248-d8c9f69387ca - - - - -] Updating bandwidth usage cache 2021-04-03 01:00:51.700 1432 ERROR oslo.messaging._drivers.impl_rabbit [req-eb71729b-0263-4577-8248-d8c9f69387ca - - - - -] Connection failed: timed out (retrying in 2.0 seconds): socket.timeout: timed out 2021-04-03 01:01:08.908 1432 ERROR oslo.messaging._drivers.impl_rabbit [req-eb71729b-0263-4577-8248-d8c9f69387ca - - - - -] Connection failed: timed out (retrying in 4.0 seconds): socket.timeout: timed out 2021-04-03 01:01:13.358 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8ded131-047e-4acb-a226-881596889c26] AMQP server on controller:5672 is unreachable: timed out. Trying again in 1 seconds.: socket.timeout: timed out 2021-04-03 01:01:27.960 1432 ERROR oslo.messaging._drivers.impl_rabbit [req-eb71729b-0263-4577-8248-d8c9f69387ca - - - - -] Connection failed: timed out (retrying in 6.0 seconds): socket.timeout: timed out 2021-04-03 01:01:37.983 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8ded131-047e-4acb-a226-881596889c26] AMQP server on controller:5672 is unreachable: timed out. Trying again in 2 seconds.: socket.timeout: timed out 2021-04-03 01:01:55.240 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8ded131-047e-4acb-a226-881596889c26] AMQP server on controller:5672 is unreachable: timed out. Trying again in 4 seconds.: socket.timeout: timed out 2021-04-03 01:02:05.262 1432 ERROR oslo.messaging._drivers.impl_rabbit [req-eb71729b-0263-4577-8248-d8c9f69387ca - - - - -] Connection failed: timed out (retrying in 8.0 seconds): socket.timeout: timed out 2021-04-03 01:02:05.262 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [3f1dbf14-730b-4f61-94b2-8674c243c457] AMQP server on controller:5672 is unreachable: Too many heartbeats missed. Trying again in 1 seconds.: amqp.exceptions.ConnectionForced: Too many heartbeats missed 2021-04-03 01:02:05.263 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [03922bfe-3f9c-41dd-8219-edd7bfe33e5f] AMQP server on controller:5672 is unreachable: Too many heartbeats missed. Trying again in 1 seconds.: amqp.exceptions.ConnectionForced: Too many heartbeats missed 2021-04-03 01:02:21.319 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [3f1dbf14-730b-4f61-94b2-8674c243c457] AMQP server on controller:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error> 2021-04-03 01:02:31.335 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [03922bfe-3f9c-41dd-8219-edd7bfe33e5f] AMQP server on controller:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error> 2021-04-03 01:02:41.351 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8ded131-047e-4acb-a226-881596889c26] AMQP server on controller:5672 is unreachable: timed out. Trying again in 6 seconds.: socket.timeout: timed out 2021-04-03 01:02:56.387 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [3f1dbf14-730b-4f61-94b2-8674c243c457] AMQP server on controller:5672 is unreachable: timed out. Trying again in 2 seconds.: socket.timeout: timed out 2021-04-03 01:03:06.406 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [03922bfe-3f9c-41dd-8219-edd7bfe33e5f] AMQP server on controller:5672 is unreachable: timed out. Trying again in 2 seconds.: socket.timeout: timed out 2021-04-03 01:03:16.417 1432 ERROR oslo.messaging._drivers.impl_rabbit [req-eb71729b-0263-4577-8248-d8c9f69387ca - - - - -] Connection failed: timed out (retrying in 10.0 seconds): socket.timeout: timed out 2021-04-03 01:03:31.457 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8ded131-047e-4acb-a226-881596889c26] AMQP server on controller:5672 is unreachable: timed out. Trying again in 8 seconds.: socket.timeout: timed out 2021-04-03 01:03:41.475 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [3f1dbf14-730b-4f61-94b2-8674c243c457] AMQP server on controller:5672 is unreachable: timed out. Trying again in 4 seconds.: socket.timeout: timed out 2021-04-03 01:03:51.495 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [03922bfe-3f9c-41dd-8219-edd7bfe33e5f] AMQP server on controller:5672 is unreachable: timed out. Trying again in 4 seconds.: socket.timeout: timed out 2021-04-03 01:04:09.540 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [3f1dbf14-730b-4f61-94b2-8674c243c457] AMQP server on controller:5672 is unreachable: timed out. Trying again in 6 seconds.: socket.timeout: timed out 2021-04-03 01:04:19.558 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [03922bfe-3f9c-41dd-8219-edd7bfe33e5f] AMQP server on controller:5672 is unreachable: timed out. Trying again in 6 seconds.: socket.timeout: timed out 2021-04-03 01:04:29.565 1432 ERROR oslo.messaging._drivers.impl_rabbit [req-eb71729b-0263-4577-8248-d8c9f69387ca - - - - -] Connection failed: timed out (retrying in 12.0 seconds): socket.timeout: timed out 2021-04-03 01:04:39.570 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8ded131-047e-4acb-a226-881596889c26] AMQP server on controller:5672 is unreachable: timed out. Trying again in 10 seconds.: socket.timeout: timed out 2021-04-03 01:04:59.594 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [3f1dbf14-730b-4f61-94b2-8674c243c457] AMQP server on controller:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error> 2021-04-03 01:05:09.607 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [03922bfe-3f9c-41dd-8219-edd7bfe33e5f] AMQP server on controller:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error> 2021-04-03 01:05:09.610 1432 WARNING nova.virt.libvirt.driver [req-b1aed986-ee9a-46ee-8408-d65a9357f430 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] [instance: 0f53309f-2196-4215-873b-ba05fb99de57] Timeout waiting for [('network-vif-plugged', 'c18f70ac-31ac-40eb-90f0-72c4e559f712')] for instance with vm_state building and task_state spawning.: eventlet.timeout.Timeout: 300 seconds   And then controller node couldn’t find the compute node I don’t know why these happen. I have checked the port of 5672 is open on controller node and Rabbitmq is running.  Please help me.

2 1

[Ironic][RFE] Enable support for espresso machines in Ironic
by Jacob Anders 04 Apr '21

04 Apr '21

Hi There, I was discussing this RFE with Julia and we decided it would be great to get some feedback on it from the wider community, ideally by the end of today. Apologies for the short notice. Here's the story: https://storyboard.openstack.org/#!/story/2008791 What are your thoughts on this? Please comment in the Story or just reply to thread. Thank you, Jacob

14 18

ML2/OVN DVR question
by Satish Patel 03 Apr '21

03 Apr '21

Folks, I have deployed openstack using ML2/OVN on 1 controller and 2 compute nodes so far everything is working fine, when i configured router it by default used L3HA and i can see active-backup router on both compute nodes. currently my all SNAT traffic going out using compute-1 I don't want bottleneck in network so i am looking for DVR deployment and after reading i found tenant VLAN doesn't support DVR https://bugzilla.redhat.com/show_bug.cgi?id=1704596 After doing more research i found that if i set manually external_mac using the following command then my vm using local compute node to send traffic in/out just like DVR instead of centralized design. root@os-infra-1-neutron-ovn-northd-container-24eea9c2:~# ovn-nbctl find NAT type=dnat_and_snat _uuid : 99bdd866-01ed-425d-853b-9362ae8572c9 external_ids : {"neutron:fip_external_mac"="fa:16:3e:2d:7e:fa", "neutron:fip_id"="025a912a-c0ee-4f36-98ad-8992bd825cfc", "neutron:fip_network_id"="9cccf39d-edba-4069-91ef-5f30afbb6604", "neutron:fip_port_id"="70ad361a-b42e-403b-a5c1-4ee39ddf5e31", "neutron:revision_number"="6", "neutron:router_name"=neutron-8af10b06-c8de-4166-9ab1-ca2f775b08a8} external_ip : "10.40.255.10" external_mac : [] logical_ip : "172.168.0.164" logical_port : "70ad361a-b42e-403b-a5c1-4ee39ddf5e31" options : {} type : dnat_and_snat _uuid : c438e7be-5ff4-472e-b053-8d6ed74cd4dc external_ids : {"neutron:fip_external_mac"="fa:16:3e:f5:9f:f0", "neutron:fip_id"="31e8cb44-0acd-453b-a4e6-39f6ab3a6da4", "neutron:fip_network_id"="9cccf39d-edba-4069-91ef-5f30afbb6604", "neutron:fip_port_id"="44a677c5-86ff-4b6b-a046-54e79f79c4cd", "neutron:revision_number"="2", "neutron:router_name"=neutron-8af10b06-c8de-4166-9ab1-ca2f775b08a8} external_ip : "10.40.255.5" external_mac : [] logical_ip : "172.168.0.67" logical_port : "44a677c5-86ff-4b6b-a046-54e79f79c4cd" options : {} type : dnat_and_snat This is how i set external mac from fip_external_mac"="fa:16:3e:2d:7e:fa" in above command. ovn-nbctl set NAT 99bdd866-01ed-425d-853b-9362ae8572c9 external_mac="fa\:16\:3e\:2d\:7e\:fa" How do i make this behavior default for every single VM, i don't want to do this manually to set the external mac address of each FIP?

1 1

launch VM on volume vs. image
by Tony Liu 03 Apr '21

03 Apr '21

Hi, With Ceph as the backend storage, launching a VM on volume takes much longer than launching on image. Why is that? Could anyone elaborate the high level workflow for those two cases? Thanks! Tony

3 9

[Requirements][all] Requirements branched, freeze lifted
by Matthew Thode 02 Apr '21

02 Apr '21

The requirements freeze is now lifted. If your project has not branched please be aware that master moves on (to xena). -- Matthew Thode

1 0

RE: [Interop][Refstack] No meeting this Friday meeting
by Kanevsky, Arkady 02 Apr '21

02 Apr '21

Team, No meeting this Friday. Happy Holiday! Will update Etherpad. Cheers, Arkady From: prakash RAMCHANDRAN <pramchan(a)yahoo.com> Sent: Thursday, April 1, 2021 1:12 PM To: Jimmy McArthur; Vida Haririan Cc: Ghanshyam Mann; Kanevsky, Arkady; openstack-discuss; Martin Kopec; Goutham Pacha Ravi Subject: Re: [Interop][Refstack] this Friday meeting [EXTERNAL EMAIL] Looks like we can skip this Friday call and sure Arkady - lets cancel it. If you have something urgent we can talk offline - Thanks Prakash On Thursday, April 1, 2021, 11:06:25 AM PDT, Vida Haririan <vhariria(a)redhat.com<mailto:vhariria@redhat.com>> wrote: Hi Arkady, Friday is a company holiday and I will be ooo. Thanks, Vida On Thu, Apr 1, 2021 at 11:10 AM Jimmy McArthur <jimmy(a)openstack.org<mailto:jimmy@openstack.org>> wrote: I forgot this is a holiday. Same on my side. Thanks, Jimmy > On Apr 1, 2021, at 9:25 AM, Ghanshyam Mann <gmann(a)ghanshyammann.com<mailto:gmann@ghanshyammann.com>> wrote: > > > ---- On Thu, 01 Apr 2021 09:15:21 -0500 Kanevsky, Arkady <Arkady.Kanevsky(a)dell.com<mailto:Arkady.Kanevsky@dell.com>> wrote ---- >> >> Team, >> This Friday is Good Friday and some people have a day off. >> Should we cancel this week meeting? >> Please, respond so we can see if we will have quorum. > > Thanks Arkady, > > I will be off from work and would not be able to join. > > -gmann > >> Thanks, >> Arkady >> >> Arkady Kanevsky, Ph.D. >> SP Chief Technologist & DE >> Dell Technologies office of CTO >> Dell Inc. One Dell Way, MS PS2-91 >> Round Rock, TX 78682, USA >> Phone: 512 7204955 >> >> >

2 1

RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller
by Braden, Albert 02 Apr '21

02 Apr '21

I opened a bug for this issue: https://bugs.launchpad.net/kolla-ansible/+bug/1922269 -----Original Message----- From: Braden, Albert Sent: Thursday, April 1, 2021 11:34 AM To: 'openstack-discuss(a)lists.openstack.org' <openstack-discuss(a)lists.openstack.org> Subject: RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller Sorry that was a typo. Stopping RMQ during the removal of the *second* controller is what causes the problem. Is there a way to tell Centos 8 Train to use RMQ 3.7.24 instead of 3.7.28? -----Original Message----- From: Braden, Albert Sent: Thursday, April 1, 2021 9:34 AM To: 'openstack-discuss(a)lists.openstack.org' <openstack-discuss(a)lists.openstack.org> Subject: RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller I did some experimenting and it looks like stopping RMQ during the removal of the first controller is what causes the problem. After deploying the first controller, stopping the RMQ container on any controller including the new centos8 controller will cause the entire cluster to stop. This crash dump appears on the controllers that stopped in sympathy: https://paste.ubuntu.com/p/ZDgFgKtQTB/ This appears in the RMQ log: https://paste.ubuntu.com/p/5D2Qjv3H8c/ -----Original Message----- From: Braden, Albert Sent: Wednesday, March 31, 2021 8:31 AM To: openstack-discuss(a)lists.openstack.org Subject: RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller Centos7: {rabbit,"RabbitMQ","3.7.24"}, "Erlang/OTP 22 [erts-10.7.2.8] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:128] [hipe]\n"}, Centos8: {rabbit,"RabbitMQ","3.7.28"}, "Erlang/OTP 22 [erts-10.7.2.8] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:128] [hipe]\n"}, When I deploy the first Centos8 controller, RMQ comes up with all 3 nodes active and seems to be working fine until I shut down the 2nd controller. The only hint of trouble when I replace the 1st node is this error message the first time I run the deployment: https://paste.ubuntu.com/p/h9HWdfwmrK/ and the crash dump that appears on control2: crash dump log: https://paste.ubuntu.com/p/MpZ8SwTJ2T/ First 1500 lines of the dump: https://paste.ubuntu.com/p/xkCyp2B8j8/ If I wait for a few minutes then RMQ recovers on control2 and the 2nd run of the deployment seems to work, and there is no trouble until I shut down control1. -----Original Message----- From: Mark Goddard <mark(a)stackhpc.com> Sent: Wednesday, March 31, 2021 4:14 AM To: Braden, Albert <C-Albert.Braden(a)charter.com> Cc: openstack-discuss(a)lists.openstack.org Subject: [EXTERNAL] Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. On Tue, 30 Mar 2021 at 13:41, Braden, Albert <C-Albert.Braden(a)charter.com> wrote: > > I’ve created a heat stack and installed Openstack Train to test the Centos7->8 upgrade following the document here: > > > > https://docs.openstack.org/kolla-ansible/train/user/centos8.html#migrating-… > > > > I used the instructions here to successfully remove and replace control0 with a Centos8 box > > > > https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hos… > > > > After this my RMQ admin page shows all 3 nodes up, including the new control0. The name of the cluster is rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-2 /]# rabbitmqctl cluster_status > > Cluster status of node rabbit@chrnc-void-testupgrade-control-2 ... > > [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', > > 'rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}]}, > > {running_nodes,['rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}, > > {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, > > {partitions,[]}, > > {alarms,[{'rabbit@chrnc-void-testupgrade-control-0-replace',[]}, > > {'rabbit@chrnc-void-testupgrade-control-1',[]}, > > {'rabbit@chrnc-void-testupgrade-control-2',[]}]}] > > > > After that I create a new VM to verify that the cluster is still working, and then perform the same procedure on control1. When I shut down services on control1, the ansible playbook finishes successfully: > > > > kolla-ansible -i ../multinode stop --yes-i-really-really-mean-it --limit control1 > > … > > control1 : ok=45 changed=22 unreachable=0 failed=0 skipped=105 rescued=0 ignored=0 > > > > After this my RMQ admin page stops responding. When I check RMQ on the new control0 and the existing control2, the container is still up but RMQ is not running: > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status > > Error: this command requires the 'rabbit' app to be running on the target node. Start it with 'rabbitmqctl start_app'. > > > > If I start it on control0 and control2, then the cluster seems normal and the admin page starts working again, and cluster status looks normal: > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status > > Cluster status of node rabbit@chrnc-void-testupgrade-control-0-replace ... > > [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', > > 'rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}]}, > > {running_nodes,['rabbit@chrnc-void-testupgrade-control-2', > > 'rabbit@chrnc-void-testupgrade-control-0-replace']}, > > {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, > > {partitions,[]}, > > {alarms,[{'rabbit@chrnc-void-testupgrade-control-2',[]}, > > {'rabbit@chrnc-void-testupgrade-control-0-replace',[]}]}] > > > > But my hypervisors are down: > > > > (openstack) [root@chrnc-void-testupgrade-build kolla-ansible]# ohll > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB | > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > | 3 | chrnc-void-testupgrade-compute-2.dev.chtrse.com | QEMU | 172.16.2.106 | down | 5 | 8 | 2560 | 30719 | > > | 6 | chrnc-void-testupgrade-compute-0.dev.chtrse.com | QEMU | 172.16.2.31 | down | 5 | 8 | 2560 | 30719 | > > | 9 | chrnc-void-testupgrade-compute-1.dev.chtrse.com | QEMU | 172.16.0.30 | down | 5 | 8 | 2560 | 30719 | > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > > > When I look at the nova-compute.log on a compute node, I see RMQ failures every 10 seconds: > > > > 172.16.2.31 compute0 > > 2021-03-30 03:07:54.893 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out > > 2021-03-30 03:07:55.905 7 INFO oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] Reconnected to AMQP server on 172.16.1.132:5672 via [amqp] client with port 56422. > > 2021-03-30 03:08:05.915 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out > > > > In the RMQ logs I see this every 10 seconds: > > > > 172.16.1.132 control2 > > [root@chrnc-void-testupgrade-control-2 ~]# tail -f /var/log/kolla/rabbitmq/rabbit\(a)chrnc-void-testupgrade-control-2.log |grep 172.16.2.31 > > 2021-03-30 03:07:54.895 [warning] <0.13247.35> closing AMQP connection <0.13247.35> (172.16.2.31:56420 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): > > client unexpectedly closed TCP connection > > 2021-03-30 03:07:55.901 [info] <0.15288.35> accepting AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) > > 2021-03-30 03:07:55.903 [info] <0.15288.35> Connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) has a client-provided name: nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e > > 2021-03-30 03:07:55.904 [info] <0.15288.35> connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e): user 'openstack' authenticated and granted access to vhost '/' > > 2021-03-30 03:08:05.916 [warning] <0.15288.35> closing AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): > > > > Why does RMQ fail when I shut down the 2nd controller after successfully replacing the first one? Hi Albert, Could you share the versions of RabbitMQ and erlang in both versions of the container? When initially testing this setup, I think we had 3.7.24 on both sides. Perhaps the CentOS 8 version has moved on sufficiently to become incompatible? Mark > > > > I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails. > > > > The contents of this e-mail message and > any attachments are intended solely for the > addressee(s) and may contain confidential > and/or legally privileged information. If you > are not the intended recipient of this message > or if this message has been addressed to you > in error, please immediately alert the sender > by reply e-mail and then delete this message > and any attachments. If you are not the > intended recipient, you are notified that > any use, dissemination, distribution, copying, > or storage of this message or any attachment > is strictly prohibited. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

1 0

PTO Monday April 5
by Herve Beraud 02 Apr '21

02 Apr '21

Hello Monday is a public holiday in France. I'll be back Tuesday. Have a nice weekend! -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

1 1

[infra][puppet] No verified score posted by zuul
by Takashi Kajinami 02 Apr '21

02 Apr '21

Hi I have asked for some help in #openstack-infra but didn't get any solution so far Many people might be now on holidays (enjoy holidays!), so let me send this email so that people involved can find this mail after getting back. A few days ago Wallaby release of openstack puppet modules were created, and the release bot submitted release patches. However for some patches zuul doesn't return any CI result(it doesn't put verified score)[1]. I posted +2+A on [2] but it is not merged, because it is not verified by zuul. I tried "recheck" but it didn't solve the problem. [1] https://review.opendev.org/c/openstack/puppet-aodh/+/784213/1 [2] https://review.opendev.org/c/openstack/puppet-cloudkitty/+/784230 Currently we don't have any job triggered for the change with .gitreview and I guess that is why we don't get verified. Actually I see that the same patch for puppet-oslo got verified +1, because tripleo job was unexpectedly triggered for the change in .gitreview [2] https://review.opendev.org/c/openstack/puppet-oslo/+/784302 The easiest solution would be to manually squash these two patches into one. However I remember that we did get verified when we created the last Victoria release, and I suspect some change in infra side which resulted in this situation. So it would be nice if I can ask some insights from infra team about this situation. Thank you, Takashi Kajinami

1 1

[release] Release countdown for week R-1 Apr 05 - Apr 09
by Herve Beraud 01 Apr '21

01 Apr '21

Development Focus ----------------- We are on the final mile of the Wallaby development cycle! Remember that the Wallaby final release will include the latest release candidate (for cycle-with-rc deliverables) or the latest intermediary release (for cycle-with-intermediary deliverables) available. April 8 is the deadline for final Wallaby release candidates as well as any last cycle-with-intermediary deliverables. We will then enter a quiet period until we tag the final release on 14 April, 2021. Teams should be prioritizing fixing release-critical bugs, before that deadline. Otherwise it's time to start planning the Xena development cycle, including discussing Forum and PTG sessions content, in preparation of PTG on the week of April 19. Actions ------- Watch for any translation patches coming through on the stable/wallaby branch and merge them quickly. If you discover a release-critical issue, please make sure to fix it on the master branch first, then backport the bugfix to the stable/wallaby branch before triggering a new release. Please drop by #openstack-release with any questions or concerns about the upcoming release! Upcoming Deadlines & Dates -------------------------- Final Wallaby release: 14 April, 2021 Xena virtual PTG: 19 - 23 April, 2021 Thanks for your attention -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

1 0

[ironic] Review Jams
by Julia Kreger 01 Apr '21

01 Apr '21

In the Ironic team's recent mid-cycle call, we discussed the need to return to occasionally having review jams in order to help streamline the review process. In other words, get eyes on a change in parallel and be able to discuss the change. The goal is to help get people on the same page in terms of what and why. Be on hand to answer questions or back-fill context. This is to hopefully avoid the more iterative back and forth nature of code review, which can draw out a long chain of patches. As always, the goal is not perfection, but forward movement especially for complex changes. We've established two time windows that will hopefully not to be too hard for some contributors to make it to. It doesn't need to be everyone, but it would help for at least some people whom actively review or want to actively participate in reviewing, or whom are even interested in a feature to join us for our meeting. I've added an entry on to our wiki page to cover this, with the current agenda and anticipated review jam topic schedule. The tl;dr is we will use meetpad[1] and meet on Mondays at 2 PM UTC and Tuesdays at 6 PM UTC. The hope is to to enable some overlap of reviewers. If people are interested in other times, please bring this up in the weekly meeting or on the mailing list. I'm not sending out calendar invites for this. Yet. :) See everyone next week! -Julia [0]: https://wiki.openstack.org/wiki/Meetings/Ironic#Review_Jams [1]: https://meetpad.opendev.org/ironic

3 2

[oslo][release] devstack-plugin-(kafka|amqp1) retirement
by Herve Beraud 01 Apr '21

01 Apr '21

Hello Osloers, Our devstack plugins (kafka and amqp1) didn't show a great amount of activity since ussuri, does it still make sense to maintain them? The latest available SHAs for the both projects comes from Victoria (merged in this period). Can we retire them or simply retire them from the coordinated releases? We (the release team) would appreciate some feedback about this point. Let's open the debat. -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

5 10

[tripleo][ci] jobs in retry_limit or skipped
by Wesley Hayutin 01 Apr '21

01 Apr '21

Greetings, Just FYI.. I believe we hit a bump in the road in upstream infra ( not sure yet ). It appears to be global and not isolated to tripleo or centos based jobs. I have a tripleo bug to track it. https://bugs.launchpad.net/tripleo/+bug/1922148 See #opendev for details, it looks like infra is very busy working and fixing the issues atm. http://eavesdrop.openstack.org/irclogs/%23opendev/%23opendev.2021-03-31.log… http://eavesdrop.openstack.org/irclogs/%23opendev/latest.log.html

3 2

[ironic] IPA image does not want to boot with UEFI
by Vuk Gojnic 01 Apr '21

01 Apr '21

Hello everybody, I am using Ironic standalone to provision the HPE Gen10+ node via iLO driver. Ironic version is 16.0.1. Server is configured with UEFI boot mode. Everything on Ironic side works fine. It creates ISO image, powers the server on and configures it to boot from it. Here is the what /var/log/ironic/ironic-conductor.log says: 2021-03-31 17:46:25.541 2618460 INFO ironic.conductor.task_manager [req-b6a19234-5f4b-4852-ab41-4102b9016bb0 - - - - -] Node ed25569f-c107-4fe0-95cd-74fcad9ab3f0 moved to provision state "cleaning" from state "manageable"; target provision state is "available" 2021-03-31 17:46:32.066 2618460 INFO ironic.drivers.modules.ilo.power [req-b6a19234-5f4b-4852-ab41-4102b9016bb0 - - - - -] The node ed25569f-c107-4fe0-95cd-74fcad9ab3f0 operation of 'power off' is completed in 4 seconds. 2021-03-31 17:46:32.088 2618460 INFO ironic.conductor.utils [req-b6a19234-5f4b-4852-ab41-4102b9016bb0 - - - - -] Successfully set node ed25569f-c107-4fe0-95cd-74fcad9ab3f0 power state to power off by power off. 2021-03-31 17:46:34.510 2618460 INFO ironic.drivers.modules.ilo.common [req-b6a19234-5f4b-4852-ab41-4102b9016bb0 - - - - -] Node ed25569f-c107-4fe0-95cd-74fcad9ab3f0 pending boot mode is uefi. 2021-03-31 17:46:37.248 2618460 INFO ironic.drivers.modules.ilo.common [req-b6a19234-5f4b-4852-ab41-4102b9016bb0 - - - - -] Set the node ed25569f-c107-4fe0-95cd-74fcad9ab3f0 to boot from URL https://10.23.137.234/tmp-images/ilo/boot-ed25569f-c107-4fe0-95cd-74fcad9ab… successfully. 2021-03-31 17:46:48.367 2618460 INFO ironic.drivers.modules.ilo.power [req-b6a19234-5f4b-4852-ab41-4102b9016bb0 - - - - -] The node ed25569f-c107-4fe0-95cd-74fcad9ab3f0 operation of 'power on' is completed in 8 seconds. 2021-03-31 17:46:48.388 2618460 INFO ironic.conductor.utils [req-b6a19234-5f4b-4852-ab41-4102b9016bb0 - - - - -] Successfully set node ed25569f-c107-4fe0-95cd-74fcad9ab3f0 power state to power on by rebooting. 2021-03-31 17:46:48.404 2618460 INFO ironic.conductor.task_manager [req-b6a19234-5f4b-4852-ab41-4102b9016bb0 - - - - -] Node ed25569f-c107-4fe0-95cd-74fcad9ab3f0 moved to provision state "clean wait" from state "cleaning"; target provision state is "available" The Grub2 starts and after I select the option “boot_partition", it starts booting and immediately freezes showing just black screen with static red underscore character. I have tried with pre-built IPA images (see below) as well as with custom IPA images made with Ubuntu 18.04 and 20.04 (built using ironic-python-agent-builder) but it is all the same. Does somebody have idea what is the problem with IPA and UEFI in this particular scenario? Output of “openstack baremetal node show” command: allocation_uuid: null automated_clean: null bios_interface: no-bios boot_interface: ilo-uefi-https chassis_uuid: null clean_step: {} conductor: 10.23.137.234 conductor_group: '' console_enabled: false console_interface: no-console created_at: '2021-03-21T13:54:25+00:00' deploy_interface: direct deploy_step: {} description: null driver: ilo5 driver_info: ilo_address: 10.23.137.137 ilo_bootloader: https://ironic-images/Images/esp.img ilo_deploy_kernel: https://ironic-images/Images/ipa-centos8-stable-victoria.kernel ilo_deploy_ramdisk: https://ironic-images/Images/ipa-centos8-stable-victoria.initramfs ilo_password: '******' ilo_username: Administrator snmp_auth_priv_password: '******' snmp_auth_prot_password: '******' snmp_auth_user: iloinspect driver_internal_info: agent_continue_if_ata_erase_failed: false agent_enable_ata_secure_erase: true agent_erase_devices_iterations: 1 agent_erase_devices_zeroize: true agent_erase_skip_read_only: false agent_secret_token: '******' agent_secret_token_pregenerated: true clean_steps: null disk_erasure_concurrency: 1 last_power_state_change: '2021-03-31T17:46:37.894667' extra: {} fault: clean failure inspect_interface: ilo inspection_finished_at: '2021-03-21T13:57:33+00:00' inspection_started_at: null instance_info: deploy_boot_mode: uefi instance_uuid: null last_error: null lessee: null maintenance: true maintenance_reason: management_interface: ilo5 name: null network_data: {} network_interface: noop owner: null power_interface: ilo power_state: power on properties: cpu_arch: x86 cpus: 64 local_gb: 2979 memory_mb: 262144 protected: false protected_reason: null provision_state: clean wait provision_updated_at: '2021-03-31T17:46:48+00:00' raid_config: {} raid_interface: no-raid rescue_interface: no-rescue reservation: null resource_class: null retired: false retired_reason: null storage_interface: noop target_power_state: null target_provision_state: available target_raid_config: {} traits: [] updated_at: '2021-03-31T17:46:48+00:00' uuid: ed25569f-c107-4fe0-95cd-74fcad9ab3f0 vendor_interface: no-vendor Many thanks! Vuk Gojnic Deutsche Telekom Technik GmbH Services & Plattforms (T-SP) Tribe Data Center Infrastructure (T-DCI) Super Squad Cloud Platforms Lifecycle (SSQ-CP) Vuk Gojnic Kubernetes Engine Squad Lead

2 1

How to customize the xml used in libvirt from GUI/opestack command line?
by Evan Zhao 01 Apr '21

01 Apr '21

Hi there, I googled this question and found two major answers: 1. ssh to the compute node and use `virsh dumpxml` and `virsh undefine/define`, etc. 2. edit nova/virt/libvrit/config.py directly. However, it's trivial to ssh to each node and do the modification, and I prefer not to touch the nova source code, is there any better ways to achieve this? I expect to edit the namespace of a certain element and append an additional element <qemu:command> to the xml file. Any information will be appreciated.

2 1

RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller
by Braden, Albert 01 Apr '21

01 Apr '21

I did some experimenting and it looks like stopping RMQ during the removal of the first controller is what causes the problem. After deploying the first controller, stopping the RMQ container on any controller including the new centos8 controller will cause the entire cluster to stop. This crash dump appears on the controllers that stopped in sympathy: https://paste.ubuntu.com/p/ZDgFgKtQTB/ This appears in the RMQ log: https://paste.ubuntu.com/p/5D2Qjv3H8c/ -----Original Message----- From: Braden, Albert Sent: Wednesday, March 31, 2021 8:31 AM To: openstack-discuss(a)lists.openstack.org Subject: RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller Centos7: {rabbit,"RabbitMQ","3.7.24"}, "Erlang/OTP 22 [erts-10.7.2.8] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:128] [hipe]\n"}, Centos8: {rabbit,"RabbitMQ","3.7.28"}, "Erlang/OTP 22 [erts-10.7.2.8] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:128] [hipe]\n"}, When I deploy the first Centos8 controller, RMQ comes up with all 3 nodes active and seems to be working fine until I shut down the 2nd controller. The only hint of trouble when I replace the 1st node is this error message the first time I run the deployment: https://paste.ubuntu.com/p/h9HWdfwmrK/ and the crash dump that appears on control2: crash dump log: https://paste.ubuntu.com/p/MpZ8SwTJ2T/ First 1500 lines of the dump: https://paste.ubuntu.com/p/xkCyp2B8j8/ If I wait for a few minutes then RMQ recovers on control2 and the 2nd run of the deployment seems to work, and there is no trouble until I shut down control1. -----Original Message----- From: Mark Goddard <mark(a)stackhpc.com> Sent: Wednesday, March 31, 2021 4:14 AM To: Braden, Albert <C-Albert.Braden(a)charter.com> Cc: openstack-discuss(a)lists.openstack.org Subject: [EXTERNAL] Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. On Tue, 30 Mar 2021 at 13:41, Braden, Albert <C-Albert.Braden(a)charter.com> wrote: > > I’ve created a heat stack and installed Openstack Train to test the Centos7->8 upgrade following the document here: > > > > https://docs.openstack.org/kolla-ansible/train/user/centos8.html#migrating-… > > > > I used the instructions here to successfully remove and replace control0 with a Centos8 box > > > > https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hos… > > > > After this my RMQ admin page shows all 3 nodes up, including the new control0. The name of the cluster is rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-2 /]# rabbitmqctl cluster_status > > Cluster status of node rabbit@chrnc-void-testupgrade-control-2 ... > > [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', > > 'rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}]}, > > {running_nodes,['rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}, > > {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, > > {partitions,[]}, > > {alarms,[{'rabbit@chrnc-void-testupgrade-control-0-replace',[]}, > > {'rabbit@chrnc-void-testupgrade-control-1',[]}, > > {'rabbit@chrnc-void-testupgrade-control-2',[]}]}] > > > > After that I create a new VM to verify that the cluster is still working, and then perform the same procedure on control1. When I shut down services on control1, the ansible playbook finishes successfully: > > > > kolla-ansible -i ../multinode stop --yes-i-really-really-mean-it --limit control1 > > … > > control1 : ok=45 changed=22 unreachable=0 failed=0 skipped=105 rescued=0 ignored=0 > > > > After this my RMQ admin page stops responding. When I check RMQ on the new control0 and the existing control2, the container is still up but RMQ is not running: > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status > > Error: this command requires the 'rabbit' app to be running on the target node. Start it with 'rabbitmqctl start_app'. > > > > If I start it on control0 and control2, then the cluster seems normal and the admin page starts working again, and cluster status looks normal: > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status > > Cluster status of node rabbit@chrnc-void-testupgrade-control-0-replace ... > > [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', > > 'rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}]}, > > {running_nodes,['rabbit@chrnc-void-testupgrade-control-2', > > 'rabbit@chrnc-void-testupgrade-control-0-replace']}, > > {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, > > {partitions,[]}, > > {alarms,[{'rabbit@chrnc-void-testupgrade-control-2',[]}, > > {'rabbit@chrnc-void-testupgrade-control-0-replace',[]}]}] > > > > But my hypervisors are down: > > > > (openstack) [root@chrnc-void-testupgrade-build kolla-ansible]# ohll > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB | > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > | 3 | chrnc-void-testupgrade-compute-2.dev.chtrse.com | QEMU | 172.16.2.106 | down | 5 | 8 | 2560 | 30719 | > > | 6 | chrnc-void-testupgrade-compute-0.dev.chtrse.com | QEMU | 172.16.2.31 | down | 5 | 8 | 2560 | 30719 | > > | 9 | chrnc-void-testupgrade-compute-1.dev.chtrse.com | QEMU | 172.16.0.30 | down | 5 | 8 | 2560 | 30719 | > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > > > When I look at the nova-compute.log on a compute node, I see RMQ failures every 10 seconds: > > > > 172.16.2.31 compute0 > > 2021-03-30 03:07:54.893 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out > > 2021-03-30 03:07:55.905 7 INFO oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] Reconnected to AMQP server on 172.16.1.132:5672 via [amqp] client with port 56422. > > 2021-03-30 03:08:05.915 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out > > > > In the RMQ logs I see this every 10 seconds: > > > > 172.16.1.132 control2 > > [root@chrnc-void-testupgrade-control-2 ~]# tail -f /var/log/kolla/rabbitmq/rabbit\(a)chrnc-void-testupgrade-control-2.log |grep 172.16.2.31 > > 2021-03-30 03:07:54.895 [warning] <0.13247.35> closing AMQP connection <0.13247.35> (172.16.2.31:56420 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): > > client unexpectedly closed TCP connection > > 2021-03-30 03:07:55.901 [info] <0.15288.35> accepting AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) > > 2021-03-30 03:07:55.903 [info] <0.15288.35> Connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) has a client-provided name: nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e > > 2021-03-30 03:07:55.904 [info] <0.15288.35> connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e): user 'openstack' authenticated and granted access to vhost '/' > > 2021-03-30 03:08:05.916 [warning] <0.15288.35> closing AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): > > > > Why does RMQ fail when I shut down the 2nd controller after successfully replacing the first one? Hi Albert, Could you share the versions of RabbitMQ and erlang in both versions of the container? When initially testing this setup, I think we had 3.7.24 on both sides. Perhaps the CentOS 8 version has moved on sufficiently to become incompatible? Mark > > > > I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails. > > > > The contents of this e-mail message and > any attachments are intended solely for the > addressee(s) and may contain confidential > and/or legally privileged information. If you > are not the intended recipient of this message > or if this message has been addressed to you > in error, please immediately alert the sender > by reply e-mail and then delete this message > and any attachments. If you are not the > intended recipient, you are notified that > any use, dissemination, distribution, copying, > or storage of this message or any attachment > is strictly prohibited. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

1 0

[ironic] IPA does not want boot with UEFI
by Vuk.Gojnic＠telekom.de 01 Apr '21

01 Apr '21

Hello everybody, I am using Ironic standalone to provision the HPE Gen10+ node via iLO driver. Ironic version is 16.0.1. Server is configured with UEFI boot mode. Everything on Ironic side works fine. It creates ISO image, powers the server on and configures it to boot from it. Here is the what /var/log/ironic/ironic-conductor.log says: 2021-03-31 17:46:25.541 2618460 INFO ironic.conductor.task_manager [req-b6a19234-5f4b-4852-ab41-4102b9016bb0 - - - - -] Node ed25569f-c107-4fe0-95cd-74fcad9ab3f0 moved to provision state "cleaning" from state "manageable"; target provision state is "available" 2021-03-31 17:46:32.066 2618460 INFO ironic.drivers.modules.ilo.power [req-b6a19234-5f4b-4852-ab41-4102b9016bb0 - - - - -] The node ed25569f-c107-4fe0-95cd-74fcad9ab3f0 operation of 'power off' is completed in 4 seconds. 2021-03-31 17:46:32.088 2618460 INFO ironic.conductor.utils [req-b6a19234-5f4b-4852-ab41-4102b9016bb0 - - - - -] Successfully set node ed25569f-c107-4fe0-95cd-74fcad9ab3f0 power state to power off by power off. 2021-03-31 17:46:34.510 2618460 INFO ironic.drivers.modules.ilo.common [req-b6a19234-5f4b-4852-ab41-4102b9016bb0 - - - - -] Node ed25569f-c107-4fe0-95cd-74fcad9ab3f0 pending boot mode is uefi. 2021-03-31 17:46:37.248 2618460 INFO ironic.drivers.modules.ilo.common [req-b6a19234-5f4b-4852-ab41-4102b9016bb0 - - - - -] Set the node ed25569f-c107-4fe0-95cd-74fcad9ab3f0 to boot from URL https://10.23.137.234/tmp-images/ilo/boot-ed25569f-c107-4fe0-95cd-74fcad9ab… successfully. 2021-03-31 17:46:48.367 2618460 INFO ironic.drivers.modules.ilo.power [req-b6a19234-5f4b-4852-ab41-4102b9016bb0 - - - - -] The node ed25569f-c107-4fe0-95cd-74fcad9ab3f0 operation of 'power on' is completed in 8 seconds. 2021-03-31 17:46:48.388 2618460 INFO ironic.conductor.utils [req-b6a19234-5f4b-4852-ab41-4102b9016bb0 - - - - -] Successfully set node ed25569f-c107-4fe0-95cd-74fcad9ab3f0 power state to power on by rebooting. 2021-03-31 17:46:48.404 2618460 INFO ironic.conductor.task_manager [req-b6a19234-5f4b-4852-ab41-4102b9016bb0 - - - - -] Node ed25569f-c107-4fe0-95cd-74fcad9ab3f0 moved to provision state "clean wait" from state "cleaning"; target provision state is "available" The Grub2 starts and after I select the option "boot_partition", it starts booting and immediately freezes with following screen (just red static underscore): [cid:image002.jpg@01D726EE.A920A5C0] I have tried with pre-built IPA images (see below) as well as with custom IPA images made with Ubuntu 18.04 and 20.04 (built using ironic-python-agent-builder) but it is all the same. Does somebody have idea what is the problem with IPA and UEFI in this particular scenario? Output of "openstack baremetal node show" command: allocation_uuid: null automated_clean: null bios_interface: no-bios boot_interface: ilo-uefi-https chassis_uuid: null clean_step: {} conductor: 10.23.137.234 conductor_group: '' console_enabled: false console_interface: no-console created_at: '2021-03-21T13:54:25+00:00' deploy_interface: direct deploy_step: {} description: null driver: ilo5 driver_info: ilo_address: 10.23.137.137 ilo_bootloader: https://ironic-images/Images/esp.img ilo_deploy_kernel: https://ironic-images/Images/ipa-centos8-stable-victoria.kernel ilo_deploy_ramdisk: https://ironic-images/Images/ipa-centos8-stable-victoria.initramfs ilo_password: '******' ilo_username: Administrator snmp_auth_priv_password: '******' snmp_auth_prot_password: '******' snmp_auth_user: iloinspect driver_internal_info: agent_continue_if_ata_erase_failed: false agent_enable_ata_secure_erase: true agent_erase_devices_iterations: 1 agent_erase_devices_zeroize: true agent_erase_skip_read_only: false agent_secret_token: '******' agent_secret_token_pregenerated: true clean_steps: null disk_erasure_concurrency: 1 last_power_state_change: '2021-03-31T17:46:37.894667' extra: {} fault: clean failure inspect_interface: ilo inspection_finished_at: '2021-03-21T13:57:33+00:00' inspection_started_at: null instance_info: deploy_boot_mode: uefi instance_uuid: null last_error: null lessee: null maintenance: true maintenance_reason: management_interface: ilo5 name: null network_data: {} network_interface: noop owner: null power_interface: ilo power_state: power on properties: cpu_arch: x86 cpus: 64 local_gb: 2979 memory_mb: 262144 protected: false protected_reason: null provision_state: clean wait provision_updated_at: '2021-03-31T17:46:48+00:00' raid_config: {} raid_interface: no-raid rescue_interface: no-rescue reservation: null resource_class: null retired: false retired_reason: null storage_interface: noop target_power_state: null target_provision_state: available target_raid_config: {} traits: [] updated_at: '2021-03-31T17:46:48+00:00' uuid: ed25569f-c107-4fe0-95cd-74fcad9ab3f0 vendor_interface: no-vendor Many thanks! Vuk Gojnic Deutsche Telekom Technik GmbH Services & Plattforms (T-SP) Tribe Data Center Infrastructure (T-DCI) Super Squad Cloud Platforms Lifecycle (SSQ-CP) Vuk Gojnic Kubernetes Engine Squad Lead

1 0

Re: [glance][openstack-ansible] Snapshots disappear during saving
by Oliver Wenz 01 Apr '21

01 Apr '21

> So according to the issue, you get 503 while trying to reach > 10.0.3.212:6002/os-objects, which is swift_account_port. > Are there any logs specificly for swift-account? > > Also I guess some adjustments are required for swift as well for this > mechanism to work. > > Eventually I believe the original issue you saw might be related to this > doc: > https://docs.openstack.org/keystone/latest/admin/manage-services.html#confi… Hi Dmitriy, I tried setting 'service_token_roles_required = false' in glance-api.conf but the error is still there. I also checked the account server and I'm seeing lots of 404's: Apr 01 09:31:33 bc1bl12 account-server[694822]: 10.0.3.212 - - [01/Apr/2021:09:31:33 +0000] "GET /os-objects/22/.misplaced_objects" 404 - "GET http://localhost/v1/.misplaced_objects?format=json&marker=&end_marker=&pref…" "tx3794c20a6ef5476f9fcf6-00606592f5" "proxy-server 13882" 0.0007 "-" 694822 - Apr 01 09:32:03 bc1bl12 account-server[694816]: 10.0.3.212 - - [01/Apr/2021:09:32:03 +0000] "HEAD /os-objects/22/.misplaced_objects" 404 - "HEAD http://localhost/v1/.misplaced_objects?format=json" "tx5491e866bc6447d8917f9-0060659313" "proxy-server 13882" 0.0008 "-" 694816 - Apr 01 09:32:03 bc1bl12 account-server[694814]: 10.0.3.212 - - [01/Apr/2021:09:32:03 +0000] "GET /os-objects/22/.misplaced_objects" 404 - "GET http://localhost/v1/.misplaced_objects?format=json&marker=&end_marker=&pref…" "txc98a949e82c940d19656b-0060659313" "proxy-server 13882" 0.0007 "-" 694814 - Apr 01 09:32:33 bc1bl12 account-server[694817]: 10.0.3.212 - - [01/Apr/2021:09:32:33 +0000] "HEAD /os-objects/22/.misplaced_objects" 404 - "HEAD http://localhost/v1/.misplaced_objects?format=json" "txd9ec46414b8e4a98a2b94-0060659331" "proxy-server 13882" 0.0008 "-" 694817 - Apr 01 09:32:33 bc1bl12 account-server[694817]: 10.0.3.212 - - [01/Apr/2021:09:32:33 +0000] "GET /os-objects/22/.misplaced_objects" 404 - "GET http://localhost/v1/.misplaced_objects?format=json&marker=&end_marker=&pref…" "tx972d968e71e94eb683558-0060659331" "proxy-server 13882" 0.0011 "-" 694817 - Apr 01 09:32:43 bc1bl12 account-server[694823]: 10.0.3.212 - - [01/Apr/2021:09:32:43 +0000] "HEAD /os-objects/220/.expiring_objects" 404 - "HEAD http://localhost/v1/.expiring_objects?format=json" "tx4d02362089c4497982940-006065933b" "proxy-server 14171" 0.0008 "-" 694823 - Apr 01 09:33:03 bc1bl12 account-server[694822]: 10.0.3.212 - - [01/Apr/2021:09:33:03 +0000] "HEAD /os-objects/22/.misplaced_objects" 404 - "HEAD http://localhost/v1/.misplaced_objects?format=json" "tx440963ca3e7948ff872ab-006065934f" "proxy-server 13882" 0.0007 "-" 694822 - Apr 01 09:33:03 bc1bl12 account-server[694823]: 10.0.3.212 - - [01/Apr/2021:09:33:03 +0000] "GET /os-objects/22/.misplaced_objects" 404 - "GET http://localhost/v1/.misplaced_objects?format=json&marker=&end_marker=&pref…" "tx547157ac468e476395a0e-006065934f" "proxy-server 13882" 0.0006 "-" 694823 - Apr 01 09:33:33 bc1bl12 account-server[694814]: 10.0.3.212 - - [01/Apr/2021:09:33:33 +0000] "HEAD /os-objects/22/.misplaced_objects" 404 - "HEAD http://localhost/v1/.misplaced_objects?format=json" "txac5c0207e90a4606a7758-006065936d" "proxy-server 13882" 0.0007 "-" 694814 - Apr 01 09:33:33 bc1bl12 account-server[694820]: 10.0.3.212 - - [01/Apr/2021:09:33:33 +0000] "GET /os-objects/22/.misplaced_objects" 404 - "GET http://localhost/v1/.misplaced_objects?format=json&marker=&end_marker=&pref…" "tx0d3afe2f421647f2a117e-006065936d" "proxy-server 13882" 0.0006 "-" 694820 - Kind regards, Oliver

1 0

[cinder][nova][requirements] RFE requested for os-brick
by Brian Rosmaita 01 Apr '21

01 Apr '21

Hello Requirements Team, The Cinder team recently became aware of a potential data-loss bug [0] that has been fixed in os-brick master [1] and backported to os-brick stable/wallaby [2]. We've proposed a release of os-brick 4.4.0 from stable/wallaby [3] and are petitioning for an RFE to include 4.4.0 in the wallaby release. We have three jobs running tempest with os-brick source in master that have passed with [1]: os-brick-src-devstack-plugin-ceph [4], os-brick-src-tempest-lvm-lio-barbican [5],and os-brick-src-tempest-nfs [6]. The difference between os-brick master (at the time the tests were run) and stable/wallaby since the 4.3.0 tag is as follows: master: d4205bd 3 days ago iSCSI: Fix flushing after multipath cfg change (Gorka Eguileor) 0e63fe8 2 weeks ago Merge "RBD: catch read exceptions prior to modifying offset" (Zuul) 28545c7 4 months ago RBD: catch read exceptions prior to modifying offset (Jon Bernard) 99b2c60 2 weeks ago Merge "Dropping explicit unicode literal" (Zuul) 7cfdb76 6 weeks ago Dropping explicit unicode literal (tushargite96) 9afa1a0 3 weeks ago Add Python3 xena unit tests (OpenStack Release Bot) ab57392 3 weeks ago Update master for stable/wallaby (OpenStack Release Bot) 91a1cca 3 weeks ago (tag: 4.3.0) Merge "NVMeOF connector driver connection information compatibility fix" (Zuul) stable/wallaby: f86944b 3 days ago Add release note prelude for os-brick 4.4.0 (Brian Rosmaita) c70d70b 3 days ago iSCSI: Fix flushing after multipath cfg change (Gorka Eguileor) 6649b8d 3 weeks ago Update TOX_CONSTRAINTS_FILE for stable/wallaby (OpenStack Release Bot) f3f93dc 3 weeks ago Update .gitreview for stable/wallaby (OpenStack Release Bot) 91a1cca 3 weeks ago (tag: 4.3.0) Merge "NVMeOF connector driver connection information compatibility fix" (Zuul) This gives us very high confidence that the results of the tests run against master also apply to stable/wallaby at f86944b. Thank you for considering this request. (I've included Nova here because the bug occurs when the configuration option that enables multipath connections on a compute is changed while volumes are attached, so if this RFE is approved, nova might want to raise the minimum version of os-brick in wallaby to 4.4.0.) [0] https://launchpad.net/bugs/1921381 [1] https://review.opendev.org/c/openstack/os-brick/+/782992 [2] https://review.opendev.org/c/openstack/os-brick/+/783207 [3] https://review.opendev.org/c/openstack/releases/+/783641 [4] https://zuul.opendev.org/t/openstack/build/30a103668e4c4a8cb6f1ef907ef3edcb [5] https://zuul.opendev.org/t/openstack/build/bb11eef737d34c41bb4a52f8433850b0 [6] https://zuul.opendev.org/t/openstack/build/3ad3359ca712432d9ef4261d72c787fa

4 9

[neutron] Drivers team meeting 02.04.2021 cancelled
by Slawek Kaplonski 31 Mar '21

31 Mar '21

Hi, As tomorrow's Good Friday is public holiday in many countries, at least in Europe, and I will also be on PTO, let's cancel the drivers meeting. Have a great holidays and see You on the meeting next week. -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

Retiring the planet.openstack.org service
by Ian Wienand 31 Mar '21

31 Mar '21

Hello, We plan to retire the planet.openstack.org RSS aggregation service soon. The host is running an unsupported distribution, and we have not found any open source alternatives that seem to be currently maintained and deployable. On consideration of our limited infra resources, we feel it will be better to sunset this service at this time. It is likely that effort will be better spent getting information to more relevant channels in 2021, such as social media sites, etc. I have extracted an OPML file of the active blogs in [1]; most every feed reader can import this file. Thanks, -i [1] https://review.opendev.org/c/opendev/system-config/+/784191

1 0