April 2021 - openstack-discuss - lists.openstack.org

[Interop][Refstack] this Friday meeting
by Kanevsky, Arkady 09 Apr '21

09 Apr '21

Team, This Friday is Good Friday and some people have a day off. Should we cancel this week meeting? Please, respond so we can see if we will have quorum. Thanks, Arkady Arkady Kanevsky, Ph.D. SP Chief Technologist & DE Dell Technologies office of CTO Dell Inc. One Dell Way, MS PS2-91 Round Rock, TX 78682, USA Phone: 512 7204955

5 6

[neutron][all] Tempest jobs running on rocky and queens branches are broken
by Slawek Kaplonski 09 Apr '21

09 Apr '21

Hi, I noticed it mostly in the neutron jobs but it seems that it's true also for other projects for jobs which still runs on Ubuntu 16.04. I Neutron case those are all jobs on stable/rocky and stable/queens branches. Due to [1] those jobs will end up with POST_FAILURE. So please don't recheck Your patches if You have such errors until that bug will be fixed. I think that gmann has or is working on fix for that. [1] https://bugs.launchpad.net/devstack/+bug/1923042 -- Slawek Kaplonski Principal Software Engineer Red Hat

2 1

Re: [Release-job-failures] Release of openstack/eslint-config-openstack for ref refs/tags/4.1.0 failed
by Herve Beraud 09 Apr '21

09 Apr '21

FYI Looks similar to that story: - http://lists.openstack.org/pipermail/openstack-discuss/2021-March/021002.ht… - http://lists.openstack.org/pipermail/openstack-discuss/2021-March/021217.ht… I proposed a patch to move to nodejs10 all our projects that depend on nodejs: https://review.opendev.org/c/openstack/project-config/+/785353 When this patch will be merged I think that this job could be reenqueued. Le jeu. 8 avr. 2021 à 11:10, <zuul(a)openstack.org> a écrit : > Build failed. > > - release-openstack-javascript > https://zuul.opendev.org/t/openstack/build/4062ea0df4e74565b9f8b443e550c0fd > : RETRY_LIMIT in 3m 53s > - announce-release https://zuul.opendev.org/t/openstack/build/None : > SKIPPED > - openstack-upload-github-mirror > https://zuul.opendev.org/t/openstack/build/2e9b658e70f340818cafefa88c5044e2 > : SUCCESS in 41s > > _______________________________________________ > Release-job-failures mailing list > Release-job-failures(a)lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/release-job-failures > -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

2 3

[ironic] APAC-Europe SPUC time?
by Dmitry Tantsur 09 Apr '21

09 Apr '21

Hi folks! The initial SPUC datetime was for 10am UTC, which was 11am for us in central Europe, now is supposed to be 12pm. On one hand, I find it more convenient to have SPUC at 11am still, on the other - I have German classes at this time for a few months starting mid-April. What do you think? Should we keep it in UTC, i.e. 12pm for us in Europe? Will that work for you Jacob? Dmitry -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

3 2

[telemetry][cyborg][heat][monasca][tacker][keystone][release] Last minute RC to land fixes
by Herve Beraud 09 Apr '21

09 Apr '21

Hello teams listed above, We identified fixes and significant changes in your repos so we proposed last minute RC to allow you to release them before the final release. Your teams patches are available here: https://review.opendev.org/q/topic:%22wallaby-final-rc%22 Deadline is today, please validate them ASAP to have a chance to see these fixes released. Patches without response from PTLs/liaisons will be abandoned. After this point final release for RC projects will be started. Notice that RC changes should be on stable/wallaby and not on master, all projects are now branched so your master branches are now for Xena purpose. Thank you for your understanding. -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

2 1

[neutron] Neutron, nftables support and other fantastic beasts
by Rodolfo Alonso Hernandez 09 Apr '21

09 Apr '21

Hello Neutrinos: During Wallaby I've been working on enabling "nftables" support in Neutron. The goal was to use the new Netfilter framework replacing the legacy tools ("iptables", "ip6tables", "arptables" and "ebtables"). Because each namespace has its own Netfilter process, isolated from other namespaces, the migration process could be segmented in several tasks: dnat, fip, router, dhcp, metadata, Linux Bridge FW and OVS hybrid FW (I think I'm not missing anything here). When swapping to the new "nftables" framework, we can use the legacy API tools provided. Those tools provide a smooth transition to the new tooling (we found some differences that are now solved). That means we can keep the current code while using "nftables". Please, read [3] before reading the next paragraph, explaining the three "Netfilter" available framework alternatives. I started creating a "nft" (the "nftables" native binary) parser [1] to implement a NFtablesManager class, same as IPtablesManager. But soon I found that the transition to the new API is not that easy. This is not only a matter of creating the equivalent rule in the "nft" API but considering how those rules are handled in "nftables". Other problems found when using the new "nft" API: - The "--checksum-fill" command used in OVN metadata and DHCP namespace has no equivalent in "nft". That means old DHCP servers incorrectly calculating the packet checksum or DKDP environments won't work correctly. - "ipset" tool, used to group IP addresses and reduce the LB FW rule size, can be converted into a "map" [3]. The problem is this is only understood by the new API, not the "nftables" binaries using the legacy API. In a nutshell, what is the current status? We support (a) legacy tools and (b) "nftables" binaries with legacy API. This is the list of patches enabling the second option: - https://review.opendev.org/c/openstack/neutron/+/784913: this problem was affecting LB FW when "ipset" was disabled (merged). - https://review.opendev.org/c/openstack/neutron/+/785177: reorder the "ebtables" rules and prevent execution error 4 with empty chains. - https://review.opendev.org/c/openstack/neutron/+/785144: this patch, on top of the other two, creates two new neutron-tempest-plugin CI jobs, based on "linuxbridge" and "openvswitch-iptables_hybrid", to test the execution with the new binaries. - https://review.opendev.org/c/openstack/neutron/+/775413: this patch tests what is implemented in the previous one but testing those jobs in the "check" queue (it is a DNM patch just for testing). About the third option, to support the native "nft" API, I don't know if now we have the resources (time) and the need for that. This could be discussed again in the next PTG and in this mail too. Regards. [1]https://review.opendev.org/c/openstack/neutron/+/759874 [2] https://review.opendev.org/c/openstack/neutron/+/785137/3/doc/source/admin/… [3] https://review.opendev.org/c/openstack/neutron/+/775413/10/neutron/agent/li…

1 0

RE: [kolla] RHEL based container image (Tony Liu)
by CHALANSONNET Stéphane (Acoss) 09 Apr '21

09 Apr '21

Hello, You need an active subscription RHOSP for doing that , but Kolla was not supported by Redhat unfortunely ... Stéphane Chalansonnet -----Message d'origine----- De : openstack-discuss-request(a)lists.openstack.org <openstack-discuss-request(a)lists.openstack.org> Envoyé : jeudi 8 avril 2021 22:04 À : openstack-discuss(a)lists.openstack.org Objet : openstack-discuss Digest, Vol 30, Issue 56 Send openstack-discuss mailing list submissions to openstack-discuss(a)lists.openstack.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss or, via email, send a message with subject or body 'help' to openstack-discuss-request(a)lists.openstack.org You can reach the person managing the list at openstack-discuss-owner(a)lists.openstack.org When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..." Today's Topics: 1. Re: [Release-job-failures] Release of openstack/eslint-config-openstack for ref refs/tags/4.1.0 failed (Jeremy Stanley) 2. Re: [zun][kuryr][neutron] Missing vxlan ports in br-tun for Zun containers? (Jason Anderson) 3. [edge][cinder][manila][swift][tripleo] Storage at the edge discussions at the PTG (Ildiko Vancsa) 4. Re: [edge][cinder][manila][swift][tripleo] Storage at the edge discussions at the PTG (John Fulton) 5. [kolla] RHEL based container image (Tony Liu) ---------------------------------------------------------------------- Message: 1 Date: Thu, 8 Apr 2021 16:50:37 +0000 From: Jeremy Stanley <fungi(a)yuggoth.org> To: openstack-discuss(a)lists.openstack.org Subject: Re: [Release-job-failures] Release of openstack/eslint-config-openstack for ref refs/tags/4.1.0 failed Message-ID: <20210408165036.6tkcfwwix5ms3ig4(a)yuggoth.org> Content-Type: text/plain; charset="utf-8" On 2021-04-08 11:54:16 +0200 (+0200), Herve Beraud wrote: [...] > I proposed a patch to move to nodejs10 all our projects that depend on > nodejs: > > https://review.opendev.org/c/openstack/project-config/+/785353 > > When this patch will be merged I think that this job could be reenqueued. I reenqueued the tag, but release-openstack-javascript failed on a different problem. NPM complains that there's already a eslint-config-openstack 4.0.1 published which can't be overwritten, but the tag is for 4.1.0... someone should probably update the version parameter in eslint-config-openstack's package.json file, which means it'll need another release tagged anyway (4.1.1?). -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210408…> ------------------------------ Message: 2 Date: Thu, 8 Apr 2021 17:00:19 +0000 From: Jason Anderson <jasonanderson(a)uchicago.edu> To: openstack-discuss <openstack-discuss(a)lists.openstack.org> Subject: Re: [zun][kuryr][neutron] Missing vxlan ports in br-tun for Zun containers? Message-ID: <0A70CB1A-35AA-4782-8BF3-496080E47341(a)uchicago.edu> Content-Type: text/plain; charset="utf-8" As usual, “rubber ducking” the openstack-discuss list yielded fruit. It turns out that I didn’t have the l2population mechanism driver enabled. I thought this was optional for some reason. It looks like enabling this and restarting the neutorn-openvswitch-agent has fixed connectivity! /Jason > On Apr 8, 2021, at 11:20 AM, Jason Anderson <jasonanderson(a)uchicago.edu> wrote: > > Hello stackers, > > I’m interested in using zun to launch containers and assign floating IPs via neutron to those containers. I am deploying zun, kuryr-libnetwork, and neutron with kolla-ansible on the Train release. I’ve configured neutron with one physical network and I’d like to use a VXLAN overlay for tenant networks. > > What works: > - I can launch containers on a neutron tenant network, they start successfully, they get an IP and can reach each other if they’re co-located on a single host. > - I can create all my neutron networks, routers, subnets, without (obvious) errors. > - I can update security groups on the container and see the iptables rules updated appropriately. > - I can directly create Docker networks using the kuryr driver/type. > > What doesn’t work: > - I can’t see any vxlan ports on the br-tun OVS bridge > - I can’t access the exposed container ports from the control/network > node via the router netns > - Because of that, I can’t assign floating IPs because NAT effectively > won’t work to reach the containers > > The fact that there are no ports on br-tun is supicious, but I’m not sure how this is supposed to work. I don’t see anything weird in neutron-openvswitch-agent logs but those logs are quite noisy and I’m not sure what to look for. > > Has anybody deployed such a setup / are there limitations I should know about? > > Thank you! > > > Jason Anderson > > DevOps Lead, Chameleon > > --- > > Department of Computer Science, University of Chicago Mathematics and > Computer Science, Argonne National Laboratory > jasonanderson(a)uchicago.edu > ------------------------------ Message: 3 Date: Thu, 8 Apr 2021 19:18:54 +0200 From: Ildiko Vancsa <ildiko.vancsa(a)gmail.com> To: OpenStack Discuss <openstack-discuss(a)lists.openstack.org> Subject: [edge][cinder][manila][swift][tripleo] Storage at the edge discussions at the PTG Message-ID: <0A7B1EBD-0715-43ED-B388-A2011D437DD0(a)gmail.com> Content-Type: text/plain; charset=utf-8 Hi, I’m reaching out to draw your attention to the Edge Computing Group sessions on the PTG in less than two weeks. We are still formalizing our agenda, but we have storage identified as one of the topics that the working group would like to discuss. It would be great to have the session also as a continuation to earlier discussions that we had on previous PTGs with relevant OpenStack project contributors. We have a few cross-community sessions scheduled already, but we still have some flexibility in our agenda to schedule this topic so the most people who are interested in participating can join. Our available options are: * Monday (April 19) between 1400 UTC and 1500 UTC * Tuesday (April) between 1400 UTC and 1600 UTC __Please let me know if you or your project would like to participate and if you have a time slot difference from the above.__ Thanks and Best Regards, Ildikó (IRC ildikov on Freenode) ------------------------------ Message: 4 Date: Thu, 8 Apr 2021 13:39:45 -0400 From: John Fulton <johfulto(a)redhat.com> To: Ildiko Vancsa <ildiko.vancsa(a)gmail.com> Cc: OpenStack Discuss <openstack-discuss(a)lists.openstack.org> Subject: Re: [edge][cinder][manila][swift][tripleo] Storage at the edge discussions at the PTG Message-ID: <CAE66OMCQWZFB+7SM1amzRr1Vq3pOm=fEgtQTuHLh0=0PPXy-sQ(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" On Thu, Apr 8, 2021 at 1:21 PM Ildiko Vancsa <ildiko.vancsa(a)gmail.com> wrote: > Hi, > > I’m reaching out to draw your attention to the Edge Computing Group > sessions on the PTG in less than two weeks. > > We are still formalizing our agenda, but we have storage identified as > one of the topics that the working group would like to discuss. It > would be great to have the session also as a continuation to earlier > discussions that we had on previous PTGs with relevant OpenStack project contributors. > > We have a few cross-community sessions scheduled already, but we still > have some flexibility in our agenda to schedule this topic so the most > people who are interested in participating can join. Our available > options > are: > > * Monday (April 19) between 1400 UTC and 1500 UTC > * Tuesday (April) between 1400 UTC and 1600 UTC > I'm not available Monday but could join Tuesday. I'd be curious to hear what others are doing with Storage on the Edge and could share some info on how TripleO does it. John > > __Please let me know if you or your project would like to participate > and if you have a time slot difference from the above.__ > > Thanks and Best Regards, > Ildikó > (IRC ildikov on Freenode) > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210408…> ------------------------------ Message: 5 Date: Thu, 8 Apr 2021 20:04:04 +0000 From: Tony Liu <tonyliu0592(a)hotmail.com> To: "openstack-discuss(a)lists.openstack.org" <openstack-discuss(a)lists.openstack.org> Subject: [kolla] RHEL based container image Message-ID: <MWHPR08MB2382486467C9EE24247052B9BD749(a)MWHPR08MB2382.namprd08.prod.outlook.com> Content-Type: text/plain; charset="us-ascii" Hi, Given [1], RHEL based container is supported on RHEL 8 by Kolla. Where can I get RHEL based container images? I see CentOS and Ubuntu based images on docker hub, but can't find RHEL based images. [1] https://docs.openstack.org/kolla-ansible/ussuri/user/support-matrix.html Thanks! Tony ------------------------------ Subject: Digest Footer _______________________________________________ openstack-discuss mailing list openstack-discuss(a)lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss ------------------------------ End of openstack-discuss Digest, Vol 30, Issue 56 *************************************************

3 2

[zun][kuryr][neutron] Missing vxlan ports in br-tun for Zun containers?
by Jason Anderson 09 Apr '21

09 Apr '21

Hello stackers, I’m interested in using zun to launch containers and assign floating IPs via neutron to those containers. I am deploying zun, kuryr-libnetwork, and neutron with kolla-ansible on the Train release. I’ve configured neutron with one physical network and I’d like to use a VXLAN overlay for tenant networks. What works: - I can launch containers on a neutron tenant network, they start successfully, they get an IP and can reach each other if they’re co-located on a single host. - I can create all my neutron networks, routers, subnets, without (obvious) errors. - I can update security groups on the container and see the iptables rules updated appropriately. - I can directly create Docker networks using the kuryr driver/type. What doesn’t work: - I can’t see any vxlan ports on the br-tun OVS bridge - I can’t access the exposed container ports from the control/network node via the router netns - Because of that, I can’t assign floating IPs because NAT effectively won’t work to reach the containers The fact that there are no ports on br-tun is supicious, but I’m not sure how this is supposed to work. I don’t see anything weird in neutron-openvswitch-agent logs but those logs are quite noisy and I’m not sure what to look for. Has anybody deployed such a setup / are there limitations I should know about? Thank you! Jason Anderson DevOps Lead, Chameleon --- Department of Computer Science, University of Chicago Mathematics and Computer Science, Argonne National Laboratory jasonanderson(a)uchicago.edu

2 2

Re: [glance][openstack-ansible] Snapshots disappear during saving
by Oliver Wenz 09 Apr '21

09 Apr '21

Hi Dmitriy, > I'm wondering if you see also stack trace in keystone logs? Running 'journalctl' on the keystone container, I don't see any tracebacks. Or is there a specific service I should check? Kind regards, Oliver

2 1

Between openstack and offline IDC, a large intranet and a large second-tier network in the world improve the feasibility of data migration and disaster recovery.
by 赵陈洲 09 Apr '21

09 Apr '21

1 0

migration maybe a iaas service for openstack
by 赵陈洲 09 Apr '21

09 Apr '21

1 0

[Group-based-policy] not able to create the policies
by Mantesh Patil 09 Apr '21

09 Apr '21

Hi, I am deploying devstack Ussuri with GBP (stable/ussuri) on ubuntu 18.04 LTS(Updated the packages), after installation I am able to create the policies as mentioned in the wiki <https://wiki.openstack.org/wiki/GroupBasedPolicy/InstallDevstack>. But after creation, I am not able to list the policies. I am using the following command to list the policies and it is giving a warning "/usr/local/lib/python3.6/dist-packages/keystoneauth1/adapter.py:235: UserWarning: Using keystoneclient sessions has been deprecated. Please update your software to use keystoneauth1. warnings.warn('Using keystoneclient sessions has been deprecated. '" Command1: source admin-openrc.sh *Command2: gbp group-create web * and also getting the following error while creating the group [image: image.png] Command3: *gbp group-list -c name -c tenant_id -f value* Please give the information that how can I get a list of group policies using CLI and new authentication.

1 0

[all][elections][tc] TC Vacancy Special Election Voting Kickoff
by Jeremy Stanley 09 Apr '21

09 Apr '21

TC Vacancy Special Election Nomination period is now over. The four already elected TC members for this term are listed as candidates in the special election, but will not appear on any resulting poll as they have already been officially elected. Only new candidates who are not the four elected TC members for this term will appear on a subsequent poll for the TC vacancy special election. The poll for the TC Vacancy Special Election is now open and will remain open until Apr 15, 2021 23:45 UTC. We are selecting 1 additional TC member, please rank all candidates in your order of preference. You are eligible to vote if you are a Foundation individual member[1] that also has committed to one of the official project teams' deliverable repositories[2] over the Apr 24, 2020 00:00 UTC - Mar 08, 2021 00:00 UTC timeframe (Victoria to Wallaby) or if you are one of the extra-atcs.[3] Please note that in order to confirm contributors are foundation members, the preferred address in Gerrit must also be included in the addresses for the corresponding member profile. What to do if you don't see the email and have a commit in at least one of the official deliverables[2]: * check the trash or spam folder of your gerrit Preferred Email address[4], in case it went into trash or spam * wait a bit and check again, in case your email server is a bit slow * find the sha of at least one commit from an official deliverable repo[2] and email the election officials[1]. If we can confirm that you are entitled to vote, we will add you to the voters list and you will be emailed a ballot. Our democratic process is important to the health of OpenStack, please exercise your right to vote. Candidate statements/platforms can be found linked to Candidate names[6]. Happy voting! Thank you, [1] https://www.openstack.org/community/members/ [2] https://opendev.org/openstack/governance/src/commit/892c4f3a851428cf41bab57… [3] Look for the extra-atcs element in [2] [4] Sign into review.openstack.org: Go to Settings > Contact Information. Look at the email listed as your preferred email. That is where the ballot has been sent. [5] https://governance.openstack.org/election/#election-officials [6] https://governance.openstack.org/election/#xena-tc-candidates -- Jeremy Stanley on behalf of the OpenStack Technical Elections Officials

1 0

[first contact] [SIG] PTG Planning!
by Kendall Nelson 08 Apr '21

08 Apr '21

Hello! I didn't think we would need a ton of time and I tried to pick a time to balance everyone's timezones so we have an hour at 22 UTC on Monday in the Austin room. We have an etherpad that was autogenerated: https://etherpad.opendev.org/p/apr2021-ptg-first-contact Please add topics if you have them and your name if you plan to join us! -Kendall Nelson (diablo_rojo)

1 0

[puppet] Proposing Alan Bishop (abishop) for puppet-cinder core and puppet-glance core
by Takashi Kajinami 08 Apr '21

08 Apr '21

Hello, I'd like to propose Alan Bishop (abishop) for the core team of puppet-cinder and puppet-glance. Alan has been actively involved in these 2 modules for a few years and has implemented some nice features like multiple backend support in glance, cinder s3 backup driver and etc, which expanded adoption of puppet-openstack. He has also provided good reviews on patches for these 2 repos based on his understanding about our code, puppet and serverspec. He is an active contributor to cinder and has deep knowledge about it. In addition He is also a core review in TripleO, which consumes our puppet modules, and mainly covers storage components like cinder and glance, so he is familiar with the way how these two components are deployed and configured. I believe adding him to our board helps us improve our review of these two modules. I'll wait for one week to hear any feedback from other core reviewers. Thank you, Takashi

6 6

[victoria][magnum]fedora-atomic-27 image
by Luke Camilleri 08 Apr '21

08 Apr '21

We have insatlled magnum following the installation guide here https://docs.openstack.org/magnum/victoria/install/install-rdo.html and the process was quite smooth but we have been having some issues with the deployment of the clusters. The image being used as per the documentation is https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-27-2… Our first issue was that podman was being used even if we specified the use_podman=false (since the image above did not include podman) but this was resulting in a timeout and the cluster would fail to deploy. We have then installed podman in the image and the cluster progressed a bit further /+ echo 'WARNING Attempt 60: Trying to install kubectl. Sleeping 5s'// //+ sleep 5s// //+ ssh -F /srv/magnum/.ssh/config root@localhost '/usr/bin/podman run --entrypoint /bin/bash --name install-kubectl --net host --privileged --rm --user root --volume /srv/magnum/bin:/host/srv/magnum/bin k8s.gcr.io/hyperkube:v1.15.7 -c '\''cp /usr/local/bin/kubectl /host/srv/magnum/bin/kubectl'\'''// //bash: /usr/bin/podman: No such file or directory// //ERROR Unable to install kubectl. Abort.// //+ i=61// //+ '[' 61 -gt 60 ']'// //+ echo 'ERROR Unable to install kubectl. Abort.'// //+ exit 1/ The cluster is now failing here at "kube_cluster_deploy" and when checking the logs on the master node we noticed the following in the log files: /Starting to run kube-apiserver-to-kubelet-role// //Waiting for Kubernetes API...// //+ echo 'Waiting for Kubernetes API...'// //++ curl --silent http://127.0.0.1:8080/healthz// //+ '[' ok = '' ']'// //+ sleep 5/ This is because the kubernetes API server is not installed either. I have noticed some scripts that should handle the installation but I would like to know if anyone here has had similar issues with a clean Victoria installation. Also should we have to install any packages in the fedora atomic image file or should the installation requirements be part of the stack? Thanks in advance for any asistance

4 9

[baremetal-sig][ironic] Tue Apr 13, 2021, 2pm UTC: Secure RBAC in Ironic
by Arne Wiebalck 08 Apr '21

08 Apr '21

Dear all, The Bare Metal SIG will meet next week Tue Apr 13, 2021, at 2pm UTC on zoom. There will be two main points on the agenda: - A "topic-of-the-day" presentation by Julia Kreger (TheJulia) on 'Secure RBAC in Ironic' and a - PTG pre-discussion on a potential integration of Ironic with Kea DHCP. As usual, all details on https://etherpad.opendev.org/p/bare-metal-sig Everyone is welcome! Cheers, Arne

1 0

[cinder] final reviews for RC-2
by Brian Rosmaita 08 Apr '21

08 Apr '21

We have 3 patches that need review/revision/approval as soon as possible before we release RC-2 tomorrow (Thursday 8 April). All 3 are updates to the release notes: Release note for mTLS support cinder->glance - https://review.opendev.org/c/openstack/cinder/+/783964 Release note about the cgroups v1 situation - https://review.opendev.org/c/openstack/cinder/+/784179 Add known issue note about RBD encrypted volumes - https://review.opendev.org/c/openstack/cinder/+/785235 Please review and leave comments as soon as you can.

2 1

[neutron] Drivers meeting agenda - 09.04.2021
by Slawek Kaplonski 08 Apr '21

08 Apr '21

Hi, Agenda for the tomorrow's drivers meeting is at [1]. We have 2 RFEs to discuss: https://bugs.launchpad.net/neutron/+bug/1922237 - [RFE][QoS] Add minimum guaranteed packet rate QoS rule https://bugs.launchpad.net/neutron/+bug/1921461 - [RFE] Enhancement to Neutron BGPaaS to directly support Neutron Routers & bgp-peering from such routers over internal & external Neutron Networks [1] https://wiki.openstack.org/wiki/Meetings/NeutronDrivers#Agenda -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[Keystone] Managing keystone tokens in high availability environment
by Taha Adel 08 Apr '21

08 Apr '21

Hello Engineers and Developers, I'm currently deploying a three-nodes openstack controller cluster, controller-01, controller-02, anc controller-03. I have installed the keystone service on the three controllers and generated fernet keys on one node and distributed the keys to the other nodes of the cluster. Hence, I have configured an HAProxy in front of them that would distribute the incoming requests over them. The issue is, when I try to access the keystone endpoint from using the VIP of the loadbalancer, the service works ONLY on the node that I have generated the keys on, and it doesn't work on the nodes that got the keys by distribution. the error message I have got is *"INTERNAL SERVER ERROR (500)"* In other words, the node that had* keystone-manage fernet_setup *command ran on it, it can run the service properly, but the others can't. Is the way of replicating the key incorrect? is there any other way? Thanks in advance

2 1

[all][tc] Technical Committee next weekly meeting on April 8th at 1500 UTC.
by Ghanshyam Mann 07 Apr '21

07 Apr '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for April 8th at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, April 7th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

[all] vPTG April 2021 Registration & Schedule
by Kendall Nelson 07 Apr '21

07 Apr '21

Hello everyone! The April 2021 Project Teams Gathering is right around the corner! The official schedule has now been posted on the PTG website [1], the PTGbot will be up to date by the end of the week [2], and we have also attached it to this email. Please double check your rooms! We did a little consolidation and shifting while maintaining the times you signed up for. Friendly reminder, if you have not already registered, please do so [3]. It is important that we get everyone to register for the event as this is how we will contact you about tooling information/passwords and other event details. Please let us know if you have any questions! Cheers, - The Kendalls (diablo_rojo & wendallkaters) [1] PTG Website www.openstack.org/ptg [2] PTGbot: http://ptg.openstack.org/ptg.html [3] PTG Registration: https://april2021-ptg.eventbrite.com

1 0

[tripleo][DIB] Second core review/w+ requested on DIB patch to unblock tripleo promotions
by Ronelle Landy 07 Apr '21

07 Apr '21

Hi diskimage-builder cores, OVB jobs have been failing on centos-8 based releases promotion jobs since the move to container-tools:3.0. https://review.opendev.org/c/openstack/diskimage-builder/+/785138 - "Make DIB_DNF_MODULE_STREAMS part of yum element" and https://review.opendev.org/c/openstack/tripleo-ci/+/785087 - "Use dib_dnf_module_streams for enabling modules" patches were added to address the failures. These patches were tested in https://review.rdoproject.org/r/c/testproject/+/33138 - see passing OVB job https://review.rdoproject.org/zuul/build/6582fb8afa2a44c6a806bb2545a9fadf. Thanks to Carlos for reviewing the DIB patch. We are looking for a second core review and w+ on this patch to clear the promotion lines. Thank you, tripleo CI ruck/rovers

1 0

[ptg] Secure RBAC and Policy Xena PTG sessoins
by Lance Bragstad 07 Apr '21

07 Apr '21

Hey all, Several projects are working through RBAC overhauls and naturally sessions are cropping up for the PTG. I tried bouncing around to various policy sessions during the Wallaby PTG, but I didn't plan things out very well. As a result, I missed sessions, had duplicate conversations with multiple groups, and ended up being more reactive than I'd like. To prevent that, Ghanshyam and I have condensed all the policy/RBAC sessions we know about in a single etherpad [0]. I know most projects are still firming up their schedules, but I've written down the session times that we know of and organized them chronologically. My hope is that this will help us group similar discussions and reach broader consensus on topics easier and quicker. For example, keystone and nova have a cross-project session on Thursday to discuss how nova should handle consuming system-scoped tokens for project-specific operations. This topic certainly isn't exclusive to nova. It'll impact just about every other service and approaching it consistently will be huge for end users and operators. Another good example of this would be the glance refactor to integrate system-scope support we're going to talk about on Wednesday (cinder and barbican are potentially facing very similar refactors). Each session in the etherpad [0] has topics, so if a topic sounds relevant to your service, please feel free to drop into those discussions. A rough outline is that: - Monday we're going to focus on QA and general policy problems (e.g., converting tempest to use system-scope, the JSON->YAML community goal, overall status from Wallaby, etc) - Tuesday we're going to find ways to adopt system-scope in cinder - Wednesday we're going to work through system-scope adoption, the meta definitions API, and test coverage in glance - Thursday we're going to discuss what the experience should be like for operators using system-scoped tokens to do project-specific operations with nova (e.g., rebooting instances) I'm contemplating hosting a 30 minute recap session on Friday that attempts to summarize everything from the week regarding RBAC discussions. If that sounds useful, I'll ask Kristi if I can use one of the keystone sessions for that recap. I know, this feels like a lot of focus for one thing and I appreciate everyone's help working through this stuff. But, I'm hopeful that better organization throughout the PTG week will result in less confusion about what we plan to do in Xena with RBAC so we can deliver something useful to users and operators. Thanks, Lance [0] https://etherpad.opendev.org/p/policy-popup-xena-ptg

2 2

[largescale-sig] Next meeting: April 7, 15utc
by Thierry Carrez 07 Apr '21

07 Apr '21

Hi everyone, Our next Large Scale SIG meeting will be this Wednesday in #openstack-meeting-3 on IRC, at 15UTC. You can doublecheck how it translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20210407T15 A number of topics have already been added to the agenda, including discussing our next video meetings and PTG participation. Feel free to add other topics to our agenda at: https://etherpad.openstack.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

Re: [Release-job-failures] Pre-release of openstack/neutron for ref refs/tags/18.0.0.0rc2 failed
by Thierry Carrez 07 Apr '21

07 Apr '21

zuul(a)openstack.org wrote: > Build failed. > > - openstack-upload-github-mirror https://zuul.opendev.org/t/openstack/build/4b31dfd25e244e38a346e097c993c519 : SUCCESS in 54s > - release-openstack-python https://zuul.opendev.org/t/openstack/build/309d80630c8e4086b52a0d83062cd431 : SUCCESS in 4m 43s > - announce-release https://zuul.opendev.org/t/openstack/build/4bde576261db4943a1dd6610aa52f6be : POST_FAILURE in 8m 33s > - propose-update-constraints https://zuul.opendev.org/t/openstack/build/18de7b1f60044445a06db64da42bbcc3 : SUCCESS in 5m 20s We are missing logs, but it looks like the job actually succeeded at announcing the release: http://lists.openstack.org/pipermail/release-announce/2021-April/011022.html So this can be safely ignored. -- Thierry Carrez (ttx)

2 1

[neutron][nova] Port binding fails when creating an instance
by Maxime d'Estienne 07 Apr '21

07 Apr '21

Hello, I spent a lot of time troubleshooting my issue, which I described here : https://serverfault.com/questions/1058969/cannot-create-an-instance-due-to-… To summarize, when I want to create an instance, binding fails on compute node, the dhcp agent seems to give an ip to the VM but I have an error. I don't know where to dig, besides what I have done. Thanks a lot for your help !

2 5

[Neutron] How to provide internet access to tier 2 instance
by Bernd Bausch 07 Apr '21

07 Apr '21

I have a pretty standard single-server Victoria Devstack, where I created this network topology: public private backend | | | | /-------\ |-- I1 |- I2 |--|Router1|--| | | \-------/ | | | | /-------\ | | |--|Router2|--| | | \-------/ | | | | I1 and I2 are instances. My question: Is it possible to give I2 access to the external world to install software and download files? I don't need access **to** I2 **from** the external world. My unsuccessful attempt: After adding a static default route via Router1 to Router2, I can ping the internet from Router2's namespace, but not from I2. My guess is that Router1 ignores traffic from networks that are not attached to it. I don't have enough experience to understand the netfilter rules in Router1's namespace, and in any case, rather than tweaking them I need a supported method to give I2 internet access, or the confirmation that it is not possible. Thanks much for any insights and suggestions. Bernd

2 1

[cinder] Bug deputy report for week of 2021-04-07
by Sofia Enriquez 07 Apr '21

07 Apr '21

Hello, This is a bug report from 2021-03-31 to 2021-04-07. You're welcome to join the next Cinder Bug Meeting later today. Weekly on Wednesday at 1500 UTC in #openstack-cinder Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Critical:- High: - Medium: - https://bugs.launchpad.net/cinder/+bug/1922408 `` Create volume from snapshot will lose encrypted head when source volume is encrypted in RBD''. Assigned to haixin. - https://bugs.launchpad.net/cinder/+bug/1922013 '' IBM SVF driver: GMCV Add vols to a group fails even if rcrel and rccg are in the same state". Unassigned. Low: - https://bugs.launchpad.net/cinder/+bug/1922255 "Dell PowerVault PVMEISCSIDriver driver cannot manage volumes". Unassigned. - https://bugs.launchpad.net/python-cinderclient/+bug/1922749 "Top-level client doesn't support auth v3". Unassigned. Undecided: - https://bugs.launchpad.net/devstack-plugin-ceph/+bug/1921897 "fast-diff in default feature set breaks 'rbd map'". Assigned to Eric Harney. Regards, Sofi -- L. Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[keystone]improvments in mapping models/support for JWT tokens
by pregusia 07 Apr '21

07 Apr '21

Hello members! Please direct your attention to some keystone modyfications - to be more precisly two of them: (1) extension to mapping engine in order to support multiple projects and assigning project by id (2) extension to authorization mechanisms - add JWT token support Ad (1): Currently mapping engine can map multiple projects for user (as stated in https://docs.openstack.org/keystone/pike/advanced-topics/federation/mapping…) but this support lacks from (a) dynamic mapping projects from assertion (for ex. projects ids are in assertion) and (b) mapping engine cannot map to list from assertion (if in asertion we have "some_field": [ id1, id2, ..., idN ] - then using this field by substitution - eg "{1}" wont map this to whole needed structure of projects in mapping). In patch I extend mapping schema by adding new field "projects_spec" of type string. This field contains string formated like project1ID:role1:role2:..:roleN,project2ID:role1:...:roleN,... and this is mapped into proper structure { "projects": [ { "id": .., "roles": [ ... ] }, ... ] } this allows the identity provider to supply permission-like informations about what projects user should have access to. The implementation is not ideal, and some work shoud be do to make it more user-friendly (for ex. configuration options for auto-creating projects by ID, and options for allowing user to log-in if project-by-id not exists etc) This is only proof of concept and question if this direction is proper according to keystone development. Ad (2) This patch adds new authorization protocol - jwt_token. It allows to access endpoint '/v3/OS-FEDERATION/identity_providers/{IDP_NAME}/protocols/jwt_token/auth' (and obtain keystone auth token) using JWT token in Authorization header field. Header value is checked against proper formating and JWT token is extracted from it. Then, token is validated against public key (RS256/RS512 algorithms only supported), against issuer and expiration date and others. Then, if this end with success, payload from token is supplied to mapping engine, when new user (and according to (1), it's permission to projects for ex.) is created and given access to proper projects/roles. Happy reviewing! pregusia

2 2

[cinder] Review of tiny patch to add Ceph RBD fast-diff to cinder-backup
by Christian Rohmann 07 Apr '21

07 Apr '21

Hey everyone, I wrote a tiny patch to add the Ceph RDB feature of fast-diff to backups created by cinder-backup: * https://review.opendev.org/c/openstack/cinder/+/766856/ Could someone please take a peek and let me know of this is sufficient to be merged? Thanks and with kind regards, Christian

3 12

[victoria]oslo_privsep.daemon.FailedToDropPrivileges
by 朝阳未烈 07 Apr '21

07 Apr '21

Hi, everyone: I tried to build an instance on the compute node but failed. I am sure that every necessary connection has been built. And I found the same error information on the controller node and the compute node , in /var/log/neutron/linuxbride-agent.log That is information: INFO neutron.common.config [-] Logging enabled! 2021-04-07 11:30:52.866 2182 INFO neutron.common.config [-] /usr/bin/neutron-linuxbridge-agent version 17.1.0 2021-04-07 11:30:52.867 2182 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Interface mappings: {'provider': 'ens160'} 2021-04-07 11:30:52.867 2182 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Bridge mappings: {} 2021-04-07 11:30:52.868 2182 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/neutron/neutron-dist.conf', '--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/linuxbridge_agent.ini', '--config-dir', '/etc/neutron/conf.d/neutron-linuxbridge-agent', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmpm5d0ytiv/privsep.sock'] 2021-04-07 11:30:53.346 2182 CRITICAL oslo.privsep.daemon [-] privsep helper command exited non-zero (1) 2021-04-07 11:30:53.346 2182 CRITICAL neutron [-] Unhandled error: oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1) 2021-04-07 11:30:53.346 2182 ERROR neutron Traceback (most recent call last): 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/bin/neutron-linuxbridge-agent", line 10, in <module> 2021-04-07 11:30:53.346 2182 ERROR neutron     sys.exit(main()) 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/neutron/cmd/eventlet/plugins/linuxbridge_neutron_agent.py", line 28, in main 2021-04-07 11:30:53.346 2182 ERROR neutron     agent_main.main() 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 1052, in main 2021-04-07 11:30:53.346 2182 ERROR neutron     manager = LinuxBridgeManager(bridge_mappings, interface_mappings) 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 79, in __init__ 2021-04-07 11:30:53.346 2182 ERROR neutron     self.validate_interface_mappings() 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 94, in validate_interface_mappings 2021-04-07 11:30:53.346 2182 ERROR neutron     if not ip_lib.device_exists(interface): 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 748, in device_exists 2021-04-07 11:30:53.346 2182 ERROR neutron     return IPDevice(device_name, namespace=namespace).exists() 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 328, in exists 2021-04-07 11:30:53.346 2182 ERROR neutron     return privileged.interface_exists(self.name, self.namespace) 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 246, in _wrap 2021-04-07 11:30:53.346 2182 ERROR neutron     self.start() 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 258, in start 2021-04-07 11:30:53.346 2182 ERROR neutron     channel = daemon.RootwrapClientChannel(context=self) 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 367, in __init__ 2021-04-07 11:30:53.346 2182 ERROR neutron     raise FailedToDropPrivileges(msg) 2021-04-07 11:30:53.346 2182 ERROR neutron oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1) 2021-04-07 11:30:53.346 2182 ERROR neutron     And it is the configuration in /etc/sudoer.d/neutron below:   Defaults:neutron !requiretty neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf * neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf     I googled for the solution but they didn’t matter. How can I solve this problem? Thanks for your advicement!

2 1

[docs] Project guides in PDF format
by Peter Matulis 07 Apr '21

07 Apr '21

Hello, I understand that there was a push [1] to make PDFs available for download from the published pages on docs.openstack.org for the various guides (Sphinx projects). Is there any steam left in this initiative? The next best thing is to have something at the repository level that people can work with. Currently, there are a limited number of guides that can be converted to PDF from the openstack-manuals repository. What is the suggested implementation for individual projects? Thanks, Peter Matulis [1]: https://etherpad.opendev.org/p/train-pdf-support-goal

5 11

Re: [glance][openstack-ansible] Snapshots disappear during saving
by Oliver Wenz 07 Apr '21

07 Apr '21

Following the suggestion from https://docs.openstack.org/swift/latest/overview_auth.html I set 'log_level=debug' for authtoken. Now I'm seeing more errors in the glance-api logs: Apr 06 15:07:40 infra1-glance-container-99614ac2 glance-wsgi-api[1837]: 2021-04-06 15:07:38.197 1837 WARNING keystonemiddleware.auth_token [req-ad8d0db9-b630-4237-9fff-d7ff282155d2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Identity response: {"error": {"code": 500, "title": "Internal Server Error", "message": "An unexpected error prevented the server from fulfilling your request."}}: keystoneauth1.exceptions.http.InternalServerError: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-55767c13-a450-4f81-90ab-5a7af6b3f672) Apr 06 15:07:46 infra1-glance-container-99614ac2 uwsgi[1822]: DAMN ! worker 13 (pid: 1836) died, killed by signal 9 :( trying respawn ... Here's the full log: http://paste.openstack.org/show/804208/ The options you suggested @Dmitriy are still there in user_variables.yml: glance_glance_api_conf_overrides: keystone_authtoken: service_token_roles_required: True service_token_roles: service Kind regards, Oliver

2 1

[dev][nova] Problem about vm migration compatibility check
by Guoyi Tu 06 Apr '21

06 Apr '21

hi there, In my test environment, i created a vm and configured the cpu with host-model, when I migrate the vm to another host with the same cpu, it failed the migration compatibility check which complains the cpu definition of domain is incompatible with target host cpu. As we know, when the domain configured as above starts, the host-model cpu definition will automatically converted to custom cpu model and with some addtional features that the KVM supported, these addtional features may contains features that the host doesn't support. In the code, the compatibility of the target host is check by calling compareCPU()(libvirt API). The compareCPU() can only recongnize the features probed by cpuid instruction on the host, but it may not recognize the features of cpu definition of domain xml (virsh dumpxml domainname) when the domain running. So the compatibility check will fail when KVM support one or more features which is considerd as disabled by the cpuid instuction. I think we should call compareHypervisorCPU() or something like that (supported by libvirt since v4.4.0) instead of compareCPU() to check the migration compatibility. My test environment is as follow: host cpu: Cascadelake libvirt-6.9 qemu-5.0 host-model cpu: <mode name='host-model' supported='yes'> <model fallback='forbid'>Cascadelake-Server</model> <vendor>Intel</vendor> <feature policy='require' name='ss'/> <feature policy='require' name='hypervisor'/> <feature policy='require' name='tsc_adjust'/> <feature policy='require' name='umip'/> <feature policy='require' name='pku'/> <feature policy='require' name='md-clear'/> <feature policy='require' name='stibp'/> <feature policy='require' name='arch-capabilities'/> <feature policy='require' name='xsaves'/> <feature policy='require' name='invtsc'/> <feature policy='require' name='rdctl-no'/> <feature policy='require' name='ibrs-all'/> <feature policy='require' name='skip-l1dfl-vmentry'/> <feature policy='require' name='mds-no'/> <feature policy='require' name='pschange-mc-no'/> <feature policy='disable' name='hle'/> <feature policy='disable' name='rtm'/> </mode> The hypervisor, umip, pschange-mc-no features block the compatibility check -- Best Regards, Guoyi Tu

2 1

How to modify staging-ovirt parameters
by Gianluca Cecchi 06 Apr '21

06 Apr '21

Hello, I have a test lab with Queens on CentOS 7 deployed with TripleO, that I use sometimes to mimic an OSP 13 environment. The Openstack nodes are oVirt VMS. I moved these VMs from one oVirt infrastructure to another one. The Openstack environment starts and operates without problems (3 controllers, 2 computes and 3 cephs), but I don't know how to modify the staging-ovirt parameters. At oVirt side I re-created as before a user ostackpm@internal with sufficient power and I also set the same password. But the ip address of the engine cannot be the same. When I deployed the environment I put all the parameters inside the instackenv.json file to do introspection and deploy. Where are they put on undercloud? Is there a way to update them? Perhaps some database entries or other kind of repos where I can update the ip? Thanks in advance, Gianluca

2 2

Unable to retrieve ceph docker image for Queens
by Gianluca Cecchi 06 Apr '21

06 Apr '21

Hello, I'm trying to extend an old small test lab Queens environment with dedicated Ceph Storage nodes. I would like to add some disks to the storage nodes and so I customize my original templates/ceph-config.yaml adding them under devices section: parameter_defaults: CephAnsibleDisksConfig: osd_scenario: lvm osd_objectstore: bluestore devices: - /dev/sda - /dev/sdb - /dev/sdc . . . When I run the overcloud deploy I get this in ceph-install-workflow.log 2021-04-06 21:24:57,955 p=32618 u=mistral | TASK [ceph-docker-common : pulling docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64 image] *** 2021-04-06 21:24:57,955 p=32618 u=mistral | Tuesday 06 April 2021 21:24:57 +0200 (0:00:00.153) 0:06:17.978 ********* 2021-04-06 21:25:13,206 p=32618 u=mistral | FAILED - RETRYING: pulling docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64 image (3 retries left). . . . 2021-04-06 21:26:03,715 p=32618 u=mistral | FAILED - RETRYING: pulling docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64 image (1 retries left). 2021-04-06 21:26:28,839 p=32618 u=mistral | fatal: [172.23.0.232]: FAILED! => {"attempts": 3, "changed": false, "cmd": ["timeout", "300s", "docker", "pull", "docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64"], "delta": "0:00:15.022274", "end": "2021-04-06 21:25:48.701980", "msg": "non-zero return code", "rc": 1, "start": "2021-04-06 21:25:33.679706", "stderr": "Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)", "stderr_lines": ["Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"], "stdout": "Trying to pull repository docker.io/ceph/daemon ... ", "stdout_lines": ["Trying to pull repository docker.io/ceph/daemon ... "]} So it seems the docker images are not there any more. My existing ceph nodes are using that image: [heat-admin@ostack-ceph0 ~]$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7ad4aeb3f4cb docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64 "/entrypoint.sh" 3 weeks ago Up 3 weeks ceph-osd-0 0dc9a9889283 docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64 "/entrypoint.sh" 3 weeks ago Up 3 weeks ceph-osd-3 8d543016cce7 docker.io/tripleoqueens/centos-binary-cron:current-tripleo "dumb-init --singl..." 11 months ago Up 3 weeks logrotate_crond [heat-admin@ostack-ceph0 ~]$ Any way or suggestion to manage this? Thanks, Gianluca

1 0

[qa][gate][stable] stable stein|train py2 devstack jobs are broken
by Ghanshyam Mann 06 Apr '21

06 Apr '21

Hello Everyone, During fixing the grenade issue on stable/train[1], there is another issue that came up for py2 devstack based jobs. I have logged the bug on devstack side as of now - https://bugs.launchpad.net/devstack/+bug/1922736 Let's wait for this to fix before you recheck on failing patches. [1] http://lists.openstack.org/pipermail/openstack-discuss/2021-April/021597.ht… -gmann

3 2

[all] Octavia LoadBalancer Error
by Adekunbi Adewojo 06 Apr '21

06 Apr '21

Hi there, I recently deployed a load balancer on our openstack private cloud. I used this manual - https://docs.openstack.org/octavia/latest/user/guides/basic-cookbook.html to create the load balancer. However, after creating and trying to access it, it returns an error message saying "No server is available to handle this request". Also on the dashboard, "Operating status" shows offline but "provisioning status" shows active. I have two web applications as members of the load balancer and I can individually access those web applications. Could someone please point me in the right direction? Thanks.

3 3

all Octavia LoadBalancer
by Adekunbi Adewojo 06 Apr '21

06 Apr '21

Hi there, I recently deployed a load balancer on our openstack private cloud. I used this manual - https://docs.openstack.org/octavia/latest/user/guides/basic-cookbook.html to create the load balancer. However, after creating and trying to access it, it returns an error message saying "No server is available to handle this request". Also on the dashboard, "Operating status" shows offline but "provisioning status" shows active. I have two web applications as members of the load balancer and I can individually access those web applications. Could someone please point me in the right direction? Thanks.

3 3

[tripleo] master and victoria promotions delayed
by Wesley Hayutin 06 Apr '21

06 Apr '21

0/ Hey just want to communicate that promotions for master and victoria are being delayed due to promotion-blockers. If you have a merged patch in a repo that is not gated by tripleo jobs you will NOT be able to use that patch until we promote. http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?org… Please review the following bugs to better understand what is blocking. https://bugs.launchpad.net/tripleo/+bugs?field.tag=promotion-blocker&orderb… I'll note, any tempest test results logged in bugs are most likely already skipped and not blocking upstream via https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/… Your focus should be on deployment failures, not tempest failures at this time. Thank you!

1 0

[qa][hacking] Proposing new core reviewers
by Martin Kopec 06 Apr '21

06 Apr '21

Hi all, I'd like to propose Sorin Sbarnea (IRC: zbr) and Radosław Piliszek (IRC: yoctozepto) to hacking core. They both are doing a great upstream work among multiple different projects and volunteered to help us with maintenance of hacking project as well. You can vote/feedback in this email thread. If no objection by 6th of April, we will add them to the list. Regards, -- Martin Kopec

3 3

[neutron] bug deputy report (3/29-4/4)
by Akihiro Motoki 06 Apr '21

06 Apr '21

Hi, I was a bug deputy last week and here is a report. Last week is relatively quite. Please check my report. ## Unassigned * OpenStack Metadata API and OVN in Neutron https://bugs.launchpad.net/neutron/+bug/1921809 It would be nice if OVN folks follow it more while haleyb replied. ## Medium, Assigned * [LB] Linux Bridge iptables firewall does not work without "ipset" https://bugs.launchpad.net/neutron/+bug/1922127 Assigned to ralonsoh ## Fix Released * Strings in tags field is limited to 60 chars https://bugs.launchpad.net/neutron/+bug/1921713 ## Almost RFE * allow using tap device on netdev enabled host https://bugs.launchpad.net/neutron/+bug/1922222 I requested the bug author to provide more information on the background ## RFE * [QoS] Add minimum guaranteed packet rate QoS rule https://bugs.launchpad.net/neutron/+bug/1922237 * [RFE] BFD for BGP Dynamic Routing https://bugs.launchpad.net/neutron/+bug/1922716

1 0

[cinder][horizon] Ussuri: Horizon shows error for managed volume(s)
by Eugen Block 06 Apr '21

06 Apr '21

Hi *, I'm struggling with a dashboard error that seems to be wrong, I can't seem to find any existing reports but I could have simply missed it so please point me to any existing reports about that. Anyway, a user imported two volumes from Ceph via 'cinder manage' in our Train version of OpenStack, we upgraded last week to Ussuri. One of those two volumes is bootable and the instance is up and running with both volumes (extended volume group). When clicking on the instance details the Horizon dashboard shows a red window saying "Failed to get attached volume." The dashboard log says: [Tue Apr 06 09:29:56.197238 2021] [wsgi:error] [pid 10140] [remote IP:59482] WARNING horizon.exceptions Recoverable error: volume_image_metadata But I can see both volumes in the details page, so the message seems incorrect. I searched the logs for this error message from before the upgrade but couldn't find any. So this seems to be new in Ussuri, I assume. Does anyone have the same experience with managed volumes? Thanks and best regards, Eugen

1 0

[victoria][neutron][horizon]l3-agent+port-forwarding
by Luke Camilleri 06 Apr '21

06 Apr '21

Hello everyone, I have enable the L3 extension for port-forwarding and can succesfully port-forward traffic after assigning an additional floating IP to the project. I would like to know if it is possible to enable the corresponding horizon functionality for this extension (port-forwarding) please? Regards

3 4

RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller
by Braden, Albert 06 Apr '21

06 Apr '21

Sorry that was a typo. Stopping RMQ during the removal of the *second* controller is what causes the problem. Is there a way to tell Centos 8 Train to use RMQ 3.7.24 instead of 3.7.28? -----Original Message----- From: Braden, Albert Sent: Thursday, April 1, 2021 9:34 AM To: 'openstack-discuss(a)lists.openstack.org' <openstack-discuss(a)lists.openstack.org> Subject: RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller I did some experimenting and it looks like stopping RMQ during the removal of the first controller is what causes the problem. After deploying the first controller, stopping the RMQ container on any controller including the new centos8 controller will cause the entire cluster to stop. This crash dump appears on the controllers that stopped in sympathy: https://paste.ubuntu.com/p/ZDgFgKtQTB/ This appears in the RMQ log: https://paste.ubuntu.com/p/5D2Qjv3H8c/ -----Original Message----- From: Braden, Albert Sent: Wednesday, March 31, 2021 8:31 AM To: openstack-discuss(a)lists.openstack.org Subject: RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller Centos7: {rabbit,"RabbitMQ","3.7.24"}, "Erlang/OTP 22 [erts-10.7.2.8] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:128] [hipe]\n"}, Centos8: {rabbit,"RabbitMQ","3.7.28"}, "Erlang/OTP 22 [erts-10.7.2.8] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:128] [hipe]\n"}, When I deploy the first Centos8 controller, RMQ comes up with all 3 nodes active and seems to be working fine until I shut down the 2nd controller. The only hint of trouble when I replace the 1st node is this error message the first time I run the deployment: https://paste.ubuntu.com/p/h9HWdfwmrK/ and the crash dump that appears on control2: crash dump log: https://paste.ubuntu.com/p/MpZ8SwTJ2T/ First 1500 lines of the dump: https://paste.ubuntu.com/p/xkCyp2B8j8/ If I wait for a few minutes then RMQ recovers on control2 and the 2nd run of the deployment seems to work, and there is no trouble until I shut down control1. -----Original Message----- From: Mark Goddard <mark(a)stackhpc.com> Sent: Wednesday, March 31, 2021 4:14 AM To: Braden, Albert <C-Albert.Braden(a)charter.com> Cc: openstack-discuss(a)lists.openstack.org Subject: [EXTERNAL] Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. On Tue, 30 Mar 2021 at 13:41, Braden, Albert <C-Albert.Braden(a)charter.com> wrote: > > I’ve created a heat stack and installed Openstack Train to test the Centos7->8 upgrade following the document here: > > > > https://docs.openstack.org/kolla-ansible/train/user/centos8.html#migrating-… > > > > I used the instructions here to successfully remove and replace control0 with a Centos8 box > > > > https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hos… > > > > After this my RMQ admin page shows all 3 nodes up, including the new control0. The name of the cluster is rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-2 /]# rabbitmqctl cluster_status > > Cluster status of node rabbit@chrnc-void-testupgrade-control-2 ... > > [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', > > 'rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}]}, > > {running_nodes,['rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}, > > {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, > > {partitions,[]}, > > {alarms,[{'rabbit@chrnc-void-testupgrade-control-0-replace',[]}, > > {'rabbit@chrnc-void-testupgrade-control-1',[]}, > > {'rabbit@chrnc-void-testupgrade-control-2',[]}]}] > > > > After that I create a new VM to verify that the cluster is still working, and then perform the same procedure on control1. When I shut down services on control1, the ansible playbook finishes successfully: > > > > kolla-ansible -i ../multinode stop --yes-i-really-really-mean-it --limit control1 > > … > > control1 : ok=45 changed=22 unreachable=0 failed=0 skipped=105 rescued=0 ignored=0 > > > > After this my RMQ admin page stops responding. When I check RMQ on the new control0 and the existing control2, the container is still up but RMQ is not running: > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status > > Error: this command requires the 'rabbit' app to be running on the target node. Start it with 'rabbitmqctl start_app'. > > > > If I start it on control0 and control2, then the cluster seems normal and the admin page starts working again, and cluster status looks normal: > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status > > Cluster status of node rabbit@chrnc-void-testupgrade-control-0-replace ... > > [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', > > 'rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}]}, > > {running_nodes,['rabbit@chrnc-void-testupgrade-control-2', > > 'rabbit@chrnc-void-testupgrade-control-0-replace']}, > > {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, > > {partitions,[]}, > > {alarms,[{'rabbit@chrnc-void-testupgrade-control-2',[]}, > > {'rabbit@chrnc-void-testupgrade-control-0-replace',[]}]}] > > > > But my hypervisors are down: > > > > (openstack) [root@chrnc-void-testupgrade-build kolla-ansible]# ohll > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB | > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > | 3 | chrnc-void-testupgrade-compute-2.dev.chtrse.com | QEMU | 172.16.2.106 | down | 5 | 8 | 2560 | 30719 | > > | 6 | chrnc-void-testupgrade-compute-0.dev.chtrse.com | QEMU | 172.16.2.31 | down | 5 | 8 | 2560 | 30719 | > > | 9 | chrnc-void-testupgrade-compute-1.dev.chtrse.com | QEMU | 172.16.0.30 | down | 5 | 8 | 2560 | 30719 | > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > > > When I look at the nova-compute.log on a compute node, I see RMQ failures every 10 seconds: > > > > 172.16.2.31 compute0 > > 2021-03-30 03:07:54.893 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out > > 2021-03-30 03:07:55.905 7 INFO oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] Reconnected to AMQP server on 172.16.1.132:5672 via [amqp] client with port 56422. > > 2021-03-30 03:08:05.915 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out > > > > In the RMQ logs I see this every 10 seconds: > > > > 172.16.1.132 control2 > > [root@chrnc-void-testupgrade-control-2 ~]# tail -f /var/log/kolla/rabbitmq/rabbit\(a)chrnc-void-testupgrade-control-2.log |grep 172.16.2.31 > > 2021-03-30 03:07:54.895 [warning] <0.13247.35> closing AMQP connection <0.13247.35> (172.16.2.31:56420 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): > > client unexpectedly closed TCP connection > > 2021-03-30 03:07:55.901 [info] <0.15288.35> accepting AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) > > 2021-03-30 03:07:55.903 [info] <0.15288.35> Connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) has a client-provided name: nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e > > 2021-03-30 03:07:55.904 [info] <0.15288.35> connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e): user 'openstack' authenticated and granted access to vhost '/' > > 2021-03-30 03:08:05.916 [warning] <0.15288.35> closing AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): > > > > Why does RMQ fail when I shut down the 2nd controller after successfully replacing the first one? Hi Albert, Could you share the versions of RabbitMQ and erlang in both versions of the container? When initially testing this setup, I think we had 3.7.24 on both sides. Perhaps the CentOS 8 version has moved on sufficiently to become incompatible? Mark > > > > I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails. > > > > The contents of this e-mail message and > any attachments are intended solely for the > addressee(s) and may contain confidential > and/or legally privileged information. If you > are not the intended recipient of this message > or if this message has been addressed to you > in error, please immediately alert the sender > by reply e-mail and then delete this message > and any attachments. If you are not the > intended recipient, you are notified that > any use, dissemination, distribution, copying, > or storage of this message or any attachment > is strictly prohibited. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

2 1

[docs][release] Creating Xena's landing pages
by Herve Beraud 06 Apr '21

06 Apr '21

Hello Docs team, This is a friendly reminder from the release team, I think that it should be safe for you to apply your process to create the new release series landing pages for docs.openstack.org. All stable branches are now created. If you want you can do the work before the final release date to avoid having to synchronize with the release team on that day. Let us know if you have any questions. Cheers -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

2 1

[all] Gate resources and performance
by Dan Smith 06 Apr '21

06 Apr '21

Hi all, I have become increasingly concerned with CI performance lately, and have been raising those concerns with various people. Most specifically, I'm worried about our turnaround time or "time to get a result", which has been creeping up lately. Right after the beginning of the year, we had a really bad week where the turnaround time was well over 24 hours. That means if you submit a patch on Tuesday afternoon, you might not get a test result until Thursday. That is, IMHO, a real problem and massively hurts our ability to quickly merge priority fixes as well as just general velocity and morale. If people won't review my code until they see a +1 from Zuul, and that is two days after I submitted it, that's bad. Things have gotten a little better since that week, due in part to getting past a rush of new year submissions (we think) and also due to some job trimming in various places (thanks Neutron!). However, things are still not great. Being in almost the last timezone of the day, the queue is usually so full when I wake up that it's quite often I don't get to see a result before I stop working that day. I would like to ask that projects review their jobs for places where they can cut out redundancy, as well as turn their eyes towards optimizations that can be made. I've been looking at both Nova and Glance jobs and have found some things I think we can do less of. I also wanted to get an idea of who is "using too much" in the way of resources, so I've been working on trying to characterize the weight of the jobs we run for a project, based on the number of worker nodes required to run all the jobs, as well as the wall clock time of how long we tie those up. The results are interesting, I think, and may help us to identify where we see some gains. The idea here is to figure out[1] how many "node hours" it takes to run all the normal jobs on a Nova patch compared to, say, a Neutron one. If the jobs were totally serialized, this is the number of hours a single computer (of the size of a CI worker) would take to do all that work. If the number is 24 hours, that means a single computer could only check *one* patch in a day, running around the clock. I chose the top five projects in terms of usage[2] to report here, as they represent 70% of the total amount of resources consumed. The next five only add up to 13%, so the "top five" seems like a good target group. Here are the results, in order of total consumption: Project % of total Node Hours Nodes ------------------------------------------ 1. TripleO 38% 31 hours 20 2. Neutron 13% 38 hours 32 3. Nova 9% 21 hours 25 4. Kolla 5% 12 hours 18 5. OSA 5% 22 hours 17 What that means is that a single computer (of the size of a CI worker) couldn't even process the jobs required to run on a single patch for Neutron or TripleO in a 24-hour period. Now, we have lots of workers in the gate, of course, but there is also other potential overhead involved in that parallelism, like waiting for nodes to be available for dependent jobs. And of course, we'd like to be able to check more than patch per day. Most projects have smaller gate job sets than check, but assuming they are equivalent, a Neutron patch from submission to commit would undergo 76 hours of testing, not including revisions and not including rechecks. That's an enormous amount of time and resource for a single patch! Now, obviously nobody wants to run fewer tests on patches before they land, and I'm not really suggesting that we take that approach necessarily. However, I think there are probably a lot of places that we can cut down the amount of *work* we do. Some ways to do this are: 1. Evaluate whether or not you need to run all of tempest on two configurations of a devstack on each patch. Maybe having a stripped-down tempest (like just smoke) to run on unique configs, or even specific tests. 2. Revisit your "irrelevant_files" lists to see where you might be able to avoid running heavy jobs on patches that only touch something small. 3. Consider moving some jobs to the experimental queue and run them on-demand for patches that touch particular subsystems or affect particular configurations. 4. Consider some periodic testing for things that maybe don't need to run on every single patch. 5. Re-examine tests that take a long time to run to see if something can be done to make them more efficient. 6. Consider performance improvements in the actual server projects, which also benefits the users. If you're a project that is not in the top ten then your job configuration probably doesn't matter that much, since your usage is dwarfed by the heavy projects. If the heavy projects would consider making changes to decrease their workload, even small gains have the ability to multiply into noticeable improvement. The higher you are on the above list, the more impact a small change will have on the overall picture. Also, thanks to Neutron and TripleO, both of which have already addressed this in some respect, and have other changes on the horizon. Thanks for listening! --Dan 1: https://gist.github.com/kk7ds/5edbfacb2a341bb18df8f8f32d01b37c 2; http://paste.openstack.org/show/C4pwUpdgwUDrpW6V6vnC/

12 36

[keystone][horizon][Victoria] scope-based policy problems
by Adam Tomas 06 Apr '21

06 Apr '21

Hi, I have some questions about horizon and keystone policies : Im trying to achieve "domain_admin" role with the ability to add/remove/update projects in particular domain and add/update/remove users in the same domain (and of course be able to see instances, networks, etc. in this domain). As described here http://lists.openstack.org/pipermail/openstack-discuss/2021-March/021105.ht… <http://lists.openstack.org/pipermail/openstack-discuss/2021-March/021105.ht…> : "Horizon does not support the system-scoped token yet as of Victoria and coming Wallaby.” So there is no way to write json/not scope-based policy for horizon and scope-based policy for keystone, because it will not work due to lack of scope information in horizon’s token? So the question is how the policies should look like? Is it possible at all to achieve such „domain admin” role? How in different way allow one user to add/remove/update projects and add/update/remove users? Another thing is, that if I use something like this in horizon/keystone policy: "identity:list_users_in_group": "rule:admin_required or (role:domain_admin and domain_id:%(domain_id)s)” then (besides of that domain users) there is also admin account in the list (so I assume admin „belongs” to all domains) - how to prevent newly created domain_admin from seeing admin account and making changes to that account? It really holds up my whole project, can you help mi guys? Best regards Adam

1 0

[kolla][glance] Few questions about images.
by Eddie Yen 06 Apr '21

06 Apr '21

Hi everyone, I want to ask about the image operating permission & Windows images issue we met since we can't find any answers on the internet. 1. Now we're still using Rocky with Ceph as storage. Sometimes we need to re-pack the image on the Openstack. We used to save as snapshot (re-pack by Nova ephemeral VM) or upload to image (re-pack by volume), then set snapshot/image's visibility to the public. But since Rocky, we can't do this anymore because when we try to set public, Horizon always shows "not enough permission" error. The workaround we're using for now is creating a nova snapshot after re-pack, download the snapshot, then upload the snapshot again as public. But it's utterly wasting time if the images are huge. So we want to know how to unleash this limitation can let us just change snapshot to public at least. 2. Openstack uses virtio as a network device by default, so we always install a virtio driver when packing Windows images. As the network performance issue in GSO/TSO enablement, we also need to disable them in device properties. But since Windows 10 2004 build (my thought) device properties always reset these settings after Sysprep. We found there's a workaround [1] to solve this issue, but may not work sometimes. Is there a better way to solve this issue? Many thanks, Eddie. [1] PersistAllDeviceInstalls | Microsoft Docs <https://docs.microsoft.com/en-us/windows-hardware/customize/desktop/unatten…>

1 1