December 2021 - openstack-discuss

Re: [ops] [kolla] RabbitMQ High Availability
by Bogdan Dobrelya 08 Dec '21

08 Dec '21

Please see inline >> I read this with great interest because we are seeing this issue. Questions: >> >> 1. We are running kola-ansible Train, and our RMQ version is 3.7.23. Should we be upgrading our Train clusters to use 3.8.x? >> 2. Document [2] recommends policy '^(?!(amq\.)|(.*_fanout_)|(reply_)).*'. I don't see this in our ansible playbooks, nor in any of the config files in the RMQ container. What would this look like in Ansible, and what should the resulting container config look like? >> 3. It appears that we are not setting "amqp_durable_queues = True". What does this setting look like in Ansible, and what file does it go into? > > Note that even having rabbit HA policies adjusted like that and its HA > replication factor [0] decreased (e.g. to a 2), there still might be > high churn caused by a large enough number of replicated durable RPC > topic queues. And that might cripple the cloud down with the incurred > I/O overhead because a durable queue requires all messages in it to be > persisted to a disk (for all the messaging cluster replicas) before they > are ack'ed by the broker. > > Given that said, Oslo messaging would likely require a more granular > control for topic exchanges and the durable queues flag - to tell it to > declare as durable only the most critical paths of a service. A single > config setting and a single control exchange per a service might be not > enough. Also note that therefore, amqp_durable_queue=True requires dedicated control exchanges configured for each service. Those that use 'openstack' as a default cannot turn the feature ON. Changing it to a service specific might also cause upgrade impact, as described in the topic [3]. [3] https://review.opendev.org/q/topic:scope-config-opts > > There are also race conditions with durable queues enabled, like [1]. A > solution could be where each service declare its own dedicated control > exchange with its own configuration. > > Finally, openstack components should add perhaps a *.next CI job to test > it with durable queues, like [2] > > [0] https://www.rabbitmq.com/ha.html#replication-factor > > [1] > https://zuul.opendev.org/t/openstack/build/aa514dd788f34cc1be3800e6d7dba0e8… > > [2] https://review.opendev.org/c/openstack/nova/+/820523 > >> >> Does anyone have a sample set of RMQ config files that they can share? >> >> It looks like my Outlook has ruined the link; reposting: >> [2] https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit > > > -- > Best regards, > Bogdan Dobrelya, > Irc #bogdando -- Best regards, Bogdan Dobrelya, Irc #bogdando

2 1

Swift Account Reaper Deletion Status
by Matthew Grinnell 08 Dec '21

08 Dec '21

Hello, I was curious about expected account-reaper behavior after deleting an account and if there is a way to be able to tell when all data for the account has been reaped. When I issue a delete on a swift account, I see subsequent HEAD requests on the account then return 410 Gone. Testing with debug logging I can see that after passing the delay_reaping time I see the reaper report my test object is deleted, HEAD requests against the account still return the same 410 Gone. Is there a way to tell when everything for an account has been purged, ie does the 410 return change at some point when that happens? Thanks, Matt

2 1

[all][tc] Technical Committee next weekly meeting on Dec 9th at 1500 UTC
by Ghanshyam Mann 08 Dec '21

08 Dec '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for Dec 9th at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Dec 8th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

[nova][placement] Cancelling Dec 21st and Dec 28th IRC meetings
by Sylvain Bauza 08 Dec '21

08 Dec '21

As we agreed on yesterday's meeting [1] we are about to cancel Dec 21st and Dec 28th meetings as we won't have a quorum for contributors during those weeks :-) -Sylvain [1] https://meetings.opendev.org/meetings/nova/2021/nova.2021-12-07-16.00.html

1 0

[Triple0] [Undercloud] Overcloud deployment getting failed at Introspection
by Lokendra Rathour 08 Dec '21

08 Dec '21

Hi Team, we are trying to deploy the triple0 Stein release on Centos 7. Undercloud is getting deployed successfully but the introspection of the overcloud node is getting failed with the reason as below: *on the DNS podman logs:* ()[root@undercloud dhcp-hostsdir]# tail -f /var/log/ironic-inspector/dnsmasq.log Dec 7 11:21:08 dnsmasq-dhcp[8]: DHCPDISCOVER(br-ctlplane) f8:f2:1e:9e:82:51 ignored Dec 7 11:21:08 dnsmasq-dhcp[8]: DHCPDISCOVER(br-ctlplane) f8:f2:1e:9e:82:50 ignored Dec 7 11:21:12 dnsmasq-dhcp[8]: DHCPDISCOVER(br-ctlplane) f8:f2:1e:9e:82:51 ignored *Checking the TCPDUMP at the same time gives:* 11:38:38.308660 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from f8:f2:1e:9e:82:51 (oui Unknown), length 300 11:38:38.435173 IP6 fe80::f53a:6448:4a29:e051 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 11:38:38.435184 IP6 fe80::f53a:6448:4a29:e051 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 11:38:41.155784 IP6 fe80::322d:3253:e3b2:316e > ff02::2: ICMP6, router solicitation, length 8 11:38:41.394304 IP6 fe80::f53a:6448:4a29:e051 > ff02::2: ICMP6, router solicitation, length 8 11:38:42.563182 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from f8:f2:1e:9e:82:50 (oui Unknown), length 300 11:38:42.611973 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from f8:f2:1e:9e:82:51 (oui Unknown), length 300 *We observed that DHCP Server is not offering the IP to the baremetal. * *DHCP discovery comes at br-ctlplane but no ip is offered.* Any input would be really helpful. Best Regards, Lokendra

2 3

[cinder] Bug deputy report for week of 12-08-2021
by Sofia Enriquez 08 Dec '21

08 Dec '21

This is a bug report from 11-24-2021 to 12-08-2021. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Medium - https://bugs.launchpad.net/cinder/+bug/1953168 ``Argument error in log message capacity filter". Assigned to caiquemello and Fix proposed. - https://bugs.launchpad.net/cinder/+bug/1952804 "cinder schedules backups on disabled backup services". Unassigned. - https://bugs.launchpad.net/cinder/+bug/1953185 "[IBM Storwize] Restore VM snapshot issue with replication volumes". Unassigned. - https://bugs.launchpad.net/cinder/+bug/1952927 "HPE 3PAR: migrating the vm attached with volume has been retyped will cause old volume to attach back again". Unassigned. - https://bugs.launchpad.net/cinder/+bug/1952805 "[posix] cinder schedules incremental backups on the wrong node". Unassigned. - https://bugs.launchpad.net/cinder/+bug/1952355 "[powerflex] Allocated capacity calculation doesn't work with powerflex driver". Unassigned. Low - https://bugs.launchpad.net/cinder/+bug/1953050 "A service should use a service specific control exchange for rpc". Fix proposed on master. *Quotas and more:* - https://bugs.launchpad.net/cinder/+bug/1952420 "Quota shows warnings on backup creation". Assigned to Gorka and Fix proposed. - https://bugs.launchpad.net/cinder/+bug/1952635 "When changing no_snapshot_gb_quota the gigabytes are wrong". Assigned to Gorka and Fix proposed. - https://bugs.launchpad.net/cinder/+bug/1952463 " Quota defaults and limits are not deleted when a volume type is deleted". Assigned to Gorka and Fix proposed. - https://bugs.launchpad.net/cinder/+bug/1952443 "Slow listings with many projects and deleted resources". Assigned to Gorka and Fix proposed. - https://bugs.launchpad.net/cinder/+bug/1952456 " Incorrect limit for private type volumes". Assigned to Gorka and Fix proposed. - https://bugs.launchpad.net/cinder/+bug/1952683 "Documentation typos". Fix proposed on master. Incomplete - https://bugs.launchpad.net/cinder/+bug/1952998 "[Ceph] Volume backup created with multiple snapshots preventing removal". Unassigned. Cheers -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[ironic] [ops] RFC: deprecating instance network boot in Ironic
by Dmitry Tantsur 08 Dec '21

08 Dec '21

Hi folks, As discussed at the PTG, I'd like to collect thoughts about this proposal. I would like to deprecate support for network-booting **normal instances**. This does not concern the deployment process, ramdisk deploy or booting from volume! Historically, Ironic has been able to boot the final instances from network (or ISO in case of virtual media). This is no longer the recommended mode, except for the ramdisk deploy or boot-from-volume. It adds quite a bit of complexity to deploy interfaces and has one serious security implication: there is no way to update kernel/ramdisk once the instance is deployed (without rebuilding it). For this reason, I don't think anybody should use instance network booting in production. I would like to announce the deprecation in the Yoga cycle with the intention of removing the feature in Z. Then we can also get rid of the boot_option capability (which is not really a capability). Am I missing something? Any objections? Dmitry -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

1 0

How to config vGPU in cyborg
by norman.kern 08 Dec '21

08 Dec '21

Hi guys, I want to config cyborg with kolla in openstack wallaby cluster, but I found that few docs can be found to config vGPU, Where can I find the details in configuration from openstack website? And I found another question, Cyborg can not config different types of vGPU for a pGPU. Waiting for you replies. Thanks.

1 0

[horizon] No Weekly meeting today
by vishal manchanda 07 Dec '21

07 Dec '21

Hello Team, Since there are no agenda items [1] to discuss for today's horizon weekly meeting. So let's cancel today's weekly meeting. Next weekly meeting will be on 15 December. Thanks & Regards, Vishal Manchanda [1] https://etherpad.opendev.org/p/horizon-release-priorities

1 0

[all][ptl][tc] Updates on Community wide goal: Current Active goals
by Ghanshyam Mann 07 Dec '21

07 Dec '21

Hello Everyone, You might know, TC has de-coupled the community-wide goals from the release cycle[1]. As per the new structure, community-wide goals are tied or have to be finished within the release cycle. Instead, it can have target dates which can be a particular cycle release date or multi-cycle or even in between a release cycle. Target dates will be selected by considering the amount of work needed and the community bandwidth. This will help to make sure the goal is ready to start and also continue working on the goal until work is completed. 'Secure RBAC' goal is one example of a new structure that has different milestones targeting the different set of work[2]. I think this will be helpful for PTLs and project teams to plan the active goals for their project. Current Selected (Active) Goals: ------------------------------------- We have two Selected/Active goals. The project team can start/continue working on: 1. Migrate from oslo.rootwrap to oslo.privsep [3]. This is contouring from the previous cycle. 2. Consistent and Secure Default RBAC [4]. New design/implementation details are finalized now, please read those and for any queries, plan to attend the policy popup team biweekly meeting[5]. [1] https://review.opendev.org/c/openstack/governance/+/816387 [2] https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rb… [3] https://governance.openstack.org/tc/goals/selected/migrate-to-privsep.html [4] https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rb… [5] https://wiki.openstack.org/wiki/Consistent_and_Secure_Default_Policies_Popu… -gmann

1 0