December 2021 - openstack-discuss

[ops]RabbitMQ High Availability
by DHilsbos＠performair.com 06 Dec '21

06 Dec '21

All; In the time I've been part of this mailing list, the subject of RabbitMQ high availability has come up several times, and each time specific recommendations for both Rabbit and Open Stack are provided. I remember it being an A or B kind of recommendation (i.e. configure Rabbit like A1, and Open Stack like A2, OR configure Rabbit like B1, and Open Stack like B2). Unfortunately, I can't find the previous threads on this topic. Does anyone have this information, that they would care to share with me? Thank you, Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

3 3

[kolla-ansible] Default masakari configuration should evacuate instances across nodes in wallaby?
by Rodrigo Lima 06 Dec '21

06 Dec '21

I'm working on upgrading an openstack farm from victoria to wallaby. After successful upgrade, I would like to enable hacluster and masakari to test HA between failing compute nodes. Everything seems to be running (pacemaker with the remote nodes OK, corosync without errors, masakari-monitors detects the 2 compute nodes online), but... when I simulate the failure of a node with shutdown, the failure and notification appears in the hostmonitor log, but the instances that were on the failed node don't evacuate, and I couldn't find documentation that explains how to do this specific configuration, if necessary. Does anyone have any ideas?

3 5

[dev][panko] deprecation doubts
by Soares Sarto, Ricardo 06 Dec '21

06 Dec '21

Hi Everyone Now that Panko has been deprecated, where are the events stored? What is the evolution strategy for OpenStack Telemetry? What is a current list of data generated by core openstack services (nova, neutron, keystone, glance, cinder, heat, etc.) to ceilometer ? (e.g. this link is from ocata … where is the current one ? https://docs.openstack.org/ocata/admin-guide/telemetry-data-collection.html ) Thank you!

1 0

OpenInfra Live - December 9, 2021 at 9am CT
by Ashlee Ferguson 06 Dec '21

06 Dec '21

Hi everyone, This week’s OpenInfra Live episode is brought to you by the OpenStack community. This new episode of the “Large Scale OpenStack” show will discuss operators' tricks and tools. In a live and direct discussion, our guests will reveal tricks and homegrown tools that they use in the trenches to keep their OpenStack clusters ticking like clockwork. Episode: Large Scale OpenStack: Operators’ tricks and tools Date and time: December 9, 2021 at 9am CT (1500 UTC) You can watch us live on: YouTube: https://www.youtube.com/watch?v=F_9KKAQE4fc LinkedIn: https://www.linkedin.com/feed/update/urn:li:ugcPost:6873686260419055616/ Facebook: https://www.facebook.com/104139126308032/posts/4596356560419577/ WeChat: recording will be posted on OpenStack WeChat after the live stream Speakers: Shatadru Bandyopadhyay (Workday) Thierry Carrez (OpenInfra Foundation) Thomas Goirand (Infomaniak) Axel Jacquet (Infomaniak) Gene Kuo (LINE) Belmiro Moreira (CERN) Arnaud Morin (OVHCloud) Adrien Pensart (OVHCloud) Have an idea for a future episode? Share it now at ideas.openinfra.live. Thanks! Ashlee Ashlee Ferguson Carlson Community & Events Open Infrastructure Foundation

1 0

[cinder] midcycle R-17 followup meeting thursday
by Brian Rosmaita 06 Dec '21

06 Dec '21

The poll results are in and our midcycle followup meeting will be held: DATE: Thursday, 9 December 2021 TIME: 1400-1430 UTC LOCATION: https://bluejeans.com/3228528973 The meeting will be recorded. The midcycle etherpad is here: https://etherpad.opendev.org/p/cinder-yoga-midcycles (Followup info starts at line 34, unless some joker has added a bunch of stuff at the top of the etherpad.) The primary topic is Gorka's proposal to revamp the quota system. There are links to the proposed spec and some POC patches on the etherpad. It's worth taking a look in advance so that we can have a short and productive meeting! cheers, brian

1 0

Re: [ops] [kolla] RabbitMQ High Availability
by Bogdan Dobrelya 06 Dec '21

06 Dec '21

> I read this with great interest because we are seeing this issue. Questions: > > 1. We are running kola-ansible Train, and our RMQ version is 3.7.23. Should we be upgrading our Train clusters to use 3.8.x? > 2. Document [2] recommends policy '^(?!(amq\.)|(.*_fanout_)|(reply_)).*'. I don't see this in our ansible playbooks, nor in any of the config files in the RMQ container. What would this look like in Ansible, and what should the resulting container config look like? > 3. It appears that we are not setting "amqp_durable_queues = True". What does this setting look like in Ansible, and what file does it go into? Note that even having rabbit HA policies adjusted like that and its HA replication factor [0] decreased (e.g. to a 2), there still might be high churn caused by a large enough number of replicated durable RPC topic queues. And that might cripple the cloud down with the incurred I/O overhead because a durable queue requires all messages in it to be persisted to a disk (for all the messaging cluster replicas) before they are ack'ed by the broker. Given that said, Oslo messaging would likely require a more granular control for topic exchanges and the durable queues flag - to tell it to declare as durable only the most critical paths of a service. A single config setting and a single control exchange per a service might be not enough. There are also race conditions with durable queues enabled, like [1]. A solution could be where each service declare its own dedicated control exchange with its own configuration. Finally, openstack components should add perhaps a *.next CI job to test it with durable queues, like [2] [0] https://www.rabbitmq.com/ha.html#replication-factor [1] https://zuul.opendev.org/t/openstack/build/aa514dd788f34cc1be3800e6d7dba0e8… [2] https://review.opendev.org/c/openstack/nova/+/820523 > > Does anyone have a sample set of RMQ config files that they can share? > > It looks like my Outlook has ruined the link; reposting: > [2] https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit -- Best regards, Bogdan Dobrelya, Irc #bogdando

1 0

[cinder][OpenstackAnsible] Xena Cycle-Trailing Release Deadline
by Előd Illés 06 Dec '21

06 Dec '21

Hello teams with trailing projects, Next week is the Xena cycle-trailing release deadline [1], and all projects following the cycle-trailing release model must release their Xena deliverables by 16 December, 2021. The following trailing projects haven't been released yet for Xena: Cinder team's deliverables: - cinderlib OSA team's deliverables: - openstack-ansible-roles - openstack-ansible Patches were generated 2 weeks ago [2], feel free to update those if they are not up to date anymore. This is just a friendly reminder to allow you to release these projects in time. Do not hesitate to ping us if you have any questions or concerns. [1] https://releases.openstack.org/yoga/schedule.html#y-cycle-trail [2] https://review.opendev.org/q/topic:trailing-xena Előd Illés irc: elodilles

1 0

[neutron] Bug deputy report for week of November 29
by Lajos Katona 06 Dec '21

06 Dec '21

Hi Neutron Team I was bug deputy in neutron last week. Summary ========== 3 bugs need more attention, one is related to db upgrade (upgrade from Rocky to Stein), one bug is an issue between Nova and Neutron and related to network-vif-plug event, and one seems to be related to IPv6 Metadata. If you have time please check them and help solve them and check the reviews I linked. Needs attention ================ * "neutron-db-manage upgrade" is interrupted with error about neutron.portuplinkstatuspropagation ( https://bugs.launchpad.net/neutron/+bug/1952675 ) * VirtualInterfaceCreateException due to "Timeout waiting for [('network-vif-plugged'..." (https://bugs.launchpad.net/neutron/+bug/1944779 ) * DHCP agent fails to fully configure DHCP namespaces because of duplicate address detected (https://bugs.launchpad.net/neutron/+bug/1953165 ) ** Seems to be an issue with IPv6 metadata. Bugs with assignees =================== * Neurton linuxbridge agent does not create VXLAN interface ( https://bugs.launchpad.net/neutron/+bug/1952611 ) ** Fix proposed by mnaser to oslo.privsep: https://review.opendev.org/c/openstack/oslo.privsep/+/819996 * Segment updates may cause unnecessary overload ( https://bugs.launchpad.net/neutron/+bug/1952730 ) ** fix from rubasov: https://review.opendev.org/c/openstack/neutron/+/819777 * Cannot setup IPv6 overlay network in xena ( https://bugs.launchpad.net/neutron/+bug/1952897 ) ** https://review.opendev.org/c/openstack/neutron/+/820376 * Allow modification of max retries in OVSNeutronAgent ( https://bugs.launchpad.net/neutron/+bug/1952898 ) ** https://review.opendev.org/c/openstack/neutron/+/819816 * Gratuitous ARPs are not sent during master transition ( https://bugs.launchpad.net/neutron/+bug/1952907 ): ** Possible fix from Liu: https://review.opendev.org/c/openstack/neutron/+/712474 * On IPv6 overlay networks linuxbridge vxlans are created always on loopback device (https://bugs.launchpad.net/neutron/+bug/1953139 ): ** https://review.opendev.org/c/openstack/neutron/+/820484 RFEs ====== * [RFE] Unify quota engine API ( https://bugs.launchpad.net/neutron/+bug/1953170 ) ** We discussed the need for this rfe on last Friday's drivers meeting: https://meetings.opendev.org/meetings/neutron_drivers/2021/neutron_drivers.… Regards Lajos Katona (lajoskatona)

1 0

[kolla] Plan to deprecate binary and unify on single distrubition
by Michał Nasiadka 06 Dec '21

06 Dec '21

Hello Koalas, On the PTG we have discussed two topics: 1) Deprecate and drop binary type of Kolla images 2) Use a common base (single Linux distribution) for Kolla images This is a call for feedback - for people that have not been attending the PTG. What this essentially mean for consumers: 1) In Yoga cycle we will deprecate binary type of Kolla images, and in Z cycle those will be dropped. 2) We are not going to support CentOS Stream 9 (cs9) as a base operating system, and the source type build will rely on CentOS Stream 8 in Z release. 3) Beginning from A release Kolla will build only Debian source images - but Kolla-Ansible will still support deployment of those images on CentOS/Ubuntu/Debian Host operating systems (and Rocky Linux to be added in Yoga to that mix). Justification: The Kolla project team is limited in numbers, therefore supporting current broad mix of operating systems (especially with CentOS Stream 9 ,,on the way’’) is a significant maintenance burden. By dropping binary type of images - users would be running more tested images (since Kolla/Kolla-Ansible CI runs source images jobs as voting). In Xena we’ve already changed the default image type Kolla-Ansible uses to source. We also feel that using a unified base OS for Kolla container images is a way to remove some of the maintenance burden (including CI cycles and Request for feedback: If any of those changes is a no go from your perspective - we’d like to hear your opinions. Best regards, Michal Nasiadka

9 17

[TripleO] IPV6 Support
by Lokendra Rathour 03 Dec '21

03 Dec '21

Hi, I am trying to install the undercloud using the ipv6 address on Centos 8 using Train release. It is seen that the deployment of undercloud is getting failed with error as mentioned below. Same deployment is working in the normal ipv4 case. So Questions: 1. is IPV6 supported in TripleO , if yes then please suggest how. Error: nied", " attempt(s): 2", "2021-11-25 07:25:31,535 WARNING: 65355 -- Retrying running container: zaqar", "2021-11-25 07:25:34,887 ERROR: 65355 -- ['/usr/bin/podman', 'run', '--user', '0', '--name', 'container-puppet-zaqar', '--env', 'PUPPET_TAGS=file,file_line,concat,augeas,cron,zaqar_config', '--env', 'NAME=zaqar', '--env', 'HOSTNAME=undercloud', '--env', 'NO_ARCHIVE=', '--env', 'STEP=6', '--env', 'NET_HOST=true', '--env', 'DEBUG=False', '--volume', '/etc/localtime:/etc/localtime:ro', '--volume', '/tmp/tmp06nvxzzz:/etc/config.pp:ro', '--volume', '/etc/puppet/:/tmp/puppet-etc/:ro', '--volume', '/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume', '/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume', '/var/lib/config-data:/var/lib/config-data/:rw', '--volume', '/var/lib/container-puppet/puppetlabs/facter.conf:/etc/puppetlabs/facter/facter.conf:ro', '--volume', '/var/lib/container-puppet/puppetlabs/:/opt/puppetlabs/:ro', '--volume', '/dev/log:/dev/log:rw', '--rm', '--log-driver', 'k8s-file', '--log-opt', 'path=/var/log/containers/stdouts/container-puppet-zaqar.log', '--security-opt', 'label=disable', '--volume', '/usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro', '--entrypoint', '/var/lib/container-puppet/container-puppet.sh', '--net', 'host', '--volume', '/etc/hosts:/etc/hosts:ro', '--volume', '/var/lib/container-puppet/container-puppet.sh:/var/lib/container-puppet/container-puppet.sh:ro', 'undercloud.ctlplane.localdomain:8787/tripleotraincentos8/centos-binary-zaqar-wsgi:current-tripleo'] run failed after Error: container_linux.go:370: starting container process caused: error adding seccomp filter rule for syscall bdflush: permission denied: OCI permission denied", " attempt(s): 3", "2021-11-25 07:25:34,888 WARNING: 65355 -- Retrying running container: zaqar", "2021-11-25 07:25:34,888 ERROR: 65355 -- Failed running container for zaqar", "2021-11-25 07:25:34,888 INFO: 65355 -- Finished processing puppet configs for zaqar", "2021-11-25 07:25:34,889 ERROR: 65345 -- ERROR configuring crond", "2021-11-25 07:25:34,889 ERROR: 65345 -- ERROR configuring glance_api", "2021-11-25 07:25:34,889 ERROR: 65345 -- ERROR configuring heat_api", "2021-11-25 07:25:34,889 ERROR: 65345 -- ERROR configuring heat", "2021-11-25 07:25:34,889 ERROR: 65345 -- ERROR configuring ironic_api", "2021-11-25 07:25:34,889 ERROR: 65345 -- ERROR configuring ironic", "2021-11-25 07:25:34,889 ERROR: 65345 -- ERROR configuring ironic_inspector", "2021-11-25 07:25:34,889 ERROR: 65345 -- ERROR configuring neutron", "2021-11-25 07:25:34,889 ERROR: 65345 -- ERROR configuring iscsid", "2021-11-25 07:25:34,889 ERROR: 65345 -- ERROR configuring keystone", "2021-11-25 07:25:34,890 ERROR: 65345 -- ERROR configuring memcached", "2021-11-25 07:25:34,890 ERROR: 65345 -- ERROR configuring mistral", "2021-11-25 07:25:34,890 ERROR: 65345 -- ERROR configuring mysql", "2021-11-25 07:25:34,890 ERROR: 65345 -- ERROR configuring nova", "2021-11-25 07:25:34,890 ERROR: 65345 -- ERROR configuring rabbitmq", "2021-11-25 07:25:34,890 ERROR: 65345 -- ERROR configuring placement", "2021-11-25 07:25:34,890 ERROR: 65345 -- ERROR configuring swift", "2021-11-25 07:25:34,890 ERROR: 65345 -- ERROR configuring swift_ringbuilder", "2021-11-25 07:25:34,890 ERROR: 65345 -- ERROR configuring zaqar" ], "stderr_lines": [], "_ansible_no_log": false, "attempts": 15 } ] ] Not cleaning working directory /home/stack/tripleo-heat-installer-templates Not cleaning ansible directory /home/stack/undercloud-ansible-mw4crw92 Install artifact is located at /home/stack/undercloud-install-20211125072536.tar.bzip2 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Deployment Failed! ERROR: Heat log files: /var/log/heat-launcher/undercloud_deploy-o_qf1b4w !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Deployment failed. Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1345, in _standalone_deploy raise exceptions.DeploymentError('Deployment failed') tripleoclient.exceptions.DeploymentError: Deployment failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1363, in _standalone_deploy raise exceptions.DeploymentError(six.text_type(e)) tripleoclient.exceptions.DeploymentError: Deployment failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1451, in take_action if self._standalone_deploy(parsed_args) != 0: File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1400, in _standalone_deploy raise exceptions.DeploymentError('Deployment failed.') tripleoclient.exceptions.DeploymentError: Deployment failed. clean_up Deploy: Deployment failed. Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1345, in _standalone_deploy raise exceptions.DeploymentError('Deployment failed') tripleoclient.exceptions.DeploymentError: Deployment failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1363, in _standalone_deploy raise exceptions.DeploymentError(six.text_type(e)) tripleoclient.exceptions.DeploymentError: Deployment failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 136, in run ret_val = super(OpenStackShell, self).run(argv) File "/usr/lib/python3.6/site-packages/cliff/app.py", line 281, in run result = self.run_subcommand(remainder) File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 176, in run_subcommand ret_value = super(OpenStackShell, self).run_subcommand(argv) File "/usr/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1451, in take_action if self._standalone_deploy(parsed_args) != 0: File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1400, in _standalone_deploy raise exceptions.DeploymentError('Deployment failed.') tripleoclient.exceptions.DeploymentError: Deployment failed. END return value: 1 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! An error has occured while deploying the Undercloud. See the previous output for details about what went wrong. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Command '['sudo', '--preserve-env', 'openstack', 'tripleo', 'deploy', '--standalone', '--standalone-role', 'Undercloud', '--stack', 'undercloud', '--local-domain=localdomain', '--local-ip=abce:abce:abce::1/64', '--templates=/usr/share/openstack-tripleo-heat-templates/', '--networks-file=network_data_undercloud.yaml', '--heat-native', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/undercloud.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/use-dns-for-vips.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/podman.yaml', '-e', '/home/stack/containers-prepare-parameter.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/ironic.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/ironic-inspector.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/mistral.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/zaqar-swift-backend.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/tempest.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/ssl/no-tls-endpoints-public-ip.yaml', '--deployment-user', 'stack', '--output-dir=/home/stack', '-e', '/home/stack/tripleo-config-generated-env-files/undercloud_parameters.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/tripleo-validations.yaml', '--debug', '--log-file=install-undercloud.log', '-e', '/usr/share/openstack-tripleo-heat-templates/undercloud-stack-vstate-dropin.yaml']' returned non-zero exit status 1. Command '['sudo', '--preserve-env', 'openstack', 'tripleo', 'deploy', '--standalone', '--standalone-role', 'Undercloud', '--stack', 'undercloud', '--local-domain=localdomain', '--local-ip=abce:abce:abce::1/64', '--templates=/usr/share/openstack-tripleo-heat-templates/', '--networks-file=network_data_undercloud.yaml', '--heat-native', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/undercloud.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/use-dns-for-vips.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/podman.yaml', '-e', '/home/stack/containers-prepare-parameter.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/ironic.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/ironic-inspector.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/mistral.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/zaqar-swift-backend.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/tempest.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/ssl/no-tls-endpoints-public-ip.yaml', '--deployment-user', 'stack', '--output-dir=/home/stack', '-e', '/home/stack/tripleo-config-generated-env-files/undercloud_parameters.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/tripleo-validations.yaml', '--debug', '--log-file=install-undercloud.log', '-e', '/usr/share/openstack-tripleo-heat-templates/undercloud-stack-vstate-dropin.yaml']' returned non-zero exit status 1. END return value: 1 [stack@undercloud ~]$

2 5