October 2021 - openstack-discuss - lists.openstack.org

[Designate] After rebuilding Queens clusters on Train, race condition causes Designate record creation to fail
by Braden, Albert 13 Oct '21

13 Oct '21

After enabling redis, and allowing TCP 6379 and 26379, I see it in /etc/designate/designate.conf in the designate_producer container: backend_url = redis://admin:<key>@10.221.176.48:26379?sentinel=kolla&sentinel_fallback=10.221.176.173:26379&sentinel_fallback=10.221.177.38:26379&db=0&socket_timeout=60&retry_on_timeout=yes And I can get to ports 6379 and 26379 with nc: (designate-producer)[root@dva3-ctrl3 /]# nc 10.221.176.173 26379 / -ERR unknown command `/`, with args beginning with: But I still see the DB error when TF rebuilds a VM: 2021-10-13 15:35:23.941 26 ERROR oslo_messaging.notify.dispatcher designate.exceptions.DuplicateRecord: Duplicate Record What am I missing? -----Original Message----- From: Michael Johnson <johnsomor(a)gmail.com> Sent: Tuesday, October 12, 2021 11:33 AM To: Braden, Albert <abraden(a)verisign.com> Cc: openstack-discuss(a)lists.openstack.org Subject: [EXTERNAL] Re: Re: Re: After rebuilding Queens clusters on Train, race condition causes Designate record creation to fail Caution: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. I don't have a good answer for you on that as it pre-dates my history with Designate a bit. I suspect it has to do with the removal of the pool-manager and the restructuring of the controller code. Maybe someone else on the discuss list has more insight. Michael On Tue, Oct 12, 2021 at 5:47 AM Braden, Albert <abraden(a)verisign.com> wrote: > > Thank you Michael, this is very helpful. Do you have any insight into why we don't experience this in Queens clusters? We aren't running a lock manager there either, and I haven't been able to duplicate the problem there. > > -----Original Message----- > From: Michael Johnson <johnsomor(a)gmail.com> > Sent: Monday, October 11, 2021 4:24 PM > To: Braden, Albert <abraden(a)verisign.com> > Cc: openstack-discuss(a)lists.openstack.org > Subject: [EXTERNAL] Re: Re: After rebuilding Queens clusters on Train, race condition causes Designate record creation to fail > > Caution: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. > > You will need one of the Tooz supported distributed lock managers: > Consul, Memcacded, Redis, or zookeeper. > > Michael > > On Mon, Oct 11, 2021 at 11:57 AM Braden, Albert <abraden(a)verisign.com> wrote: > > > > After investigating further, I realized that we're not running redis, and I think that means that redis_connection_string doesn't get set. Does this mean that we must run redis, or is there a workaround? > > > > -----Original Message----- > > From: Braden, Albert > > Sent: Monday, October 11, 2021 2:48 PM > > To: 'johnsomor(a)gmail.com' <johnsomor(a)gmail.com> > > Cc: 'openstack-discuss(a)lists.openstack.org' <openstack-discuss(a)lists.openstack.org> > > Subject: RE: [EXTERNAL] Re: After rebuilding Queens clusters on Train, race condition causes Designate record creation to fail > > > > I think so. I see this: > > > > ansible/roles/designate/templates/designate.conf.j2:backend_url = {{ redis_connection_string }} > > > > ansible/group_vars/all.yml:redis_connection_string: "redis://{% for host in groups['redis'] %}{% if host == groups['redis'][0] %}admin:{{ redis_master_password }}@{{ 'api' | kolla_address(host) | put_address_in_context('url') }}:{{ redis_sentinel_port }}?sentinel=kolla{% else %}&sentinel_fallback={{ 'api' | kolla_address(host) | put_address_in_context('url') }}:{{ redis_sentinel_port }}{% endif %}{% endfor %}&db=0&socket_timeout=60&retry_on_timeout=yes" > > > > Did anything with the distributed lock manager between Queens and Train? > > > > -----Original Message----- > > From: Michael Johnson <johnsomor(a)gmail.com> > > Sent: Monday, October 11, 2021 1:15 PM > > To: Braden, Albert <abraden(a)verisign.com> > > Cc: openstack-discuss(a)lists.openstack.org > > Subject: [EXTERNAL] Re: After rebuilding Queens clusters on Train, race condition causes Designate record creation to fail > > > > Caution: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. > > > > Hi Albert, > > > > Have you configured your distributed lock manager for Designate? > > > > [coordination] > > backend_url = <DLM URL> > > > > Michael > > > > On Fri, Oct 8, 2021 at 7:38 PM Braden, Albert <abraden(a)verisign.com> wrote: > > > > > > Hello everyone. It’s great to be back working on OpenStack again. I’m at Verisign now. I can hardly describe how happy I am to have an employer that does not attach nonsense to the bottom of my emails! > > > > > > > > > > > > We are rebuilding our clusters from Queens to Train. On the new Train clusters, customers are complaining that deleting a VM and then immediately creating a new one with the same name (via Terraform for example) intermittently results in a missing DNS record. We can duplicate the issue by building a VM with terraform, tainting it, and applying. > > > > > > > > > > > > Before applying the change, we see the DNS record in the recordset: > > > > > > > > > > > > $ openstack recordset list dva3.vrsn.com. --all |grep openstack-terra > > > > > > | f9aa73c1-84ba-4854-be71-cbb616de672c | 8d1c84082a044a53abe0d519ed9e8c60 | openstack-terra-test-host.dev-ostck.dva3.vrsn.com. | A | 10.220.4.89 | ACTIVE | NONE | > > > > > > $ > > > > > > > > > > > > and we can pull it from the DNS server on the controllers: > > > > > > > > > > > > $ for i in {1..3}; do dig @dva3-ctrl${i}.cloud.vrsn.com -t axfr dva3.vrsn.com. |grep openstack-terra; done > > > > > > openstack-terra-test-host.dev-ostck.dva3.vrsn.com. 1 IN A 10.220.4.89 > > > > > > openstack-terra-test-host.dev-ostck.dva3.vrsn.com. 1 IN A 10.220.4.89 > > > > > > openstack-terra-test-host.dev-ostck.dva3.vrsn.com. 1 IN A 10.220.4.89 > > > > > > > > > > > > After applying the change, we don’t see it: > > > > > > > > > > > > $ openstack recordset list dva3.vrsn.com. --all |grep openstack-terra > > > > > > | f9aa73c1-84ba-4854-be71-cbb616de672c | 8d1c84082a044a53abe0d519ed9e8c60 | openstack-terra-test-host.dev-ostck.dva3.vrsn.com. | A | 10.220.4.89 | ACTIVE | NONE | > > > > > > $ > > > > > > $ for i in {1..3}; do dig @dva3-ctrl${i}.cloud.vrsn.com -t axfr dva3.vrsn.com. |grep openstack-terra; done > > > > > > $ openstack recordset list dva3.vrsn.com. --all |grep openstack-terra > > > > > > $ > > > > > > > > > > > > We see this in the logs: > > > > > > > > > > > > 2021-10-09 01:53:44.307 27 ERROR oslo_messaging.notify.dispatcher oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'c70e693b4c47402db088c43a5a177134-openstack-terra-test-host.de...' for key 'unique_recordset'") > > > > > > 2021-10-09 01:53:44.307 27 ERROR oslo_messaging.notify.dispatcher [SQL: INSERT INTO recordsets (id, version, created_at, zone_shard, tenant_id, zone_id, name, type, ttl, reverse_name) VALUES (%(id)s, %(version)s, %(created_at)s, %(zone_shard)s, %(tenant_id)s, %(zone_id)s, %(name)s, %(type)s, %(ttl)s, %(reverse_name)s)] > > > > > > 2021-10-09 01:53:44.307 27 ERROR oslo_messaging.notify.dispatcher [parameters: {'id': 'dbbb904c347241a791aa01ca33a87b23', 'version': 1, 'created_at': datetime.datetime(2021, 10, 9, 1, 53, 44, 182652), 'zone_shard': 3184, 'tenant_id': '8d1c84082a044a53abe0d519ed9e8c60', 'zone_id': 'c70e693b4c47402db088c43a5a177134', 'name': 'openstack-terra-test-host.dev-ostck.dva3.vrsn.com.', 'type': 'A', 'ttl': None, 'reverse_name': '.moc.nsrv.3avd.kctso-ved.tsoh-tset-arret-kcatsnepo'}] > > > > > > > > > > > > It appears that Designate is trying to create the new record before the deletion of the old one finishes. > > > > > > > > > > > > Is anyone else seeing this on Train? The same set of actions doesn’t cause this error in Queens. Do we need to change something in our Designate config, to make it wait until the old records are finished deleting before attempting to create the new ones?

1 0

Open stack Ansible 23.1.0.dev35 VM launching errors
by Midhunlal Nb 13 Oct '21

13 Oct '21

Hi team, -->Successfully I installed Openstack ansible 23.1.0.dev35. --->I logged in to horizon and created a new network and launched a vm but I am getting an error. Error: Failed to perform requested operation on instance "hope", the instance has an error status: Please try again later [Error: Build of instance b6f42229-06d6-4365-8a20-07df869f1610 aborted: Failed to allocate the network(s), not rescheduling.]. -->Then I checked log | fault | {'code': 500, 'created': '2021-10-08T12:26:44Z', 'message': 'Build of instance b6f42229-06d6-4365-8a20-07df869f1610 aborted: Failed to allocate the network(s), not rescheduling.', 'details': 'Traceback (most recent call last):\n File "/openstack/venvs/nova-23.1.0.dev42/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7235, in _create_guest_with_network\n post_xml_callback=post_xml_callback)\n File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__\n next(self.gen)\n File "/openstack/venvs/nova-23.1.0.dev42/lib/python3.6/site-packages/nova/compute/manager.py", line 479, in wait_for_instance_event\n actual_event = event.wait()\n File "/openstack/venvs/nova-23.1.0.dev42/lib/python3.6/site-packages/eventlet/event.py", line 125, in wait\n result = hub.switch()\n File "/openstack/venvs/nova-23.1.0.dev42/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 313, in switch\n return self.greenlet.switch()\neventlet.timeout.Timeout: 300 seconds\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/openstack/venvs/nova-23.1.0.dev42/lib/python3.6/site-packages/nova/compute/manager.py", line 2397, in _build_and_run_instance\n accel_info=accel_info)\n File "/openstack/venvs/nova-23.1.0.dev42/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 4200, in spawn\n cleanup_instance_disks=created_disks)\n File "/openstack/venvs/nova-23.1.0.dev42/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7258, in _create_guest_with_network\n raise exception.VirtualInterfaceCreateException()\nnova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/openstack/venvs/nova-23.1.0.dev42/lib/python3.6/site-packages/nova/compute/manager.py", line 2219, in _do_build_and_run_instance\n filter_properties, request_spec, accel_uuids)\n File "/openstack/venvs/nova-23.1.0.dev42/lib/python3.6/site-packages/nova/compute/manager.py", line 2458, in _build_and_run_instance\n reason=msg)\nnova.exception.BuildAbortException: Build of instance b6f42229-06d6-4365-8a20-07df869f1610 aborted: Failed to allocate the network(s), not rescheduling.\n'} | Please help me with this error. Thanks & Regards Midhunlal N B

3 9

Learn how users are using OpenStack at OpenInfra Live Keynotes
by Kendall Nelson 13 Oct '21

13 Oct '21

Hello Everyone, You might have heard that the OpenInfra Foundation is hosting its largest free virtual event of this year—OpenInfra Live: Keynote OpenInfra Live: Keynotes: https://openinfra.dev/live/keynotes Date and time: November 17-18 (1500-1700 UTC on each day) Register for free: https://www.eventbrite.com/e/openinfra-live-keynotes-tickets-169507530587 This two day special episode is your best opportunity to meet the newest players to the OpenInfra space and hear about how open source projects, such as OpenStack and Kubernetes, are supporting OpenInfra use cases like hybrid cloud. You will also have the chance to deep dive into the OpenStack user survey results since the launch of the OpenStack User Survey in 2013. Here is a preview of the OpenStack user survey results findings: - Over 300 OpenStack deployments were logged this year, including a significant number of new clouds—in the last 18 months, over 100 new OpenStack clouds have been built, growing the total number of cores under OpenStack management to more than 25,000,000. - Hybrid cloud scenarios continue to be popular, but over half of User Survey respondents indicated that the majority of their cloud infrastructure runs on OpenStack. Upgrades continue to be a challenge that the upstream community tackles with each additional release, but the User Survey shows the majority of organizations are running within the last seven releases. The full report of the OpenStack user survey will be distributed during the OpenInfra Live: Keynotes, so make sure you are registered for the event [2]. Can’t make it to the event? Register anyway, and we will email you a link to the recording after the event! At OpenInfra Live Keynotes, you will also have the opportunity to - interact with leaders of open source projects like OpenStack and Kubernetes to hear how the projects are supporting OpenInfra use cases like hybrid cloud - gain insight into public cloud economics and the role open source technologies play - celebrate as we announce this year’s Superuser Awards winner. This will be the one time everyone will be coming together this year. Come interact with the global OpenInfra community—Live! -Kendall Nelson (diablo_rojo) [1]: https://openinfra.dev/live/keynotes [2]: https://www.eventbrite.com/e/openinfra-live-keynotes-tickets-169507530587

1 0

[cinder] festival of XS reviews 15 October 2021
by Brian Rosmaita 13 Oct '21

13 Oct '21

Hello Cinder community members, This is a reminder that the most recent edition of the Cinder Festival of XS Reviews will be held at the end of this week on Friday 15 October. who: Everyone! what: The Cinder Festival of XS Reviews when: Friday 15 October 2021 from 1400-1600 UTC where: https://meetpad.opendev.org/cinder-festival-of-reviews This recurring meeting can be placed on your calendar by using this handy ICS file: http://eavesdrop.openstack.org/calendars/cinder-festival-of-reviews.ics See you there! brian

1 0

[nova][placement] No meeting next week
by Sylvain Bauza 13 Oct '21

13 Oct '21

As agreed during yesterday's meeting [1], Tuesday Oct 19th's meeting is *CANCELLED* as all of us will be attending the virtual PTG. I'm more than happy tho to see all of you next week thru video ! I'll add the PTG connection details in https://etherpad.opendev.org/p/nova-yoga-ptg -Sylvain [1] https://meetings.opendev.org/meetings/nova/2021/nova.2021-10-12-16.00.log.h…

1 0

[tripleo] Unable to execute pre-introspection and pre-deployment command
by Anirudh Gupta 13 Oct '21

13 Oct '21

3 6

[docs] has_project_guide key
by Peter Matulis 13 Oct '21

13 Oct '21

Does the has_project_guide key in the www/project-data/<RELEASE>.yaml file have any meaning? One of my projects has been dragging this key along from release to release and I do not see it documented [1]. I want to avoid unintended results if that key is removed. Thanks. [1]: https://docs.openstack.org/doc-contrib-guide/doc-tools/template-generator.h… Peter

1 0

[cinder] Bug deputy report for week of 10-13-2021
by Sofia Enriquez 13 Oct '21

13 Oct '21

This is a bug report from 10-06-2021-15-09 to 10-13-2021. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Medium - https://bugs.launchpad.net/cinder/+bug/1946736 'Powerflex driver: update supported storage versions'. - https://bugs.launchpad.net/cinder/+bug/1946350 '[gate-failure] nova-live-migration evacuation failure due to slow lvchange -a command in c-vol during volume attachment update'. - https://bugs.launchpad.net/cinder/+bug/1946340 '[gate-failure] Unable to stack on fedora34 due to cinder pulling in oslo.vmware and suds-jurko that uses use_2to3 that is invalid with setuptools >=58.0.0'. - https://bugs.launchpad.net/cinder/+bug/1946263 'NetApp ONTAP Failing migrating volume from/to FlexGroup pool'. Low - https://bugs.launchpad.net/cinder/+bug/1946618 'Add same volume to the group-update does not show proper error'. Wishlist - https://bugs.launchpad.net/cinder/+bug/1946645 '[doc] Install and configure a storage node in cinder'. Cheers -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[KEYSTONE][POLICIES] - Overrides that don't work?
by Gaël THEROND 13 Oct '21

13 Oct '21

Hi team, I'm having a weird behavior with my Openstack platform that makes me think I may have misunderstood some mechanisms on the way policies are working and especially the overriding. So, long story short, I've few services that get custom policies such as glance that behave as expected, Keystone's one aren't. All in all, here is what I'm understanding of the mechanism: This is the keystone policy that I'm looking to override: https://paste.openstack.org/show/bwuF6jFISscRllWdUURL/ This policy default can be found in here: https://opendev.org/openstack/keystone/src/branch/master/keystone/common/po… Here is the policy that I'm testing: https://paste.openstack.org/show/bHQ0PXvOro4lXNTlxlie/ I know, this policy isn't taking care of the admin role but it's not the point. >From my understanding, any user with the project-manager role should be able to add any available user on any available group as long as the project-manager domain is the same as the target. However, when I'm doing that, keystone complains that I'm not authorized to do so because the user token scope is 'PROJECT' where it should be 'SYSTEM' or 'DOMAIN'. Now, I wouldn't be surprised of that message being thrown out with the default policy as it's stated on the code with the following: https://opendev.org/openstack/keystone/src/branch/stable/ussuri/keystone/co… So the question is, if the custom policy doesn't override the default scope_types how am I supposed to make it work? I hope it was clear enough, but if not, feel free to ask me for more information. PS: I've tried to assign this role with a domain scope to my user and I've still the same issue. Thanks a lot everyone!

2 4

[kolla-ansible][neutron] configuration question
by Ignazio Cassano 13 Oct '21

13 Oct '21

Hello Everyone, I need to know if it possible configure kolla neutron ovs with more than one bridge mappings, for example: bridge_mappings = physnet1:br-ex,physnet2:br-ex2 I figure out that in standard configuration ansible playbook create br-ex and add the interface with variable "neutron_external_interface" under br-ex. What can I do if I need to do if I wand more than one bridge ? How kolla ansible playbook can help in this case ? I could use multiple bridges in /etc/kolla/config neutron configuration files, but I do not know how ansible playbook can do the job. because I do not see any variable can help me in /etc/kolla/globals.yml Thanks Ignazio

2 3