October 2021 - openstack-discuss - lists.openstack.org

[neutron][PTG] Schedule for yoga PTG
by Lajos Katona 14 Oct '21

14 Oct '21

Hi, I made a first schedule for the PTG next week, please check it: https://etherpad.opendev.org/p/neutron-yoga-ptg If you have anything to change or add please raise your voice :-) There's still some parts which can change, as we will have multiple cross-project sessions. See you next week. Lajos Katona (lajoskatona)

1 0

[neutron] Team meeting
by Lajos Katona 14 Oct '21

14 Oct '21

Hi Neutrinos, As we have PTG next week, let's cancel the team meeting. Cheers Lajos Katona (lajoskatona)

1 0

[neutron] CI meeting Tuesday 19.10
by Slawek Kaplonski 14 Oct '21

14 Oct '21

Hi, As we have PTG next week, let's cancel CI meeting. See You all at the PTG sessions and on the CI meeting on Tuesday 26th of October. -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[cinder][backup] backup big volumes leads to oom kill of cinder-backup
by Stefan Hoffmann 14 Oct '21

14 Oct '21

Hi cinder team, we have the issue, that doing backups of big volumes (5TB) fails and cinder-backup get oom killed. Looks like cinder-backup is allocating memory but didn't release it correctly. Badly we are still using cinder queens. Is this a known issue and fixed in newer releases or should I create a bug report? We found a similar bug [1] with backup restore, that got fixed. I guess something like this is also needed for backup create. Thanks for you help Stefan [1] https://bugs.launchpad.net/cinder/+bug/1865011 -- Stefan Hoffmann DevOps-Engineer Cloud&Heat Technologies GmbH Königsbrücker Straße 96 (Halle 15) | 01099 Dresden +49 351 479 367 36 stefan.hoffmann(a)cloudandheat.com | www.cloudandheat.com Die grüne Cloud für KI und ML. Think Green: Mach Deine Anwendung grüner. https://thinkgreen.cloudandheat.com/ Commercial Register: District Court Dresden Register Number: HRB 30549 VAT ID No.: DE281093504 Managing Director: Nicolas Röhrs Authorized signatory: Dr. Marius Feldmann Authorized signatory: Kristina Rübenkamp

1 0

Networks in openstack
by Midhunlal Nb 14 Oct '21

14 Oct '21

Hi Team, I have some doubt in different network types Vlan,Vxlan and flat networks. How these networks helps in openstack.What is the use of each network? Could you Please provide me a detailed answer or suggest me any document regarding this networks. Show quoted text

3 3

[kayobe][kolla][ironic] kayobe overcloud provision fails because ironic compute hosts use their inspection DHCP pool IPs
by Manuel Holtgrewe 14 Oct '21

14 Oct '21

Dear list, I am experimenting with kayobe to deploy a test installation of OpenStack wallaby. You can find my configuration here: https://github.com/openstack/kayobe-config/compare/stable/wallaby...holtgre… I am following the kayobe documentation and have successfully setup a controller and a seed node. I am at the point where I have the nodes configured and they show up in bifrost baremetal node list. I can control them via IPMI/iDRAC/RedFish and boot them into the IPA image and the nodes can be inspected and actually go into the "manageable" status. kayobe is capable of using the inspection results and assigning the root device, so far, so good. I don't know whether my network configuration is good. I want to pin the IPs of stack-1 to stack-4 and the names resolve the correct IP addresses throughout my network. Below are some more details. In summary, I have trouble because `kayobe overcloud provision` makes my 4 overcloud bare metal host boot into IPA with DHCP enabled and they get the same IPs assigned that were given to them earlier in inspection. This means that the overcloud provision command cannot SSH into the nodes because it knows them by the wrong IPs. I must be really missing something here. What is it? Below are more details. Here is what kayobe pulled from the bifrost inspection (I believe). # cat etc/kayobe/inventory/overcloud [controllers] stack-1 ipmi_address=172.16.66.41 bmc_type=idrac stack-2 ipmi_address=172.16.66.42 bmc_type=idrac stack-3 ipmi_address=172.16.66.43 bmc_type=idrac stack-4 ipmi_address=172.16.66.44 bmc_type=idrac The IPs are also fixed here # etc/kayobe/network-allocation.yml compute_net_ips: stack-1: 172.16.32.11 stack-2: 172.16.32.12 stack-3: 172.16.32.13 stack-4: 172.16.32.14 stack-seed: 172.16.32.6 However, I thought I had to provide allocation ranges for DHCP for getting introspection to work. Thus, I have the following # etc/kayobe/networks.yml compute_net_cidr: 172.16.32.0/19 compute_net_gateway: 172.16.32.1 compute_net_vip_address: 172.16.32.2 compute_net_allocation_pool_start: 172.16.32.101 compute_net_allocation_pool_end: 172.16.32.200 compute_net_inspection_allocation_pool_start: 172.16.32.201 compute_net_inspection_allocation_pool_end: 172.16.32.250 This leads to the following dnsmasq leases in the bifrost host. # cat /var/lib/dnsmasq/dnsmasq.leases 1634187260 REDACTED 172.16.32.215 * REDACTED 1634187271 REDACTED 172.16.32.243 * REDACTED 1634187257 REDACTED 172.16.32.207 * REDACTED 1634187258 REDACTED 172.16.32.218 * REDACTED What am I missing? Best wishes, Manuel

2 1

[tripleo] Unable to deploy Overcloud Nodes
by Anirudh Gupta 14 Oct '21

14 Oct '21

Hi Team, As per the link below, While executing the command to deploy the overcloud nodes https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/deploym… I am executing the command - openstack overcloud deploy --templates On running this command, I am getting the following error *ERROR: (pymysql.err.OperationalError) (1045, "Access denied for user 'heat'@'10.255.255.4' (using password: YES)")(Background on this error at: http://sqlalche.me/e/e3q8 <http://sqlalche.me/e/e3q8>)* 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: subprocess.CalledProcessError: Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 1. 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last): 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args) 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args) 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1277, in take_action 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.setup_ephemeral_heat(parsed_args) 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 767, in setup_ephemeral_heat 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud utils.launch_heat(self.heat_launcher, restore_db=restore_db) 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 2706, in launch_heat 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud launcher.heat_db_sync(restore_db) 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/heat_launcher.py", line 530, in heat_db_sync 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud subprocess.check_call(cmd) 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise CalledProcessError(retcode, cmd) 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud subprocess.CalledProcessError: Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 1. 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 2021-10-13 05:46:28.400 183680 ERROR openstack [-] Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 1.: subprocess.CalledProcessError: Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 1. 2021-10-13 05:46:28.401 183680 INFO osc_lib.shell [-] END return value: 1 Can someone please help in resolving this issue. Are there any parameters, templates that need to be passed in order to make it work. Regards Anirudh Gupta

2 1

[tc] [ptl] 2021 User Survey Project Specific Feedback Responses
by helena＠openstack.org 13 Oct '21

13 Oct '21

4 3

[all][tc] Technical Committee next weekly meeting on Oct 14th at 1500 UTC
by Ghanshyam Mann 13 Oct '21

13 Oct '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for Oct 14th at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Oct 13th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

[Designate] After rebuilding Queens clusters on Train, race condition causes Designate record creation to fail
by Braden, Albert 13 Oct '21

13 Oct '21

After enabling redis, and allowing TCP 6379 and 26379, I see it in /etc/designate/designate.conf in the designate_producer container: backend_url = redis://admin:<key>@10.221.176.48:26379?sentinel=kolla&sentinel_fallback=10.221.176.173:26379&sentinel_fallback=10.221.177.38:26379&db=0&socket_timeout=60&retry_on_timeout=yes And I can get to ports 6379 and 26379 with nc: (designate-producer)[root@dva3-ctrl3 /]# nc 10.221.176.173 26379 / -ERR unknown command `/`, with args beginning with: But I still see the DB error when TF rebuilds a VM: 2021-10-13 15:35:23.941 26 ERROR oslo_messaging.notify.dispatcher designate.exceptions.DuplicateRecord: Duplicate Record What am I missing? -----Original Message----- From: Michael Johnson <johnsomor(a)gmail.com> Sent: Tuesday, October 12, 2021 11:33 AM To: Braden, Albert <abraden(a)verisign.com> Cc: openstack-discuss(a)lists.openstack.org Subject: [EXTERNAL] Re: Re: Re: After rebuilding Queens clusters on Train, race condition causes Designate record creation to fail Caution: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. I don't have a good answer for you on that as it pre-dates my history with Designate a bit. I suspect it has to do with the removal of the pool-manager and the restructuring of the controller code. Maybe someone else on the discuss list has more insight. Michael On Tue, Oct 12, 2021 at 5:47 AM Braden, Albert <abraden(a)verisign.com> wrote: > > Thank you Michael, this is very helpful. Do you have any insight into why we don't experience this in Queens clusters? We aren't running a lock manager there either, and I haven't been able to duplicate the problem there. > > -----Original Message----- > From: Michael Johnson <johnsomor(a)gmail.com> > Sent: Monday, October 11, 2021 4:24 PM > To: Braden, Albert <abraden(a)verisign.com> > Cc: openstack-discuss(a)lists.openstack.org > Subject: [EXTERNAL] Re: Re: After rebuilding Queens clusters on Train, race condition causes Designate record creation to fail > > Caution: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. > > You will need one of the Tooz supported distributed lock managers: > Consul, Memcacded, Redis, or zookeeper. > > Michael > > On Mon, Oct 11, 2021 at 11:57 AM Braden, Albert <abraden(a)verisign.com> wrote: > > > > After investigating further, I realized that we're not running redis, and I think that means that redis_connection_string doesn't get set. Does this mean that we must run redis, or is there a workaround? > > > > -----Original Message----- > > From: Braden, Albert > > Sent: Monday, October 11, 2021 2:48 PM > > To: 'johnsomor(a)gmail.com' <johnsomor(a)gmail.com> > > Cc: 'openstack-discuss(a)lists.openstack.org' <openstack-discuss(a)lists.openstack.org> > > Subject: RE: [EXTERNAL] Re: After rebuilding Queens clusters on Train, race condition causes Designate record creation to fail > > > > I think so. I see this: > > > > ansible/roles/designate/templates/designate.conf.j2:backend_url = {{ redis_connection_string }} > > > > ansible/group_vars/all.yml:redis_connection_string: "redis://{% for host in groups['redis'] %}{% if host == groups['redis'][0] %}admin:{{ redis_master_password }}@{{ 'api' | kolla_address(host) | put_address_in_context('url') }}:{{ redis_sentinel_port }}?sentinel=kolla{% else %}&sentinel_fallback={{ 'api' | kolla_address(host) | put_address_in_context('url') }}:{{ redis_sentinel_port }}{% endif %}{% endfor %}&db=0&socket_timeout=60&retry_on_timeout=yes" > > > > Did anything with the distributed lock manager between Queens and Train? > > > > -----Original Message----- > > From: Michael Johnson <johnsomor(a)gmail.com> > > Sent: Monday, October 11, 2021 1:15 PM > > To: Braden, Albert <abraden(a)verisign.com> > > Cc: openstack-discuss(a)lists.openstack.org > > Subject: [EXTERNAL] Re: After rebuilding Queens clusters on Train, race condition causes Designate record creation to fail > > > > Caution: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. > > > > Hi Albert, > > > > Have you configured your distributed lock manager for Designate? > > > > [coordination] > > backend_url = <DLM URL> > > > > Michael > > > > On Fri, Oct 8, 2021 at 7:38 PM Braden, Albert <abraden(a)verisign.com> wrote: > > > > > > Hello everyone. It’s great to be back working on OpenStack again. I’m at Verisign now. I can hardly describe how happy I am to have an employer that does not attach nonsense to the bottom of my emails! > > > > > > > > > > > > We are rebuilding our clusters from Queens to Train. On the new Train clusters, customers are complaining that deleting a VM and then immediately creating a new one with the same name (via Terraform for example) intermittently results in a missing DNS record. We can duplicate the issue by building a VM with terraform, tainting it, and applying. > > > > > > > > > > > > Before applying the change, we see the DNS record in the recordset: > > > > > > > > > > > > $ openstack recordset list dva3.vrsn.com. --all |grep openstack-terra > > > > > > | f9aa73c1-84ba-4854-be71-cbb616de672c | 8d1c84082a044a53abe0d519ed9e8c60 | openstack-terra-test-host.dev-ostck.dva3.vrsn.com. | A | 10.220.4.89 | ACTIVE | NONE | > > > > > > $ > > > > > > > > > > > > and we can pull it from the DNS server on the controllers: > > > > > > > > > > > > $ for i in {1..3}; do dig @dva3-ctrl${i}.cloud.vrsn.com -t axfr dva3.vrsn.com. |grep openstack-terra; done > > > > > > openstack-terra-test-host.dev-ostck.dva3.vrsn.com. 1 IN A 10.220.4.89 > > > > > > openstack-terra-test-host.dev-ostck.dva3.vrsn.com. 1 IN A 10.220.4.89 > > > > > > openstack-terra-test-host.dev-ostck.dva3.vrsn.com. 1 IN A 10.220.4.89 > > > > > > > > > > > > After applying the change, we don’t see it: > > > > > > > > > > > > $ openstack recordset list dva3.vrsn.com. --all |grep openstack-terra > > > > > > | f9aa73c1-84ba-4854-be71-cbb616de672c | 8d1c84082a044a53abe0d519ed9e8c60 | openstack-terra-test-host.dev-ostck.dva3.vrsn.com. | A | 10.220.4.89 | ACTIVE | NONE | > > > > > > $ > > > > > > $ for i in {1..3}; do dig @dva3-ctrl${i}.cloud.vrsn.com -t axfr dva3.vrsn.com. |grep openstack-terra; done > > > > > > $ openstack recordset list dva3.vrsn.com. --all |grep openstack-terra > > > > > > $ > > > > > > > > > > > > We see this in the logs: > > > > > > > > > > > > 2021-10-09 01:53:44.307 27 ERROR oslo_messaging.notify.dispatcher oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'c70e693b4c47402db088c43a5a177134-openstack-terra-test-host.de...' for key 'unique_recordset'") > > > > > > 2021-10-09 01:53:44.307 27 ERROR oslo_messaging.notify.dispatcher [SQL: INSERT INTO recordsets (id, version, created_at, zone_shard, tenant_id, zone_id, name, type, ttl, reverse_name) VALUES (%(id)s, %(version)s, %(created_at)s, %(zone_shard)s, %(tenant_id)s, %(zone_id)s, %(name)s, %(type)s, %(ttl)s, %(reverse_name)s)] > > > > > > 2021-10-09 01:53:44.307 27 ERROR oslo_messaging.notify.dispatcher [parameters: {'id': 'dbbb904c347241a791aa01ca33a87b23', 'version': 1, 'created_at': datetime.datetime(2021, 10, 9, 1, 53, 44, 182652), 'zone_shard': 3184, 'tenant_id': '8d1c84082a044a53abe0d519ed9e8c60', 'zone_id': 'c70e693b4c47402db088c43a5a177134', 'name': 'openstack-terra-test-host.dev-ostck.dva3.vrsn.com.', 'type': 'A', 'ttl': None, 'reverse_name': '.moc.nsrv.3avd.kctso-ved.tsoh-tset-arret-kcatsnepo'}] > > > > > > > > > > > > It appears that Designate is trying to create the new record before the deletion of the old one finishes. > > > > > > > > > > > > Is anyone else seeing this on Train? The same set of actions doesn’t cause this error in Queens. Do we need to change something in our Designate config, to make it wait until the old records are finished deleting before attempting to create the new ones?

1 0