October 2021 - openstack-discuss - lists.openstack.org

[neutron] Drivers meeting agenda - 15.10.2021
by Lajos Katona 15 Oct '21

15 Oct '21

Hi Neutron Drivers! As we have no quorum last week to decide on https://bugs.launchpad.net/neutron/+bug/1946251 tomorrow we can check it again and vote. The logs of the meeting from last week: https://meetings.opendev.org/meetings/neutron_drivers/2021/neutron_drivers.… (Sorry it is 4 days long as I missed to end the meeting.....) As we have PTG next week, let's cancel drivers meeting for that Friday, and I will be on PTO the week after (29. October) so I can't chair that one. See you online tomorrow. Lajos Katona (lajoskatona)

2 1

Problème with image from snapshot
by Franck VEDEL 14 Oct '21

14 Oct '21

Hello and first sorry for my english… thanks google. Something is wrong with what I want to do: I use Wallaby, it works very well (apart from VpnaaS, I wasted too much time this summer to make it work, without success, and the bug does not seem to be fixed). Here is what I want to do and which does not work as I want: - With an admin account, I launch a Win10 instance from the image I created. The instance is working but it takes about 10 minutes to get Win10 up and running. I wanted to take a snapshot of this instance and then create a new image from this snapshot. And that users use this new image. I create the snapshot, I place the "--public" parameter on the new image. I'm trying to create a new instance from this snapshot with the admin account: it works. I create a new user, who has his project, and sees all the images. I try to create an instance with this new image and I get the message: Block Device Mapping is Invalid: failed to get snapshot f12c04f2-51e7-4817-ab9b-eda63c5b9aff. (HTTP 400) (Request-ID: req-c26dab86-c25f-409a-8390-8aa0ea8fe1cb) Is it a legal problem? Is it possible to do as I do? otherwise how should we do it? Thanks if you have ideas for helping me Franck VEDEL

3 8

[tc] No TC weekly meeting next week due to meeting in PTG
by Ghanshyam Mann 14 Oct '21

14 Oct '21

Hello Everyone, As we will be meeting in PTG next week, we are cancelling the TC's next week (21st Oct) IRC meeting. -gmann

1 0

[dev][cinder] Consultation about new cinder-backup features
by Daniel de Oliveira Pereira 14 Oct '21

14 Oct '21

Hello everyone, We have prototyped some new features on Cinder for our clients, and we think that they are nice features and good candidates to be part of upstream Cinder, so we would like to get feedback from OpenStack community about these features and if you would be willing to accept them in upstream OpenStack. Our team implemented the following features for cinder-backup service: 1. A multi-backend backup driver, that allow OpenStack users to choose, via API/CLI/Horizon, which backup driver (Ceph or NFS, in our prototype) will be used during a backup operation to create a new volume backup. 2. An improved NFS backup driver, that allow OpenStack users to back up their volumes to private NFS servers, providing the NFS hostpath at runtime via API/CLI/Horizon, while creating the volume backup. Considering that cinder was configured to use the multi-backend backup driver, this is how it works: During a volume backup operation, the user provides a "location" parameter to indicate which backend will be used, and the backup hostpath, if applicable (for NFS driver), to create the volume backup. For instance: - Creating a backup using Ceph backend: $ openstack volume backup create --name <backup_name> --location ceph <volume_id> - Creating a backup using the improved NFS backend: $ openstack volume backup create --name <backup_name> --location nfs://my.nfs.server:/backups <volume_id> If the user chooses Ceph backend, the Ceph driver will be used to create the backup. If the user chooses the NFS backend, the improved NFS driver, previously mentioned, will be used to create the backup. The backup location, if provided, is stored on Cinder database, and can be seen fetching the backup details: $ openstack volume backup show <backup_name> Briefly, this is how the features were implemented: - Cinder API was updated to add an optional location parameter to "create backup" method. Horizon, and OpenStack and Cinder CLIs were updated accordingly, to handle the new parameter. - Cinder backup controller was updated to handle the backup location parameter, and a validator for the parameter was implemented using the oslo config library. - Cinder backup object model was updated to add a nullable location property, so that the backup location could be stored on cinder database. - a new backup driver base class, that extends BackupDriver and accepts a backup context object, was implemented to handle the backup configuration provided at runtime by the user. This new backup base class requires that the concrete drivers implement a method to validate the backup context (similar to BackupDriver.check_for_setup_error) - the 2 new backup drivers, previously mentioned, were implemented using these new backup base class. - in BackupManager class, the "service" attribute, that on upstream OpenStack holds the backup driver class name, was re-implemented as a factory function that accepts a backup context object and return an instance of a backup driver, according to the backup driver configured on cinder.conf file and the backup context provided at runtime by the user. - All the backup operations continue working as usual. Could you please let us know your thoughts about these features and if you would be open to adding them to upstream Cinder? If yes, we would be willing to submit the specs and work on the upstream implementation, if they are approved. Regards, Daniel Pereira

5 6

[manila][ptg] No IRC meetings on 21st and 28th Oct 2021
by Goutham Pacha Ravi 14 Oct '21

14 Oct '21

Hi Zorillas, Since we'll be at the PTG [1], we'll skip the IRC meeting [2] on the 21st of Oct; and since a number of us may be taking some time off the week after, we'll skip that weekly occurrence (28th Oct) as well. Please feel free to grab attention to any issue via this mailing list, or hop over to #openstack-manila on OFTC. Our next weekly IRC meeting will be on 4th Nov 2021. Thanks, and see you at the PTG! Goutham [1] https://etherpad.opendev.org/p/yoga-ptg-manila-planning [2] https://wiki.openstack.org/wiki/Manila/Meetings

1 0

[neutron][PTG] Schedule for yoga PTG
by Lajos Katona 14 Oct '21

14 Oct '21

Hi, I made a first schedule for the PTG next week, please check it: https://etherpad.opendev.org/p/neutron-yoga-ptg If you have anything to change or add please raise your voice :-) There's still some parts which can change, as we will have multiple cross-project sessions. See you next week. Lajos Katona (lajoskatona)

1 0

[neutron] Team meeting
by Lajos Katona 14 Oct '21

14 Oct '21

Hi Neutrinos, As we have PTG next week, let's cancel the team meeting. Cheers Lajos Katona (lajoskatona)

1 0

[neutron] CI meeting Tuesday 19.10
by Slawek Kaplonski 14 Oct '21

14 Oct '21

Hi, As we have PTG next week, let's cancel CI meeting. See You all at the PTG sessions and on the CI meeting on Tuesday 26th of October. -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[cinder][backup] backup big volumes leads to oom kill of cinder-backup
by Stefan Hoffmann 14 Oct '21

14 Oct '21

Hi cinder team, we have the issue, that doing backups of big volumes (5TB) fails and cinder-backup get oom killed. Looks like cinder-backup is allocating memory but didn't release it correctly. Badly we are still using cinder queens. Is this a known issue and fixed in newer releases or should I create a bug report? We found a similar bug [1] with backup restore, that got fixed. I guess something like this is also needed for backup create. Thanks for you help Stefan [1] https://bugs.launchpad.net/cinder/+bug/1865011 -- Stefan Hoffmann DevOps-Engineer Cloud&Heat Technologies GmbH Königsbrücker Straße 96 (Halle 15) | 01099 Dresden +49 351 479 367 36 stefan.hoffmann(a)cloudandheat.com | www.cloudandheat.com Die grüne Cloud für KI und ML. Think Green: Mach Deine Anwendung grüner. https://thinkgreen.cloudandheat.com/ Commercial Register: District Court Dresden Register Number: HRB 30549 VAT ID No.: DE281093504 Managing Director: Nicolas Röhrs Authorized signatory: Dr. Marius Feldmann Authorized signatory: Kristina Rübenkamp

1 0

Networks in openstack
by Midhunlal Nb 14 Oct '21

14 Oct '21

Hi Team, I have some doubt in different network types Vlan,Vxlan and flat networks. How these networks helps in openstack.What is the use of each network? Could you Please provide me a detailed answer or suggest me any document regarding this networks. Show quoted text

3 3

[kayobe][kolla][ironic] kayobe overcloud provision fails because ironic compute hosts use their inspection DHCP pool IPs
by Manuel Holtgrewe 14 Oct '21

14 Oct '21

Dear list, I am experimenting with kayobe to deploy a test installation of OpenStack wallaby. You can find my configuration here: https://github.com/openstack/kayobe-config/compare/stable/wallaby...holtgre… I am following the kayobe documentation and have successfully setup a controller and a seed node. I am at the point where I have the nodes configured and they show up in bifrost baremetal node list. I can control them via IPMI/iDRAC/RedFish and boot them into the IPA image and the nodes can be inspected and actually go into the "manageable" status. kayobe is capable of using the inspection results and assigning the root device, so far, so good. I don't know whether my network configuration is good. I want to pin the IPs of stack-1 to stack-4 and the names resolve the correct IP addresses throughout my network. Below are some more details. In summary, I have trouble because `kayobe overcloud provision` makes my 4 overcloud bare metal host boot into IPA with DHCP enabled and they get the same IPs assigned that were given to them earlier in inspection. This means that the overcloud provision command cannot SSH into the nodes because it knows them by the wrong IPs. I must be really missing something here. What is it? Below are more details. Here is what kayobe pulled from the bifrost inspection (I believe). # cat etc/kayobe/inventory/overcloud [controllers] stack-1 ipmi_address=172.16.66.41 bmc_type=idrac stack-2 ipmi_address=172.16.66.42 bmc_type=idrac stack-3 ipmi_address=172.16.66.43 bmc_type=idrac stack-4 ipmi_address=172.16.66.44 bmc_type=idrac The IPs are also fixed here # etc/kayobe/network-allocation.yml compute_net_ips: stack-1: 172.16.32.11 stack-2: 172.16.32.12 stack-3: 172.16.32.13 stack-4: 172.16.32.14 stack-seed: 172.16.32.6 However, I thought I had to provide allocation ranges for DHCP for getting introspection to work. Thus, I have the following # etc/kayobe/networks.yml compute_net_cidr: 172.16.32.0/19 compute_net_gateway: 172.16.32.1 compute_net_vip_address: 172.16.32.2 compute_net_allocation_pool_start: 172.16.32.101 compute_net_allocation_pool_end: 172.16.32.200 compute_net_inspection_allocation_pool_start: 172.16.32.201 compute_net_inspection_allocation_pool_end: 172.16.32.250 This leads to the following dnsmasq leases in the bifrost host. # cat /var/lib/dnsmasq/dnsmasq.leases 1634187260 REDACTED 172.16.32.215 * REDACTED 1634187271 REDACTED 172.16.32.243 * REDACTED 1634187257 REDACTED 172.16.32.207 * REDACTED 1634187258 REDACTED 172.16.32.218 * REDACTED What am I missing? Best wishes, Manuel

2 1

[tripleo] Unable to deploy Overcloud Nodes
by Anirudh Gupta 14 Oct '21

14 Oct '21

Hi Team, As per the link below, While executing the command to deploy the overcloud nodes https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/deploym… I am executing the command - openstack overcloud deploy --templates On running this command, I am getting the following error *ERROR: (pymysql.err.OperationalError) (1045, "Access denied for user 'heat'@'10.255.255.4' (using password: YES)")(Background on this error at: http://sqlalche.me/e/e3q8 <http://sqlalche.me/e/e3q8>)* 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: subprocess.CalledProcessError: Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 1. 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last): 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args) 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args) 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1277, in take_action 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.setup_ephemeral_heat(parsed_args) 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 767, in setup_ephemeral_heat 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud utils.launch_heat(self.heat_launcher, restore_db=restore_db) 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 2706, in launch_heat 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud launcher.heat_db_sync(restore_db) 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/heat_launcher.py", line 530, in heat_db_sync 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud subprocess.check_call(cmd) 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise CalledProcessError(retcode, cmd) 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud subprocess.CalledProcessError: Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 1. 2021-10-13 05:46:28.390 183680 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 2021-10-13 05:46:28.400 183680 ERROR openstack [-] Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 1.: subprocess.CalledProcessError: Command '['sudo', 'podman', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overcloud/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 1. 2021-10-13 05:46:28.401 183680 INFO osc_lib.shell [-] END return value: 1 Can someone please help in resolving this issue. Are there any parameters, templates that need to be passed in order to make it work. Regards Anirudh Gupta

2 1

[tc] [ptl] 2021 User Survey Project Specific Feedback Responses
by helena＠openstack.org 13 Oct '21

13 Oct '21

4 3

[all][tc] Technical Committee next weekly meeting on Oct 14th at 1500 UTC
by Ghanshyam Mann 13 Oct '21

13 Oct '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for Oct 14th at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Oct 13th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

[Designate] After rebuilding Queens clusters on Train, race condition causes Designate record creation to fail
by Braden, Albert 13 Oct '21

13 Oct '21

After enabling redis, and allowing TCP 6379 and 26379, I see it in /etc/designate/designate.conf in the designate_producer container: backend_url = redis://admin:<key>@10.221.176.48:26379?sentinel=kolla&sentinel_fallback=10.221.176.173:26379&sentinel_fallback=10.221.177.38:26379&db=0&socket_timeout=60&retry_on_timeout=yes And I can get to ports 6379 and 26379 with nc: (designate-producer)[root@dva3-ctrl3 /]# nc 10.221.176.173 26379 / -ERR unknown command `/`, with args beginning with: But I still see the DB error when TF rebuilds a VM: 2021-10-13 15:35:23.941 26 ERROR oslo_messaging.notify.dispatcher designate.exceptions.DuplicateRecord: Duplicate Record What am I missing? -----Original Message----- From: Michael Johnson <johnsomor(a)gmail.com> Sent: Tuesday, October 12, 2021 11:33 AM To: Braden, Albert <abraden(a)verisign.com> Cc: openstack-discuss(a)lists.openstack.org Subject: [EXTERNAL] Re: Re: Re: After rebuilding Queens clusters on Train, race condition causes Designate record creation to fail Caution: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. I don't have a good answer for you on that as it pre-dates my history with Designate a bit. I suspect it has to do with the removal of the pool-manager and the restructuring of the controller code. Maybe someone else on the discuss list has more insight. Michael On Tue, Oct 12, 2021 at 5:47 AM Braden, Albert <abraden(a)verisign.com> wrote: > > Thank you Michael, this is very helpful. Do you have any insight into why we don't experience this in Queens clusters? We aren't running a lock manager there either, and I haven't been able to duplicate the problem there. > > -----Original Message----- > From: Michael Johnson <johnsomor(a)gmail.com> > Sent: Monday, October 11, 2021 4:24 PM > To: Braden, Albert <abraden(a)verisign.com> > Cc: openstack-discuss(a)lists.openstack.org > Subject: [EXTERNAL] Re: Re: After rebuilding Queens clusters on Train, race condition causes Designate record creation to fail > > Caution: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. > > You will need one of the Tooz supported distributed lock managers: > Consul, Memcacded, Redis, or zookeeper. > > Michael > > On Mon, Oct 11, 2021 at 11:57 AM Braden, Albert <abraden(a)verisign.com> wrote: > > > > After investigating further, I realized that we're not running redis, and I think that means that redis_connection_string doesn't get set. Does this mean that we must run redis, or is there a workaround? > > > > -----Original Message----- > > From: Braden, Albert > > Sent: Monday, October 11, 2021 2:48 PM > > To: 'johnsomor(a)gmail.com' <johnsomor(a)gmail.com> > > Cc: 'openstack-discuss(a)lists.openstack.org' <openstack-discuss(a)lists.openstack.org> > > Subject: RE: [EXTERNAL] Re: After rebuilding Queens clusters on Train, race condition causes Designate record creation to fail > > > > I think so. I see this: > > > > ansible/roles/designate/templates/designate.conf.j2:backend_url = {{ redis_connection_string }} > > > > ansible/group_vars/all.yml:redis_connection_string: "redis://{% for host in groups['redis'] %}{% if host == groups['redis'][0] %}admin:{{ redis_master_password }}@{{ 'api' | kolla_address(host) | put_address_in_context('url') }}:{{ redis_sentinel_port }}?sentinel=kolla{% else %}&sentinel_fallback={{ 'api' | kolla_address(host) | put_address_in_context('url') }}:{{ redis_sentinel_port }}{% endif %}{% endfor %}&db=0&socket_timeout=60&retry_on_timeout=yes" > > > > Did anything with the distributed lock manager between Queens and Train? > > > > -----Original Message----- > > From: Michael Johnson <johnsomor(a)gmail.com> > > Sent: Monday, October 11, 2021 1:15 PM > > To: Braden, Albert <abraden(a)verisign.com> > > Cc: openstack-discuss(a)lists.openstack.org > > Subject: [EXTERNAL] Re: After rebuilding Queens clusters on Train, race condition causes Designate record creation to fail > > > > Caution: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. > > > > Hi Albert, > > > > Have you configured your distributed lock manager for Designate? > > > > [coordination] > > backend_url = <DLM URL> > > > > Michael > > > > On Fri, Oct 8, 2021 at 7:38 PM Braden, Albert <abraden(a)verisign.com> wrote: > > > > > > Hello everyone. It’s great to be back working on OpenStack again. I’m at Verisign now. I can hardly describe how happy I am to have an employer that does not attach nonsense to the bottom of my emails! > > > > > > > > > > > > We are rebuilding our clusters from Queens to Train. On the new Train clusters, customers are complaining that deleting a VM and then immediately creating a new one with the same name (via Terraform for example) intermittently results in a missing DNS record. We can duplicate the issue by building a VM with terraform, tainting it, and applying. > > > > > > > > > > > > Before applying the change, we see the DNS record in the recordset: > > > > > > > > > > > > $ openstack recordset list dva3.vrsn.com. --all |grep openstack-terra > > > > > > | f9aa73c1-84ba-4854-be71-cbb616de672c | 8d1c84082a044a53abe0d519ed9e8c60 | openstack-terra-test-host.dev-ostck.dva3.vrsn.com. | A | 10.220.4.89 | ACTIVE | NONE | > > > > > > $ > > > > > > > > > > > > and we can pull it from the DNS server on the controllers: > > > > > > > > > > > > $ for i in {1..3}; do dig @dva3-ctrl${i}.cloud.vrsn.com -t axfr dva3.vrsn.com. |grep openstack-terra; done > > > > > > openstack-terra-test-host.dev-ostck.dva3.vrsn.com. 1 IN A 10.220.4.89 > > > > > > openstack-terra-test-host.dev-ostck.dva3.vrsn.com. 1 IN A 10.220.4.89 > > > > > > openstack-terra-test-host.dev-ostck.dva3.vrsn.com. 1 IN A 10.220.4.89 > > > > > > > > > > > > After applying the change, we don’t see it: > > > > > > > > > > > > $ openstack recordset list dva3.vrsn.com. --all |grep openstack-terra > > > > > > | f9aa73c1-84ba-4854-be71-cbb616de672c | 8d1c84082a044a53abe0d519ed9e8c60 | openstack-terra-test-host.dev-ostck.dva3.vrsn.com. | A | 10.220.4.89 | ACTIVE | NONE | > > > > > > $ > > > > > > $ for i in {1..3}; do dig @dva3-ctrl${i}.cloud.vrsn.com -t axfr dva3.vrsn.com. |grep openstack-terra; done > > > > > > $ openstack recordset list dva3.vrsn.com. --all |grep openstack-terra > > > > > > $ > > > > > > > > > > > > We see this in the logs: > > > > > > > > > > > > 2021-10-09 01:53:44.307 27 ERROR oslo_messaging.notify.dispatcher oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'c70e693b4c47402db088c43a5a177134-openstack-terra-test-host.de...' for key 'unique_recordset'") > > > > > > 2021-10-09 01:53:44.307 27 ERROR oslo_messaging.notify.dispatcher [SQL: INSERT INTO recordsets (id, version, created_at, zone_shard, tenant_id, zone_id, name, type, ttl, reverse_name) VALUES (%(id)s, %(version)s, %(created_at)s, %(zone_shard)s, %(tenant_id)s, %(zone_id)s, %(name)s, %(type)s, %(ttl)s, %(reverse_name)s)] > > > > > > 2021-10-09 01:53:44.307 27 ERROR oslo_messaging.notify.dispatcher [parameters: {'id': 'dbbb904c347241a791aa01ca33a87b23', 'version': 1, 'created_at': datetime.datetime(2021, 10, 9, 1, 53, 44, 182652), 'zone_shard': 3184, 'tenant_id': '8d1c84082a044a53abe0d519ed9e8c60', 'zone_id': 'c70e693b4c47402db088c43a5a177134', 'name': 'openstack-terra-test-host.dev-ostck.dva3.vrsn.com.', 'type': 'A', 'ttl': None, 'reverse_name': '.moc.nsrv.3avd.kctso-ved.tsoh-tset-arret-kcatsnepo'}] > > > > > > > > > > > > It appears that Designate is trying to create the new record before the deletion of the old one finishes. > > > > > > > > > > > > Is anyone else seeing this on Train? The same set of actions doesn’t cause this error in Queens. Do we need to change something in our Designate config, to make it wait until the old records are finished deleting before attempting to create the new ones?

1 0

Open stack Ansible 23.1.0.dev35 VM launching errors
by Midhunlal Nb 13 Oct '21

13 Oct '21

Hi team, -->Successfully I installed Openstack ansible 23.1.0.dev35. --->I logged in to horizon and created a new network and launched a vm but I am getting an error. Error: Failed to perform requested operation on instance "hope", the instance has an error status: Please try again later [Error: Build of instance b6f42229-06d6-4365-8a20-07df869f1610 aborted: Failed to allocate the network(s), not rescheduling.]. -->Then I checked log | fault | {'code': 500, 'created': '2021-10-08T12:26:44Z', 'message': 'Build of instance b6f42229-06d6-4365-8a20-07df869f1610 aborted: Failed to allocate the network(s), not rescheduling.', 'details': 'Traceback (most recent call last):\n File "/openstack/venvs/nova-23.1.0.dev42/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7235, in _create_guest_with_network\n post_xml_callback=post_xml_callback)\n File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__\n next(self.gen)\n File "/openstack/venvs/nova-23.1.0.dev42/lib/python3.6/site-packages/nova/compute/manager.py", line 479, in wait_for_instance_event\n actual_event = event.wait()\n File "/openstack/venvs/nova-23.1.0.dev42/lib/python3.6/site-packages/eventlet/event.py", line 125, in wait\n result = hub.switch()\n File "/openstack/venvs/nova-23.1.0.dev42/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 313, in switch\n return self.greenlet.switch()\neventlet.timeout.Timeout: 300 seconds\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/openstack/venvs/nova-23.1.0.dev42/lib/python3.6/site-packages/nova/compute/manager.py", line 2397, in _build_and_run_instance\n accel_info=accel_info)\n File "/openstack/venvs/nova-23.1.0.dev42/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 4200, in spawn\n cleanup_instance_disks=created_disks)\n File "/openstack/venvs/nova-23.1.0.dev42/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7258, in _create_guest_with_network\n raise exception.VirtualInterfaceCreateException()\nnova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/openstack/venvs/nova-23.1.0.dev42/lib/python3.6/site-packages/nova/compute/manager.py", line 2219, in _do_build_and_run_instance\n filter_properties, request_spec, accel_uuids)\n File "/openstack/venvs/nova-23.1.0.dev42/lib/python3.6/site-packages/nova/compute/manager.py", line 2458, in _build_and_run_instance\n reason=msg)\nnova.exception.BuildAbortException: Build of instance b6f42229-06d6-4365-8a20-07df869f1610 aborted: Failed to allocate the network(s), not rescheduling.\n'} | Please help me with this error. Thanks & Regards Midhunlal N B

3 9

Learn how users are using OpenStack at OpenInfra Live Keynotes
by Kendall Nelson 13 Oct '21

13 Oct '21

Hello Everyone, You might have heard that the OpenInfra Foundation is hosting its largest free virtual event of this year—OpenInfra Live: Keynote OpenInfra Live: Keynotes: https://openinfra.dev/live/keynotes Date and time: November 17-18 (1500-1700 UTC on each day) Register for free: https://www.eventbrite.com/e/openinfra-live-keynotes-tickets-169507530587 This two day special episode is your best opportunity to meet the newest players to the OpenInfra space and hear about how open source projects, such as OpenStack and Kubernetes, are supporting OpenInfra use cases like hybrid cloud. You will also have the chance to deep dive into the OpenStack user survey results since the launch of the OpenStack User Survey in 2013. Here is a preview of the OpenStack user survey results findings: - Over 300 OpenStack deployments were logged this year, including a significant number of new clouds—in the last 18 months, over 100 new OpenStack clouds have been built, growing the total number of cores under OpenStack management to more than 25,000,000. - Hybrid cloud scenarios continue to be popular, but over half of User Survey respondents indicated that the majority of their cloud infrastructure runs on OpenStack. Upgrades continue to be a challenge that the upstream community tackles with each additional release, but the User Survey shows the majority of organizations are running within the last seven releases. The full report of the OpenStack user survey will be distributed during the OpenInfra Live: Keynotes, so make sure you are registered for the event [2]. Can’t make it to the event? Register anyway, and we will email you a link to the recording after the event! At OpenInfra Live Keynotes, you will also have the opportunity to - interact with leaders of open source projects like OpenStack and Kubernetes to hear how the projects are supporting OpenInfra use cases like hybrid cloud - gain insight into public cloud economics and the role open source technologies play - celebrate as we announce this year’s Superuser Awards winner. This will be the one time everyone will be coming together this year. Come interact with the global OpenInfra community—Live! -Kendall Nelson (diablo_rojo) [1]: https://openinfra.dev/live/keynotes [2]: https://www.eventbrite.com/e/openinfra-live-keynotes-tickets-169507530587

1 0

[cinder] festival of XS reviews 15 October 2021
by Brian Rosmaita 13 Oct '21

13 Oct '21

Hello Cinder community members, This is a reminder that the most recent edition of the Cinder Festival of XS Reviews will be held at the end of this week on Friday 15 October. who: Everyone! what: The Cinder Festival of XS Reviews when: Friday 15 October 2021 from 1400-1600 UTC where: https://meetpad.opendev.org/cinder-festival-of-reviews This recurring meeting can be placed on your calendar by using this handy ICS file: http://eavesdrop.openstack.org/calendars/cinder-festival-of-reviews.ics See you there! brian

1 0

[nova][placement] No meeting next week
by Sylvain Bauza 13 Oct '21

13 Oct '21

As agreed during yesterday's meeting [1], Tuesday Oct 19th's meeting is *CANCELLED* as all of us will be attending the virtual PTG. I'm more than happy tho to see all of you next week thru video ! I'll add the PTG connection details in https://etherpad.opendev.org/p/nova-yoga-ptg -Sylvain [1] https://meetings.opendev.org/meetings/nova/2021/nova.2021-10-12-16.00.log.h…

1 0

[tripleo] Unable to execute pre-introspection and pre-deployment command
by Anirudh Gupta 13 Oct '21

13 Oct '21

3 6

[docs] has_project_guide key
by Peter Matulis 13 Oct '21

13 Oct '21

Does the has_project_guide key in the www/project-data/<RELEASE>.yaml file have any meaning? One of my projects has been dragging this key along from release to release and I do not see it documented [1]. I want to avoid unintended results if that key is removed. Thanks. [1]: https://docs.openstack.org/doc-contrib-guide/doc-tools/template-generator.h… Peter

1 0

[cinder] Bug deputy report for week of 10-13-2021
by Sofia Enriquez 13 Oct '21

13 Oct '21

This is a bug report from 10-06-2021-15-09 to 10-13-2021. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Medium - https://bugs.launchpad.net/cinder/+bug/1946736 'Powerflex driver: update supported storage versions'. - https://bugs.launchpad.net/cinder/+bug/1946350 '[gate-failure] nova-live-migration evacuation failure due to slow lvchange -a command in c-vol during volume attachment update'. - https://bugs.launchpad.net/cinder/+bug/1946340 '[gate-failure] Unable to stack on fedora34 due to cinder pulling in oslo.vmware and suds-jurko that uses use_2to3 that is invalid with setuptools >=58.0.0'. - https://bugs.launchpad.net/cinder/+bug/1946263 'NetApp ONTAP Failing migrating volume from/to FlexGroup pool'. Low - https://bugs.launchpad.net/cinder/+bug/1946618 'Add same volume to the group-update does not show proper error'. Wishlist - https://bugs.launchpad.net/cinder/+bug/1946645 '[doc] Install and configure a storage node in cinder'. Cheers -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[KEYSTONE][POLICIES] - Overrides that don't work?
by Gaël THEROND 13 Oct '21

13 Oct '21

Hi team, I'm having a weird behavior with my Openstack platform that makes me think I may have misunderstood some mechanisms on the way policies are working and especially the overriding. So, long story short, I've few services that get custom policies such as glance that behave as expected, Keystone's one aren't. All in all, here is what I'm understanding of the mechanism: This is the keystone policy that I'm looking to override: https://paste.openstack.org/show/bwuF6jFISscRllWdUURL/ This policy default can be found in here: https://opendev.org/openstack/keystone/src/branch/master/keystone/common/po… Here is the policy that I'm testing: https://paste.openstack.org/show/bHQ0PXvOro4lXNTlxlie/ I know, this policy isn't taking care of the admin role but it's not the point. >From my understanding, any user with the project-manager role should be able to add any available user on any available group as long as the project-manager domain is the same as the target. However, when I'm doing that, keystone complains that I'm not authorized to do so because the user token scope is 'PROJECT' where it should be 'SYSTEM' or 'DOMAIN'. Now, I wouldn't be surprised of that message being thrown out with the default policy as it's stated on the code with the following: https://opendev.org/openstack/keystone/src/branch/stable/ussuri/keystone/co… So the question is, if the custom policy doesn't override the default scope_types how am I supposed to make it work? I hope it was clear enough, but if not, feel free to ask me for more information. PS: I've tried to assign this role with a domain scope to my user and I've still the same issue. Thanks a lot everyone!

2 4

[kolla-ansible][neutron] configuration question
by Ignazio Cassano 13 Oct '21

13 Oct '21

Hello Everyone, I need to know if it possible configure kolla neutron ovs with more than one bridge mappings, for example: bridge_mappings = physnet1:br-ex,physnet2:br-ex2 I figure out that in standard configuration ansible playbook create br-ex and add the interface with variable "neutron_external_interface" under br-ex. What can I do if I need to do if I wand more than one bridge ? How kolla ansible playbook can help in this case ? I could use multiple bridges in /etc/kolla/config neutron configuration files, but I do not know how ansible playbook can do the job. because I do not see any variable can help me in /etc/kolla/globals.yml Thanks Ignazio

2 3

Re: [neutron] OVN and dynamic routing
by Francois 13 Oct '21

13 Oct '21

...forgot to add the mailing list in the reply On Wed, 13 Oct 2021 at 10:01, Francois <rigault.francois(a)gmail.com> wrote: > > On Tue, 12 Oct 2021 at 21:46, Dan Sneddon <dsneddon(a)redhat.com> wrote: > > > > On 10/12/21 07:03, Francois wrote: > > > Hello Neutron! > > > I am looking into running stacks with OVN on a leaf-spine network, and > > > have some floating IPs routed between racks. > > > > > > Basically each rack is assigned its own set of subnets. > > > Some VLANs are stretched across all racks: the provisioning VLAN used > > > by tripleo to deploy the stack, and the VLANs for the controllers API > > > IPs. However, each tenant subnet is local to a rack: for example each > > > OVN chassis has a ovn-encap-ip with an IP of the tenant subnet of its > > > own rack. Traffic between 2 racks is sent to a spine, and leaves and > > > spines run some eVPN-like thing: each pair of ToR is a vtep, traffic > > > is encapsulated as VXLAN, and routes between vteps are exchanged with > > > BGP. > > > > > > > There has been a lot of work put into TripleO to allow you to provision > > hosts across L3 boundaries using DHCP relay. You can create a routed > > provisioning network using "helper-address" or vendor-specific commands > > on your top-of-rack switches, and a different subnet and DHCP address > > pool per rack. > Yes I saw that in the doc. I was not planning on using this for reasons I mentioned in another reply (this provisioning network is ""useless most of the time"" since there is almost no provisioning happening :D ) If any, I would love to work on Ironic DHCP-less deployments which was almost working last time I tried and I saw Ironic team contributing fixes since then. > > > > > > I am looking into supporting floating IPs in there: I expect floating > > > IPs to be able to move between racks, as such I am looking into > > > publishing the route for a FIP towards an hypervisor, through BGP. > > > Each fip is a /32 subnet with an hypervisor tenant's IP as next hop. > > > > This is becoming a very common architecture, and that is why there are > > several projects working to achieve the same goal with slightly > > different implementations. > > > > > > > > It seems there are several ideas to achieve this (it was discussed > > > [before][1] in ovs conference) > > > - using [neutron-dynamic-routing][2] - that seems to have some gaps > > > for OVN. It uses os-ken to talk to switches and exchange routes > > > - using [OVN BGP agent][3] that uses FRR, it seems there is a related > > > [RFE][4] for integration in tripleo > > > > > > There is btw also a [BGPVPN][5] project (it does not match my usecase > > > as far as I tried to understand it) that also has some code that talks > > > BGP to switches, already integrated in tripleo. > > > > > > For my tests, I was able to use the neutron-dynamic-routing project > > > (almost) as documented, with a few changes: > > > - for traffic going from VMs to outside the stack, the hypervisor was > > > trying to resolve the "gateway of fIPs" with ARP request which does > > > not make any sense. I created a dummy port with the mac address of the > > > virtual router of the switches: > > > ``` > > > $ openstack port list --mac-address 00:1c:73:00:00:11 -f yaml > > > - Fixed IP Addresses: > > > - ip_address: 10.64.254.1 > > > subnet_id: 8f37 > > > ID: 4028 > > > MAC Address: 00:1c:73:00:00:11 > > > Name: lagw > > > Status: DOWN > > > ``` > > > this prevent the hypervisor to send ARP requests to a non existent gateway > > > - for traffic coming back, we start the neutron-bgp-dragent agent on > > > the controllers. We create the right bgp speaker, peers, etc. > > > - neutron-bgp-dragent seems to work primarily with ovs ml2 plugin, it > > > selects fips and join with ports owned by a "floatingip_agent_gateway" > > > which does not exist on OVN. We can define ourselves some ports so > > > that the dragent is able to find the tenant IP of a host: > > > ``` > > > openstack port create --network provider --device-owner > > > network:floatingip_agent_gateway --host cpu35d.cloud --fixed-ip > > > ip-address=10.64.245.102 ag2 > > > ``` > > > - when creating a floating IP and assigning a port to it, Neutron > > > reads changes from OVN SB and fills the binding information into the > > > port: > > > ``` > > > $ openstack port show -c binding_host_id `openstack floating ip show > > > 10.64.254.177 -f value -c port_id` > > > +-----------------+----------------------------------------+ > > > | Field | Value | > > > +-----------------+----------------------------------------+ > > > | binding_host_id | cpu35d.cloud | > > > +-----------------+----------------------------------------+ > > > ``` > > > this allows the dragent to publish the route for the fip > > > ``` > > > $ openstack bgp speaker list advertised routes bgpspeaker > > > +------------------+---------------+ > > > | Destination | Nexthop | > > > +------------------+---------------+ > > > | 10.64.254.177/32 | 10.64.245.102 | > > > +------------------+---------------+ > > > ``` > > > - traffic reaches the hypervisor but (for reason I don't understand) I > > > had to add a rule > > > ``` > > > $ ip rule > > > 0: from all lookup local > > > 32765: from all iif vlan1234 lookup ovn > > > 32766: from all lookup main > > > 32767: from all lookup default > > > $ ip route show table ovn > > > 10.64.254.177 dev vlan1234 scope link > > > ``` > > > so that the traffic coming for the fip is not immediately discarded by > > > the hypervisor (it's not an ideal solution but it is a workaround that > > > makes my one fIP work!) > > > > > > So all in all it seems it would be possible to use the > > > neutron-dynamic-routing agent, with some minor modifications (eg: to > > > also publish the fip of the OVN L3 gateway router). > > > > > > I am wondering whether I have overlooked anything, and if such kind of > > > deployment (OVN + neutron dynamic routing or similar) is already in > > > use somewhere. Does it make sense to have a RFE for better integration > > > between OVN and neutron-dynamic-routing? > > > > I have been helping to contribute to integrating FRR with OVN in order > > to advertise FIPs and provider network IPs into BGP. The OVN BGP Agent > > is very new, and I'm pretty sure that nobody is using it in production > > yet. However the initial implementation is fairly simple and hopefully > > it will mature quickly. > > > > As you discovered, the solution that uses neutron-bgp-dragent and os-ken > > is not compatible with OVN > Pretty much the contrary, it basically worked. There are a few differences but the gap seems very tiny (unless I overlooked something and I'm fundamentally wrong) I don't understand why a new project would be needed to make it work for OVN. > > >, that is why ovs-bgp-agent is being > > developed. You should be able to try the ovs-bgp-agent with FRR and > > properly configured routing switches, it functions for the basic use case. > > > > The OVN BGP Agent will ensure that FIP and provider network IPs are > > present in the kernel as a /32 or /128 host route, which is then > > advertised into the BGP fabric using the FRR BGP daemon. If the default > > route is received from BGP it will be installed into the kernel by the > > FRR zebra daemon which syncs kernel routes with the FRR BGP routing > > table. The OVN BGP agent installs flows for the Neutron network gateways > > that hand off traffic to the kernel for routing. Since the kernel > > routing table is used, the agent isn't compatible with DPDK fast > > datapath yet. > > > > We don't have good documentation for the OVN BGP integration yet. I've > > only recently been able to make it my primary priority, and some of the > > other engineers which have done the initial proof of concept are moving > > on to other projects. There will be some discussions at the upcoming > > OpenStack PTG about this work, but I am hopeful that the missing pieces > > for your use case will come about in the Yoga cycle. > I did not try to run the OVN BGP agent but I saw your blog posts and I think it's enough to get started with. I still don't get why an extra OVN BGP agent would be needed. One thing I was wondering from the blog posts (and your reply here) is whether every single compute would need connectivity to the physical switches to publish the routes - as the dragent runs on the controller node you only need to configure connectivity between the controllers and the physical switches while in the FRR case you need to open much more. > Would the developments in the Yoga cycle be focused on the OVN BGP agent only, and so there is no interest in improving the neutron-dynamic-routing project ? > Thanks for your insightful comments :) > > > > > > > > > Thanks > > > Francois

1 0