October 2021 - openstack-discuss - lists.openstack.org

[neutron] OVN and dynamic routing
by Francois 12 Oct '21

12 Oct '21

Hello Neutron! I am looking into running stacks with OVN on a leaf-spine network, and have some floating IPs routed between racks. Basically each rack is assigned its own set of subnets. Some VLANs are stretched across all racks: the provisioning VLAN used by tripleo to deploy the stack, and the VLANs for the controllers API IPs. However, each tenant subnet is local to a rack: for example each OVN chassis has a ovn-encap-ip with an IP of the tenant subnet of its own rack. Traffic between 2 racks is sent to a spine, and leaves and spines run some eVPN-like thing: each pair of ToR is a vtep, traffic is encapsulated as VXLAN, and routes between vteps are exchanged with BGP. I am looking into supporting floating IPs in there: I expect floating IPs to be able to move between racks, as such I am looking into publishing the route for a FIP towards an hypervisor, through BGP. Each fip is a /32 subnet with an hypervisor tenant's IP as next hop. It seems there are several ideas to achieve this (it was discussed [before][1] in ovs conference) - using [neutron-dynamic-routing][2] - that seems to have some gaps for OVN. It uses os-ken to talk to switches and exchange routes - using [OVN BGP agent][3] that uses FRR, it seems there is a related [RFE][4] for integration in tripleo There is btw also a [BGPVPN][5] project (it does not match my usecase as far as I tried to understand it) that also has some code that talks BGP to switches, already integrated in tripleo. For my tests, I was able to use the neutron-dynamic-routing project (almost) as documented, with a few changes: - for traffic going from VMs to outside the stack, the hypervisor was trying to resolve the "gateway of fIPs" with ARP request which does not make any sense. I created a dummy port with the mac address of the virtual router of the switches: ``` $ openstack port list --mac-address 00:1c:73:00:00:11 -f yaml - Fixed IP Addresses: - ip_address: 10.64.254.1 subnet_id: 8f37 ID: 4028 MAC Address: 00:1c:73:00:00:11 Name: lagw Status: DOWN ``` this prevent the hypervisor to send ARP requests to a non existent gateway - for traffic coming back, we start the neutron-bgp-dragent agent on the controllers. We create the right bgp speaker, peers, etc. - neutron-bgp-dragent seems to work primarily with ovs ml2 plugin, it selects fips and join with ports owned by a "floatingip_agent_gateway" which does not exist on OVN. We can define ourselves some ports so that the dragent is able to find the tenant IP of a host: ``` openstack port create --network provider --device-owner network:floatingip_agent_gateway --host cpu35d.cloud --fixed-ip ip-address=10.64.245.102 ag2 ``` - when creating a floating IP and assigning a port to it, Neutron reads changes from OVN SB and fills the binding information into the port: ``` $ openstack port show -c binding_host_id `openstack floating ip show 10.64.254.177 -f value -c port_id` +-----------------+----------------------------------------+ | Field | Value | +-----------------+----------------------------------------+ | binding_host_id | cpu35d.cloud | +-----------------+----------------------------------------+ ``` this allows the dragent to publish the route for the fip ``` $ openstack bgp speaker list advertised routes bgpspeaker +------------------+---------------+ | Destination | Nexthop | +------------------+---------------+ | 10.64.254.177/32 | 10.64.245.102 | +------------------+---------------+ ``` - traffic reaches the hypervisor but (for reason I don't understand) I had to add a rule ``` $ ip rule 0: from all lookup local 32765: from all iif vlan1234 lookup ovn 32766: from all lookup main 32767: from all lookup default $ ip route show table ovn 10.64.254.177 dev vlan1234 scope link ``` so that the traffic coming for the fip is not immediately discarded by the hypervisor (it's not an ideal solution but it is a workaround that makes my one fIP work!) So all in all it seems it would be possible to use the neutron-dynamic-routing agent, with some minor modifications (eg: to also publish the fip of the OVN L3 gateway router). I am wondering whether I have overlooked anything, and if such kind of deployment (OVN + neutron dynamic routing or similar) is already in use somewhere. Does it make sense to have a RFE for better integration between OVN and neutron-dynamic-routing? Thanks Francois [1]: https://www.openvswitch.org/support/ovscon2020/slides/OVS-CONF-2020-OVN-WIT… [2]: https://docs.openstack.org/neutron/latest/admin/config-bgp-dynamic-routing.… [3]: https://ltomasbo.wordpress.com/2021/02/04/openstack-networking-with-bgp/ [4]: https://specs.openstack.org/openstack/tripleo-specs/specs/wallaby/triplo-bg… [5]: https://github.com/openstack/tripleo-heat-templates/blob/master/environment…

4 5

After rebuilding Queens clusters on Train, race condition causes Designate record creation to fail
by Braden, Albert 12 Oct '21

12 Oct '21

Hello everyone. It's great to be back working on OpenStack again. I'm at Verisign now. I can hardly describe how happy I am to have an employer that does not attach nonsense to the bottom of my emails! We are rebuilding our clusters from Queens to Train. On the new Train clusters, customers are complaining that deleting a VM and then immediately creating a new one with the same name (via Terraform for example) intermittently results in a missing DNS record. We can duplicate the issue by building a VM with terraform, tainting it, and applying. Before applying the change, we see the DNS record in the recordset: $ openstack recordset list dva3.vrsn.com. --all |grep openstack-terra | f9aa73c1-84ba-4854-be71-cbb616de672c | 8d1c84082a044a53abe0d519ed9e8c60 | openstack-terra-test-host.dev-ostck.dva3.vrsn.com. | A | 10.220.4.89 | ACTIVE | NONE | $ and we can pull it from the DNS server on the controllers: $ for i in {1..3}; do dig @dva3-ctrl${i}.cloud.vrsn.com -t axfr dva3.vrsn.com. |grep openstack-terra; done openstack-terra-test-host.dev-ostck.dva3.vrsn.com. 1 IN A 10.220.4.89 openstack-terra-test-host.dev-ostck.dva3.vrsn.com. 1 IN A 10.220.4.89 openstack-terra-test-host.dev-ostck.dva3.vrsn.com. 1 IN A 10.220.4.89 After applying the change, we don't see it: $ openstack recordset list dva3.vrsn.com. --all |grep openstack-terra | f9aa73c1-84ba-4854-be71-cbb616de672c | 8d1c84082a044a53abe0d519ed9e8c60 | openstack-terra-test-host.dev-ostck.dva3.vrsn.com. | A | 10.220.4.89 | ACTIVE | NONE | $ $ for i in {1..3}; do dig @dva3-ctrl${i}.cloud.vrsn.com -t axfr dva3.vrsn.com. |grep openstack-terra; done $ openstack recordset list dva3.vrsn.com. --all |grep openstack-terra $ We see this in the logs: 2021-10-09 01:53:44.307 27 ERROR oslo_messaging.notify.dispatcher oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'c70e693b4c47402db088c43a5a177134-openstack-terra-test-host.de...' for key 'unique_recordset'") 2021-10-09 01:53:44.307 27 ERROR oslo_messaging.notify.dispatcher [SQL: INSERT INTO recordsets (id, version, created_at, zone_shard, tenant_id, zone_id, name, type, ttl, reverse_name) VALUES (%(id)s, %(version)s, %(created_at)s, %(zone_shard)s, %(tenant_id)s, %(zone_id)s, %(name)s, %(type)s, %(ttl)s, %(reverse_name)s)] 2021-10-09 01:53:44.307 27 ERROR oslo_messaging.notify.dispatcher [parameters: {'id': 'dbbb904c347241a791aa01ca33a87b23', 'version': 1, 'created_at': datetime.datetime(2021, 10, 9, 1, 53, 44, 182652), 'zone_shard': 3184, 'tenant_id': '8d1c84082a044a53abe0d519ed9e8c60', 'zone_id': 'c70e693b4c47402db088c43a5a177134', 'name': 'openstack-terra-test-host.dev-ostck.dva3.vrsn.com.', 'type': 'A', 'ttl': None, 'reverse_name': '.moc.nsrv.3avd.kctso-ved.tsoh-tset-arret-kcatsnepo'}] It appears that Designate is trying to create the new record before the deletion of the old one finishes. Is anyone else seeing this on Train? The same set of actions doesn't cause this error in Queens. Do we need to change something in our Designate config, to make it wait until the old records are finished deleting before attempting to create the new ones?

2 6

[Octavia] Can not create LB on SRIOV network
by Zhang, Jing C. (Nokia - CA/Ottawa) 12 Oct '21

12 Oct '21

I can not create Octavia LB on SRIOV network in Train. I went to Octavia story board, did a search but was unable to figure out (the story for SRIOV...). I left a comment under this story, I re-post my questions there, hoping someone knows the answer. Thank you so much Jing https://storyboard.openstack.org/#!/story/2006886 Add VM SRIOV Interface Config Guide (Openstack) Hi, In Openstack train release, creating Octavia LB on SRIOV network fails. I come here to search if there is already a plan to add this support, and see this story. This story gives the impression that the capability is already supported, it is a matter of adding user guide. So, my question is, in which Openstack release, creating LB on SRIOV network is supported? Thank you

3 7

[Xena] It works!
by tjoen 12 Oct '21

12 Oct '21

Just testing every release since Train on an LFS system with Python-3.9 cryptography-35.0.0 is necessary Thank you all

3 5

[ironic] No weekly meeting on Oct18
by Iury Gregory 12 Oct '21

12 Oct '21

Hello ironicers! Just a reminder that on Oct 18, we won't have our weekly meeting because we have a session in the PTG. -- *Att[]'sIury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Part of the ironic-core and puppet-manager-core team in OpenStack* *Software Engineer at Red Hat Czech* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory(a)gmail.com <iurygregory(a)gmail.com>*

1 0

[nova & libvirt] about attach disk on aarch64
by Jaze Lee 12 Oct '21

12 Oct '21

Hello, We run stein openstack in our environment. And already set libvirt value in nova.conf hw_machine_type=aarch64=virt num_pcie_ports = 15 We test and find sometimes disks can not be attached correctly. For example, Built vm with 6 disks, only three disk be there. The others will be inactive in virsh. No obvious error can be found in nova-compute, libvirt, vm os logs. libvirt:5.0.0 qemu:2.12.0-44 openstack-nova: 19.3.2 librbd1:ibrbd1-14.2.16-1.el7.aarch64 Any Suggestions? Thanks a lot

2 1

[wallaby][neutron] Distributed floating IP
by Ammad Syed 12 Oct '21

12 Oct '21

Hi, I am using openstack wallaby. I have seen an issue not sure if its a bug or configuration related issue. I am using ml2/ovn backend with distributed floating IP enabled. I have made my compute node 1 as a gateway chassis where the routers are scheduled. I have then created an instance and NATed a public IP. The instance deployed on compute 2. When I see the IP address via curl ipinfo.io it shows the floating IP that I have NATed. Then I migrated the instance to compute node 1. I had many ping drops for a couple of seconds then its back to normal. I have then seen the IP address via curl ipinfo.io. It showed me the SNAT IP address of router. Then I migrated the instance back to compute node 2, I had ping drops for 20 seconds and then the instance came back. I have seen the IP via curl, it showed the floating IP that I have nated with instance. Is it the expected behavior ? Ammad

2 2

[docs] Keystone docs missing for Xena
by Ben Nemec 12 Oct '21

12 Oct '21

Hey, I was just looking for the Keystone docs and discovered that they are not listed on https://docs.openstack.org/xena/projects.html. If I s/wallaby/xena/ on the wallaby version then it resolves, so it looks like the docs are published they just aren't included in the index for some reason. -Ben

2 1

[kuryr] Virtual PTG October 2021
by Maysa De Macedo Souza 12 Oct '21

12 Oct '21

Hello, I booked the following slots for Kuryr during the Yoga PTG: Oct 19 7-8 UTC and Oct 22 13-14 UTC. If you have any topic ideas you would like to discuss, please include them in the etherpad[1], also it would be interesting to include your name there if you plan to attend any Kuryr session. See you on the next PTG. Cheers, Maysa. [1] https://etherpad.opendev.org/p/kuryr-yoga-ptg

1 2

Restarting Openstack Victoria using kolla-ansible
by Karera Tony 12 Oct '21

12 Oct '21

Hello Team, I am trying to deploy openstack Victoria ..... Below I install kolla-ansible on the deployment server , I first clone the * git clone --branch stable/victoria https://opendev.org/openstack/kolla-ansible* but when I run the deployment without uncommenting openstack_release .. By default it deploys wallaby And when I uncomment and type victoria ...Some of the containers keep restarting esp Horizon Any idea on how to resolve this ? Even the kolla content that I use for deployment, I get it from the kolla-ansible directory that I cloned Regards Tony Karera

2 2