Hello Neutron folks,
Regarding the conversation started in March [1] about the use of OVN interconnect with Neutron, I would like to evolve the discussion in relation to resources allocated by OVN-IC and which are not managed by Neutron. In the March PTG [2], the problem related to the db_sync tool was presented, and a proposed solution in which Neutron does not manage these resources.
After discussing some architectural designs with Felix/StackIT, it seems to make sense to come up with a proposal to change the mech_driver to validate the external_ids key and not remove resources present in the OVN backend without Neutron "signature".
The discussion here is more complex than it seems, because architecturally Neutron does not allow us to interconnect workloads in multiple AZs (with different OpenStacks), but this could be a requirement for a high availability cloud solution (Cloud Region). Additionally, this OVN-IC solution allows interconnecting other cloud backends that use OVN, in the case of kubernetes with ovn-kube.
We tested an OVN interconnect integrated with 3 OpenStack installations and it worked very well !!! considering direct L3 traffic at the router level between different network infrastructures.
SYNC_REPAIR - problem
* Static Routes (learned OVN-IC routes) * Router Port -> Transit Switches
Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in OVN but not in Neutron, port_id=rt2-admin-tenant1 Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': ' 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': '10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not in Neutron
Any suggestions on how to resolve this db_sync issue? since all other Neutron modules did not present any problem with OVN-IC (as far as I tested). I remember Terry was keen to suggest some things to help. I believe that before proposing some complex mechanism to solve this simple problem, I would like to hear the community opinion. In our use case, a bit change in db_sync with filter by neutron key in external_ids would be enough.
Regards, Roberto
[1] https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.ht... [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg
Additional logs:
OpenStack 1
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 IPv4 Routes Route Table <main>: 20.0.1.0/24 169.254.200.2 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 IPv4 Routes Route Table <main>: 10.0.1.0/24 169.254.100.2 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
OpenStack 2
root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc IPv4 Routes Route Table <main>: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list ce45f681-6454-43fe-974f-81344bb8113a IPv4 Routes Route Table <main>: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip
OpenStack 3
root@os-infra-1-neutron-ovn-northd-container-f237db97:~# root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 IPv4 Routes Route Table <main>: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.1.0/24 169.254.200.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 IPv4 Routes Route Table <main>: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.1.0/24 169.254.100.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip
OVN-IC Global database
root@ovn-global-db1:~# ovn-ic-sbctl show availability-zone osp1 gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 hostname: osp1-gwnode1 type: geneve ip: 192.168.200.28 port admin-rt1-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"] port admin-rt1-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] availability-zone osp2 gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 hostname: osp2-gwnode1 type: geneve ip: 192.168.200.128 port admin-rt2-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"] port admin-rt2-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] availability-zone osp3 gateway 97595af9-7896-40d0-a883-beadbff1aa5b hostname: osp3-gwnode1 type: geneve ip: 192.168.200.228 port admin-rt3-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"] port admin-rt3-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"]