[neutron] unmanaged router resources - OVN interconnect
Hello Neutron folks,
Regarding the conversation started in March [1] about the use of OVN interconnect with Neutron, I would like to evolve the discussion in relation to resources allocated by OVN-IC and which are not managed by Neutron. In the March PTG [2], the problem related to the db_sync tool was presented, and a proposed solution in which Neutron does not manage these resources.
After discussing some architectural designs with Felix/StackIT, it seems to make sense to come up with a proposal to change the mech_driver to validate the external_ids key and not remove resources present in the OVN backend without Neutron "signature".
The discussion here is more complex than it seems, because architecturally Neutron does not allow us to interconnect workloads in multiple AZs (with different OpenStacks), but this could be a requirement for a high availability cloud solution (Cloud Region). Additionally, this OVN-IC solution allows interconnecting other cloud backends that use OVN, in the case of kubernetes with ovn-kube.
We tested an OVN interconnect integrated with 3 OpenStack installations and it worked very well !!! considering direct L3 traffic at the router level between different network infrastructures.
SYNC_REPAIR - problem
* Static Routes (learned OVN-IC routes) * Router Port -> Transit Switches
Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in OVN but not in Neutron, port_id=rt2-admin-tenant1 Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': ' 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': '10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not in Neutron
Any suggestions on how to resolve this db_sync issue? since all other Neutron modules did not present any problem with OVN-IC (as far as I tested). I remember Terry was keen to suggest some things to help. I believe that before proposing some complex mechanism to solve this simple problem, I would like to hear the community opinion. In our use case, a bit change in db_sync with filter by neutron key in external_ids would be enough.
Regards, Roberto
[1] https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.ht... [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg
Additional logs:
OpenStack 1
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 IPv4 Routes Route Table <main>: 20.0.1.0/24 169.254.200.2 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 IPv4 Routes Route Table <main>: 10.0.1.0/24 169.254.100.2 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
OpenStack 2
root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc IPv4 Routes Route Table <main>: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list ce45f681-6454-43fe-974f-81344bb8113a IPv4 Routes Route Table <main>: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip
OpenStack 3
root@os-infra-1-neutron-ovn-northd-container-f237db97:~# root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 IPv4 Routes Route Table <main>: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.1.0/24 169.254.200.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 IPv4 Routes Route Table <main>: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.1.0/24 169.254.100.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip
OVN-IC Global database
root@ovn-global-db1:~# ovn-ic-sbctl show availability-zone osp1 gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 hostname: osp1-gwnode1 type: geneve ip: 192.168.200.28 port admin-rt1-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"] port admin-rt1-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] availability-zone osp2 gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 hostname: osp2-gwnode1 type: geneve ip: 192.168.200.128 port admin-rt2-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"] port admin-rt2-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] availability-zone osp3 gateway 97595af9-7896-40d0-a883-beadbff1aa5b hostname: osp3-gwnode1 type: geneve ip: 192.168.200.228 port admin-rt3-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"] port admin-rt3-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"]
Hello Roberto:
We talked about this possible RFE during the PTG [1] but I don't remember having any proposal. Actually what I remember (and was written in the etherpad) is that you were going to present an RFE. Can you explain it briefly?
We also talked about the idea, proposed by Felix in the mailing list, of having a different network type only to hold the Transit Switches and thus the LRP. If you have a proposal, please present it.
Regards.
[1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506
On Tue, Jul 11, 2023 at 10:50 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
Hello Neutron folks,
Regarding the conversation started in March [1] about the use of OVN interconnect with Neutron, I would like to evolve the discussion in relation to resources allocated by OVN-IC and which are not managed by Neutron. In the March PTG [2], the problem related to the db_sync tool was presented, and a proposed solution in which Neutron does not manage these resources.
After discussing some architectural designs with Felix/StackIT, it seems to make sense to come up with a proposal to change the mech_driver to validate the external_ids key and not remove resources present in the OVN backend without Neutron "signature".
The discussion here is more complex than it seems, because architecturally Neutron does not allow us to interconnect workloads in multiple AZs (with different OpenStacks), but this could be a requirement for a high availability cloud solution (Cloud Region). Additionally, this OVN-IC solution allows interconnecting other cloud backends that use OVN, in the case of kubernetes with ovn-kube.
We tested an OVN interconnect integrated with 3 OpenStack installations and it worked very well !!! considering direct L3 traffic at the router level between different network infrastructures.
SYNC_REPAIR - problem
- Static Routes (learned OVN-IC routes)
- Router Port -> Transit Switches
Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in OVN but not in Neutron, port_id=rt2-admin-tenant1 Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': ' 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': '10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not in Neutron
Any suggestions on how to resolve this db_sync issue? since all other Neutron modules did not present any problem with OVN-IC (as far as I tested). I remember Terry was keen to suggest some things to help. I believe that before proposing some complex mechanism to solve this simple problem, I would like to hear the community opinion. In our use case, a bit change in db_sync with filter by neutron key in external_ids would be enough.
Regards, Roberto
[1] https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.ht... [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg
Additional logs:
OpenStack 1
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 IPv4 Routes Route Table <main>: 20.0.1.0/24 169.254.200.2 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 IPv4 Routes Route Table <main>: 10.0.1.0/24 169.254.100.2 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
OpenStack 2
root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc IPv4 Routes Route Table <main>: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list ce45f681-6454-43fe-974f-81344bb8113a IPv4 Routes Route Table <main>: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip
OpenStack 3
root@os-infra-1-neutron-ovn-northd-container-f237db97:~# root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 IPv4 Routes Route Table <main>: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.1.0/24 169.254.200.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 IPv4 Routes Route Table <main>: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.1.0/24 169.254.100.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip
OVN-IC Global database
root@ovn-global-db1:~# ovn-ic-sbctl show availability-zone osp1 gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 hostname: osp1-gwnode1 type: geneve ip: 192.168.200.28 port admin-rt1-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"] port admin-rt1-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] availability-zone osp2 gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 hostname: osp2-gwnode1 type: geneve ip: 192.168.200.128 port admin-rt2-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"] port admin-rt2-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] availability-zone osp3 gateway 97595af9-7896-40d0-a883-beadbff1aa5b hostname: osp3-gwnode1 type: geneve ip: 192.168.200.228 port admin-rt3-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"] port admin-rt3-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"]
*‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.*
*‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
Hi Rodolfo,
Thanks for the feedback.
I liked the idea of having a different network type only to hold the Transit Switches and thus the LRP but it depends on some factors like multi-tenancy.
From the OVN perspective, the TS is global and will only be linked to a
tenant when it is plugged into the tenant router port. My concern here is that if Neutron manages the TS, it will need to assume the dynamic characteristics of this TS function, ovn-ic creates and removes the TS in NB_Global, in addition to plugging and removing ports in this switch according to the network topology. Additionally, Neutron would still need to handle the learned remote routes (which are listed as static routes from the tenant router perspective).
This is an open discussion, Felix can add ideas here.
In general, it seems to me that we have no alternatives for this type of solution other than OVN-IC (note that ovn-bgp-agent does not learn remote routes at the SDN level / OVN). OpenStack seems to be designed to run like a "bubble" and this traffic from one VM to another always needs to be routed at the FIP level and external routing layers.
Regards, Roberto
[image: image.png]
Em qua., 12 de jul. de 2023 às 05:11, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Hello Roberto:
We talked about this possible RFE during the PTG [1] but I don't remember having any proposal. Actually what I remember (and was written in the etherpad) is that you were going to present an RFE. Can you explain it briefly?
We also talked about the idea, proposed by Felix in the mailing list, of having a different network type only to hold the Transit Switches and thus the LRP. If you have a proposal, please present it.
Regards.
[1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506
On Tue, Jul 11, 2023 at 10:50 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
Hello Neutron folks,
Regarding the conversation started in March [1] about the use of OVN interconnect with Neutron, I would like to evolve the discussion in relation to resources allocated by OVN-IC and which are not managed by Neutron. In the March PTG [2], the problem related to the db_sync tool was presented, and a proposed solution in which Neutron does not manage these resources.
After discussing some architectural designs with Felix/StackIT, it seems to make sense to come up with a proposal to change the mech_driver to validate the external_ids key and not remove resources present in the OVN backend without Neutron "signature".
The discussion here is more complex than it seems, because architecturally Neutron does not allow us to interconnect workloads in multiple AZs (with different OpenStacks), but this could be a requirement for a high availability cloud solution (Cloud Region). Additionally, this OVN-IC solution allows interconnecting other cloud backends that use OVN, in the case of kubernetes with ovn-kube.
We tested an OVN interconnect integrated with 3 OpenStack installations and it worked very well !!! considering direct L3 traffic at the router level between different network infrastructures.
SYNC_REPAIR - problem
- Static Routes (learned OVN-IC routes)
- Router Port -> Transit Switches
Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in OVN but not in Neutron, port_id=rt2-admin-tenant1 Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': ' 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': '10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not in Neutron
Any suggestions on how to resolve this db_sync issue? since all other Neutron modules did not present any problem with OVN-IC (as far as I tested). I remember Terry was keen to suggest some things to help. I believe that before proposing some complex mechanism to solve this simple problem, I would like to hear the community opinion. In our use case, a bit change in db_sync with filter by neutron key in external_ids would be enough.
Regards, Roberto
[1] https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.ht... [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg
Additional logs:
OpenStack 1
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 IPv4 Routes Route Table <main>: 20.0.1.0/24 169.254.200.2 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 IPv4 Routes Route Table <main>: 10.0.1.0/24 169.254.100.2 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
OpenStack 2
root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc IPv4 Routes Route Table <main>: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list ce45f681-6454-43fe-974f-81344bb8113a IPv4 Routes Route Table <main>: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip
OpenStack 3
root@os-infra-1-neutron-ovn-northd-container-f237db97:~# root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 IPv4 Routes Route Table <main>: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.1.0/24 169.254.200.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 IPv4 Routes Route Table <main>: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.1.0/24 169.254.100.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip
OVN-IC Global database
root@ovn-global-db1:~# ovn-ic-sbctl show availability-zone osp1 gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 hostname: osp1-gwnode1 type: geneve ip: 192.168.200.28 port admin-rt1-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"] port admin-rt1-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] availability-zone osp2 gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 hostname: osp2-gwnode1 type: geneve ip: 192.168.200.128 port admin-rt2-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"] port admin-rt2-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] availability-zone osp3 gateway 97595af9-7896-40d0-a883-beadbff1aa5b hostname: osp3-gwnode1 type: geneve ip: 192.168.200.228 port admin-rt3-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"] port admin-rt3-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"]
*‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.*
*‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
Hi Rodolfo,
Thanks for the feedback.
I liked the idea of having a different network type only to hold the Transit Switches and thus the LRP but it depends on some factors like multi-tenancy.
From the OVN perspective, the TS is global and will only be linked to a
tenant when it is plugged into the tenant router port. My concern here is that if Neutron manages the TS, it will need to assume the dynamic characteristics of this TS function, ovn-ic creates and removes the TS in NB_Global, in addition to plugging and removing ports in this switch according to the network topology. Additionally, Neutron would still need to handle the learned remote routes (which are listed as static routes from the tenant router perspective).
This is an open discussion, Felix can add ideas here.
In general, it seems to me that we have no alternatives for this type of solution other than OVN-IC (note that ovn-bgp-agent does not learn remote routes at the SDN level / OVN). OpenStack seems to be designed to run like a "bubble" and this traffic from one VM to another always needs to be routed at the FIP level and external routing layers.
Regards, Roberto
Em qua., 12 de jul. de 2023 às 05:11, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Hello Roberto:
We talked about this possible RFE during the PTG [1] but I don't remember having any proposal. Actually what I remember (and was written in the etherpad) is that you were going to present an RFE. Can you explain it briefly?
We also talked about the idea, proposed by Felix in the mailing list, of having a different network type only to hold the Transit Switches and thus the LRP. If you have a proposal, please present it.
Regards.
[1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506
On Tue, Jul 11, 2023 at 10:50 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
Hello Neutron folks,
Regarding the conversation started in March [1] about the use of OVN interconnect with Neutron, I would like to evolve the discussion in relation to resources allocated by OVN-IC and which are not managed by Neutron. In the March PTG [2], the problem related to the db_sync tool was presented, and a proposed solution in which Neutron does not manage these resources.
After discussing some architectural designs with Felix/StackIT, it seems to make sense to come up with a proposal to change the mech_driver to validate the external_ids key and not remove resources present in the OVN backend without Neutron "signature".
The discussion here is more complex than it seems, because architecturally Neutron does not allow us to interconnect workloads in multiple AZs (with different OpenStacks), but this could be a requirement for a high availability cloud solution (Cloud Region). Additionally, this OVN-IC solution allows interconnecting other cloud backends that use OVN, in the case of kubernetes with ovn-kube.
We tested an OVN interconnect integrated with 3 OpenStack installations and it worked very well !!! considering direct L3 traffic at the router level between different network infrastructures.
SYNC_REPAIR - problem
- Static Routes (learned OVN-IC routes)
- Router Port -> Transit Switches
Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in OVN but not in Neutron, port_id=rt2-admin-tenant1 Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': ' 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': '10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not in Neutron
Any suggestions on how to resolve this db_sync issue? since all other Neutron modules did not present any problem with OVN-IC (as far as I tested). I remember Terry was keen to suggest some things to help. I believe that before proposing some complex mechanism to solve this simple problem, I would like to hear the community opinion. In our use case, a bit change in db_sync with filter by neutron key in external_ids would be enough.
Regards, Roberto
[1] https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.ht... [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg
Additional logs:
OpenStack 1
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 IPv4 Routes Route Table <main>: 20.0.1.0/24 169.254.200.2 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 IPv4 Routes Route Table <main>: 10.0.1.0/24 169.254.100.2 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
OpenStack 2
root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc IPv4 Routes Route Table <main>: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list ce45f681-6454-43fe-974f-81344bb8113a IPv4 Routes Route Table <main>: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip
OpenStack 3
root@os-infra-1-neutron-ovn-northd-container-f237db97:~# root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 IPv4 Routes Route Table <main>: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.1.0/24 169.254.200.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 IPv4 Routes Route Table <main>: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.1.0/24 169.254.100.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip
OVN-IC Global database
root@ovn-global-db1:~# ovn-ic-sbctl show availability-zone osp1 gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 hostname: osp1-gwnode1 type: geneve ip: 192.168.200.28 port admin-rt1-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"] port admin-rt1-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] availability-zone osp2 gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 hostname: osp2-gwnode1 type: geneve ip: 192.168.200.128 port admin-rt2-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"] port admin-rt2-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] availability-zone osp3 gateway 97595af9-7896-40d0-a883-beadbff1aa5b hostname: osp3-gwnode1 type: geneve ip: 192.168.200.228 port admin-rt3-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"] port admin-rt3-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"]
*‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.*
*‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
Em qua., 12 de jul. de 2023 às 09:05, Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> escreveu:
Hi Rodolfo,
Thanks for the feedback.
I liked the idea of having a different network type only to hold the Transit Switches and thus the LRP but it depends on some factors like multi-tenancy. From the OVN perspective, the TS is global and will only be linked to a tenant when it is plugged into the tenant router port. My concern here is that if Neutron manages the TS, it will need to assume the dynamic characteristics of this TS function, ovn-ic creates and removes the TS in NB_Global, in addition to plugging and removing ports in this switch according to the network topology. Additionally, Neutron would still need to handle the learned remote routes (which are listed as static routes from the tenant router perspective).
sorry: NB_Global -> Northbound Database.
This is an open discussion, Felix can add ideas here.
In general, it seems to me that we have no alternatives for this type of solution other than OVN-IC (note that ovn-bgp-agent does not learn remote routes at the SDN level / OVN). OpenStack seems to be designed to run like a "bubble" and this traffic from one VM to another always needs to be routed at the FIP level and external routing layers.
Regards, Roberto
Em qua., 12 de jul. de 2023 às 05:11, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Hello Roberto:
We talked about this possible RFE during the PTG [1] but I don't remember having any proposal. Actually what I remember (and was written in the etherpad) is that you were going to present an RFE. Can you explain it briefly?
We also talked about the idea, proposed by Felix in the mailing list, of having a different network type only to hold the Transit Switches and thus the LRP. If you have a proposal, please present it.
Regards.
[1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506
On Tue, Jul 11, 2023 at 10:50 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
Hello Neutron folks,
Regarding the conversation started in March [1] about the use of OVN interconnect with Neutron, I would like to evolve the discussion in relation to resources allocated by OVN-IC and which are not managed by Neutron. In the March PTG [2], the problem related to the db_sync tool was presented, and a proposed solution in which Neutron does not manage these resources.
After discussing some architectural designs with Felix/StackIT, it seems to make sense to come up with a proposal to change the mech_driver to validate the external_ids key and not remove resources present in the OVN backend without Neutron "signature".
The discussion here is more complex than it seems, because architecturally Neutron does not allow us to interconnect workloads in multiple AZs (with different OpenStacks), but this could be a requirement for a high availability cloud solution (Cloud Region). Additionally, this OVN-IC solution allows interconnecting other cloud backends that use OVN, in the case of kubernetes with ovn-kube.
We tested an OVN interconnect integrated with 3 OpenStack installations and it worked very well !!! considering direct L3 traffic at the router level between different network infrastructures.
SYNC_REPAIR - problem
- Static Routes (learned OVN-IC routes)
- Router Port -> Transit Switches
Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in OVN but not in Neutron, port_id=rt2-admin-tenant1 Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': ' 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': '10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not in Neutron
Any suggestions on how to resolve this db_sync issue? since all other Neutron modules did not present any problem with OVN-IC (as far as I tested). I remember Terry was keen to suggest some things to help. I believe that before proposing some complex mechanism to solve this simple problem, I would like to hear the community opinion. In our use case, a bit change in db_sync with filter by neutron key in external_ids would be enough.
Regards, Roberto
[1] https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.ht... [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg
Additional logs:
OpenStack 1
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 IPv4 Routes Route Table <main>: 20.0.1.0/24 169.254.200.2 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 IPv4 Routes Route Table <main>: 10.0.1.0/24 169.254.100.2 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
OpenStack 2
root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc IPv4 Routes Route Table <main>: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list ce45f681-6454-43fe-974f-81344bb8113a IPv4 Routes Route Table <main>: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip
OpenStack 3
root@os-infra-1-neutron-ovn-northd-container-f237db97:~# root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 IPv4 Routes Route Table <main>: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.1.0/24 169.254.200.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 IPv4 Routes Route Table <main>: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.1.0/24 169.254.100.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip
OVN-IC Global database
root@ovn-global-db1:~# ovn-ic-sbctl show availability-zone osp1 gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 hostname: osp1-gwnode1 type: geneve ip: 192.168.200.28 port admin-rt1-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"] port admin-rt1-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] availability-zone osp2 gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 hostname: osp2-gwnode1 type: geneve ip: 192.168.200.128 port admin-rt2-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"] port admin-rt2-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] availability-zone osp3 gateway 97595af9-7896-40d0-a883-beadbff1aa5b hostname: osp3-gwnode1 type: geneve ip: 192.168.200.228 port admin-rt3-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"] port admin-rt3-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"]
*‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.*
*‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
Hello Roberto:
I think you have provided a very detailed explanation of the issue but you still didn't present any possible solution. I would ask you at least for a proposal and for specific details of what is failing in the OVN sync tool.
For example, if you add a TS between two clouds, you'll need to create (1) routers (NAT registers), (2) router ports (LRP) and (3) some static routes (LRSR). All these elements are monitored by the DB sync tool and will fail because the counterparts in the Neutron DB don't exist. I guess that you have some script or tool or at least a document to manually add these resources; sharing it could help to find a solution.
If these elements are manually created during the deployment phase, you can also control the information provided. I'm thinking about the "external_ids" field; we can push some defined constant that will make these registers transparent to the OVN DB sync tool.
Please check this idea and if it is feasible. In that case, please add a topic to the Neutron drivers meeting agenda [1] to discuss it.
Regards.
[1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers
On Wed, Jul 12, 2023 at 2:12 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
Em qua., 12 de jul. de 2023 às 09:05, Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> escreveu:
Hi Rodolfo,
Thanks for the feedback.
I liked the idea of having a different network type only to hold the Transit Switches and thus the LRP but it depends on some factors like multi-tenancy. From the OVN perspective, the TS is global and will only be linked to a tenant when it is plugged into the tenant router port. My concern here is that if Neutron manages the TS, it will need to assume the dynamic characteristics of this TS function, ovn-ic creates and removes the TS in NB_Global, in addition to plugging and removing ports in this switch according to the network topology. Additionally, Neutron would still need to handle the learned remote routes (which are listed as static routes from the tenant router perspective).
sorry: NB_Global -> Northbound Database.
This is an open discussion, Felix can add ideas here.
In general, it seems to me that we have no alternatives for this type of solution other than OVN-IC (note that ovn-bgp-agent does not learn remote routes at the SDN level / OVN). OpenStack seems to be designed to run like a "bubble" and this traffic from one VM to another always needs to be routed at the FIP level and external routing layers.
Regards, Roberto
Em qua., 12 de jul. de 2023 às 05:11, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Hello Roberto:
We talked about this possible RFE during the PTG [1] but I don't remember having any proposal. Actually what I remember (and was written in the etherpad) is that you were going to present an RFE. Can you explain it briefly?
We also talked about the idea, proposed by Felix in the mailing list, of having a different network type only to hold the Transit Switches and thus the LRP. If you have a proposal, please present it.
Regards.
[1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506
On Tue, Jul 11, 2023 at 10:50 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
Hello Neutron folks,
Regarding the conversation started in March [1] about the use of OVN interconnect with Neutron, I would like to evolve the discussion in relation to resources allocated by OVN-IC and which are not managed by Neutron. In the March PTG [2], the problem related to the db_sync tool was presented, and a proposed solution in which Neutron does not manage these resources.
After discussing some architectural designs with Felix/StackIT, it seems to make sense to come up with a proposal to change the mech_driver to validate the external_ids key and not remove resources present in the OVN backend without Neutron "signature".
The discussion here is more complex than it seems, because architecturally Neutron does not allow us to interconnect workloads in multiple AZs (with different OpenStacks), but this could be a requirement for a high availability cloud solution (Cloud Region). Additionally, this OVN-IC solution allows interconnecting other cloud backends that use OVN, in the case of kubernetes with ovn-kube.
We tested an OVN interconnect integrated with 3 OpenStack installations and it worked very well !!! considering direct L3 traffic at the router level between different network infrastructures.
SYNC_REPAIR - problem
- Static Routes (learned OVN-IC routes)
- Router Port -> Transit Switches
Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in OVN but not in Neutron, port_id=rt2-admin-tenant1 Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': ' 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': '10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not in Neutron
Any suggestions on how to resolve this db_sync issue? since all other Neutron modules did not present any problem with OVN-IC (as far as I tested). I remember Terry was keen to suggest some things to help. I believe that before proposing some complex mechanism to solve this simple problem, I would like to hear the community opinion. In our use case, a bit change in db_sync with filter by neutron key in external_ids would be enough.
Regards, Roberto
[1] https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.ht... [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg
Additional logs:
OpenStack 1
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 IPv4 Routes Route Table <main>: 20.0.1.0/24 169.254.200.2 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 IPv4 Routes Route Table <main>: 10.0.1.0/24 169.254.100.2 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
OpenStack 2
root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc IPv4 Routes Route Table <main>: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list ce45f681-6454-43fe-974f-81344bb8113a IPv4 Routes Route Table <main>: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip
OpenStack 3
root@os-infra-1-neutron-ovn-northd-container-f237db97:~# root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 IPv4 Routes Route Table <main>: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.1.0/24 169.254.200.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 IPv4 Routes Route Table <main>: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.1.0/24 169.254.100.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip
OVN-IC Global database
root@ovn-global-db1:~# ovn-ic-sbctl show availability-zone osp1 gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 hostname: osp1-gwnode1 type: geneve ip: 192.168.200.28 port admin-rt1-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"] port admin-rt1-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] availability-zone osp2 gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 hostname: osp2-gwnode1 type: geneve ip: 192.168.200.128 port admin-rt2-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"] port admin-rt2-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] availability-zone osp3 gateway 97595af9-7896-40d0-a883-beadbff1aa5b hostname: osp3-gwnode1 type: geneve ip: 192.168.200.228 port admin-rt3-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"] port admin-rt3-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"]
*‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.*
*‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
*‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.*
*‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
Hi Rodolfo,
Em qui., 13 de jul. de 2023 às 13:45, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Hello Roberto:
I think you have provided a very detailed explanation of the issue but you still didn't present any possible solution. I would ask you at least for a proposal and for specific details of what is failing in the OVN sync tool.
For example, if you add a TS between two clouds, you'll need to create (1) routers (NAT registers), (2) router ports (LRP) and (3) some static routes (LRSR). All these elements are monitored by the DB sync tool and will fail because the counterparts in the Neutron DB don't exist. I guess that you have some script or tool or at least a document to manually add these resources; sharing it could help to find a solution.
If these elements are manually created during the deployment phase, you can also control the information provided. I'm thinking about the "external_ids" field; we can push some defined constant that will make these registers transparent to the OVN DB sync tool.
Please check this idea and if it is feasible. In that case, please add a topic to the Neutron drivers meeting agenda [1] to discuss it.
That sounds good, thanks. https://bugs.launchpad.net/neutron/+bug/2027742
Regards, Roberto
Regards.
[1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers
On Wed, Jul 12, 2023 at 2:12 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
Em qua., 12 de jul. de 2023 às 09:05, Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> escreveu:
Hi Rodolfo,
Thanks for the feedback.
I liked the idea of having a different network type only to hold the Transit Switches and thus the LRP but it depends on some factors like multi-tenancy. From the OVN perspective, the TS is global and will only be linked to a tenant when it is plugged into the tenant router port. My concern here is that if Neutron manages the TS, it will need to assume the dynamic characteristics of this TS function, ovn-ic creates and removes the TS in NB_Global, in addition to plugging and removing ports in this switch according to the network topology. Additionally, Neutron would still need to handle the learned remote routes (which are listed as static routes from the tenant router perspective).
sorry: NB_Global -> Northbound Database.
This is an open discussion, Felix can add ideas here.
In general, it seems to me that we have no alternatives for this type of solution other than OVN-IC (note that ovn-bgp-agent does not learn remote routes at the SDN level / OVN). OpenStack seems to be designed to run like a "bubble" and this traffic from one VM to another always needs to be routed at the FIP level and external routing layers.
Regards, Roberto
Em qua., 12 de jul. de 2023 às 05:11, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Hello Roberto:
We talked about this possible RFE during the PTG [1] but I don't remember having any proposal. Actually what I remember (and was written in the etherpad) is that you were going to present an RFE. Can you explain it briefly?
We also talked about the idea, proposed by Felix in the mailing list, of having a different network type only to hold the Transit Switches and thus the LRP. If you have a proposal, please present it.
Regards.
[1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506
On Tue, Jul 11, 2023 at 10:50 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
Hello Neutron folks,
Regarding the conversation started in March [1] about the use of OVN interconnect with Neutron, I would like to evolve the discussion in relation to resources allocated by OVN-IC and which are not managed by Neutron. In the March PTG [2], the problem related to the db_sync tool was presented, and a proposed solution in which Neutron does not manage these resources.
After discussing some architectural designs with Felix/StackIT, it seems to make sense to come up with a proposal to change the mech_driver to validate the external_ids key and not remove resources present in the OVN backend without Neutron "signature".
The discussion here is more complex than it seems, because architecturally Neutron does not allow us to interconnect workloads in multiple AZs (with different OpenStacks), but this could be a requirement for a high availability cloud solution (Cloud Region). Additionally, this OVN-IC solution allows interconnecting other cloud backends that use OVN, in the case of kubernetes with ovn-kube.
We tested an OVN interconnect integrated with 3 OpenStack installations and it worked very well !!! considering direct L3 traffic at the router level between different network infrastructures.
SYNC_REPAIR - problem
- Static Routes (learned OVN-IC routes)
- Router Port -> Transit Switches
Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in OVN but not in Neutron, port_id=rt2-admin-tenant1 Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': ' 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': ' 10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not in Neutron
Any suggestions on how to resolve this db_sync issue? since all other Neutron modules did not present any problem with OVN-IC (as far as I tested). I remember Terry was keen to suggest some things to help. I believe that before proposing some complex mechanism to solve this simple problem, I would like to hear the community opinion. In our use case, a bit change in db_sync with filter by neutron key in external_ids would be enough.
Regards, Roberto
[1] https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.ht... [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg
Additional logs:
OpenStack 1
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 IPv4 Routes Route Table <main>: 20.0.1.0/24 169.254.200.2 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 IPv4 Routes Route Table <main>: 10.0.1.0/24 169.254.100.2 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
OpenStack 2
root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc IPv4 Routes Route Table <main>: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list ce45f681-6454-43fe-974f-81344bb8113a IPv4 Routes Route Table <main>: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip
OpenStack 3
root@os-infra-1-neutron-ovn-northd-container-f237db97:~# root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 IPv4 Routes Route Table <main>: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.1.0/24 169.254.200.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 IPv4 Routes Route Table <main>: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.1.0/24 169.254.100.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip
OVN-IC Global database
root@ovn-global-db1:~# ovn-ic-sbctl show availability-zone osp1 gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 hostname: osp1-gwnode1 type: geneve ip: 192.168.200.28 port admin-rt1-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"] port admin-rt1-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] availability-zone osp2 gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 hostname: osp2-gwnode1 type: geneve ip: 192.168.200.128 port admin-rt2-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"] port admin-rt2-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] availability-zone osp3 gateway 97595af9-7896-40d0-a883-beadbff1aa5b hostname: osp3-gwnode1 type: geneve ip: 192.168.200.228 port admin-rt3-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"] port admin-rt3-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"]
*‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.*
*‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
*‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.*
*‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
Hi everyone,
On Thu, Jul 13, 2023 at 07:29:51PM -0300, Roberto Bartzen Acosta wrote:
Hi Rodolfo,
Em qui., 13 de jul. de 2023 às 13:45, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Hello Roberto:
I think you have provided a very detailed explanation of the issue but you still didn't present any possible solution. I would ask you at least for a proposal and for specific details of what is failing in the OVN sync tool.
For example, if you add a TS between two clouds, you'll need to create (1) routers (NAT registers), (2) router ports (LRP) and (3) some static routes (LRSR). All these elements are monitored by the DB sync tool and will fail because the counterparts in the Neutron DB don't exist. I guess that you have some script or tool or at least a document to manually add these resources; sharing it could help to find a solution.
If these elements are manually created during the deployment phase, you can also control the information provided. I'm thinking about the "external_ids" field; we can push some defined constant that will make these registers transparent to the OVN DB sync tool.
As these resources are also created by tools not necessarily controlled by us we need to avoid changing them. I would therefor propose to not add a "ignore" constant to the "external_ids", to to rather check for the existence of the neutron keys in "external_ids" (i think thats also what is described in the bug below).
Please check this idea and if it is feasible. In that case, please add a topic to the Neutron drivers meeting agenda [1] to discuss it.
That sounds good, thanks. https://bugs.launchpad.net/neutron/+bug/2027742
From my perspective there is also an additional step after the above
REF. Currently it just makes it possible to use ovn-ic without neutron killing it. However if we could get neutron to treat a Transit Switch as a external network we would skip the need to implement the NAT, LRP and LRSR logic in custom tooling and could use the existing neutron code. This would probably require creating a custom type of provider network and then specifying the transit switch name in the provider options.
Is that something that from your perspective we should take a look at right away, or rather first focus und neutron not burning the transit switch.
Regards Felix
Regards, Roberto
Regards.
[1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers
On Wed, Jul 12, 2023 at 2:12 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
Em qua., 12 de jul. de 2023 às 09:05, Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> escreveu:
Hi Rodolfo,
Thanks for the feedback.
I liked the idea of having a different network type only to hold the Transit Switches and thus the LRP but it depends on some factors like multi-tenancy. From the OVN perspective, the TS is global and will only be linked to a tenant when it is plugged into the tenant router port. My concern here is that if Neutron manages the TS, it will need to assume the dynamic characteristics of this TS function, ovn-ic creates and removes the TS in NB_Global, in addition to plugging and removing ports in this switch according to the network topology. Additionally, Neutron would still need to handle the learned remote routes (which are listed as static routes from the tenant router perspective).
sorry: NB_Global -> Northbound Database.
This is an open discussion, Felix can add ideas here.
In general, it seems to me that we have no alternatives for this type of solution other than OVN-IC (note that ovn-bgp-agent does not learn remote routes at the SDN level / OVN). OpenStack seems to be designed to run like a "bubble" and this traffic from one VM to another always needs to be routed at the FIP level and external routing layers.
Regards, Roberto
Em qua., 12 de jul. de 2023 às 05:11, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Hello Roberto:
We talked about this possible RFE during the PTG [1] but I don't remember having any proposal. Actually what I remember (and was written in the etherpad) is that you were going to present an RFE. Can you explain it briefly?
We also talked about the idea, proposed by Felix in the mailing list, of having a different network type only to hold the Transit Switches and thus the LRP. If you have a proposal, please present it.
Regards.
[1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506
On Tue, Jul 11, 2023 at 10:50 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
Hello Neutron folks,
Regarding the conversation started in March [1] about the use of OVN interconnect with Neutron, I would like to evolve the discussion in relation to resources allocated by OVN-IC and which are not managed by Neutron. In the March PTG [2], the problem related to the db_sync tool was presented, and a proposed solution in which Neutron does not manage these resources.
After discussing some architectural designs with Felix/StackIT, it seems to make sense to come up with a proposal to change the mech_driver to validate the external_ids key and not remove resources present in the OVN backend without Neutron "signature".
The discussion here is more complex than it seems, because architecturally Neutron does not allow us to interconnect workloads in multiple AZs (with different OpenStacks), but this could be a requirement for a high availability cloud solution (Cloud Region). Additionally, this OVN-IC solution allows interconnecting other cloud backends that use OVN, in the case of kubernetes with ovn-kube.
We tested an OVN interconnect integrated with 3 OpenStack installations and it worked very well !!! considering direct L3 traffic at the router level between different network infrastructures.
SYNC_REPAIR - problem
- Static Routes (learned OVN-IC routes)
- Router Port -> Transit Switches
Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in OVN but not in Neutron, port_id=rt2-admin-tenant1 Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': ' 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': ' 10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not in Neutron
Any suggestions on how to resolve this db_sync issue? since all other Neutron modules did not present any problem with OVN-IC (as far as I tested). I remember Terry was keen to suggest some things to help. I believe that before proposing some complex mechanism to solve this simple problem, I would like to hear the community opinion. In our use case, a bit change in db_sync with filter by neutron key in external_ids would be enough.
Regards, Roberto
[1] https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.ht... [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg
Additional logs:
OpenStack 1
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 IPv4 Routes Route Table <main>: 20.0.1.0/24 169.254.200.2 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 IPv4 Routes Route Table <main>: 10.0.1.0/24 169.254.100.2 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
OpenStack 2
root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc IPv4 Routes Route Table <main>: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list ce45f681-6454-43fe-974f-81344bb8113a IPv4 Routes Route Table <main>: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip
OpenStack 3
root@os-infra-1-neutron-ovn-northd-container-f237db97:~# root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 IPv4 Routes Route Table <main>: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.1.0/24 169.254.200.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 IPv4 Routes Route Table <main>: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.1.0/24 169.254.100.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip
IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip
OVN-IC Global database
root@ovn-global-db1:~# ovn-ic-sbctl show availability-zone osp1 gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 hostname: osp1-gwnode1 type: geneve ip: 192.168.200.28 port admin-rt1-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"] port admin-rt1-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] availability-zone osp2 gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 hostname: osp2-gwnode1 type: geneve ip: 192.168.200.128 port admin-rt2-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"] port admin-rt2-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] availability-zone osp3 gateway 97595af9-7896-40d0-a883-beadbff1aa5b hostname: osp3-gwnode1 type: geneve ip: 192.168.200.228 port admin-rt3-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"] port admin-rt3-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"]
*‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.*
*‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
*‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.*
*‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
--
_‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’._
- **‘Apesar do Magazine Luiza tomar
todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail.
Hinweise zum Datenschutz finden Sie hierhttps://www.datenschutz.schwarz.
This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail.
Information on data protection can be found herehttps://www.datenschutz.schwarz.
Hello Felix:
That's the point: the launchpad bug describes the registers created by the IC configuration but these registers do not have any identification that informs about their relationship with the IC TS.
Rather than modifying the Neutron resources (a network is a generic resource that is used by multiple backends), I would try to create a NB-IC resource map and the corresponding NB registers (NAT, LRP and LRSR). If you can link a NB-IC TS with the corresponding NB resources, then the problem is fixed because knowing them we can explicitly skip them from the DB sync tool.
If that is not the case, then please present an alternative with the suggested idea of creating an external network type.
BTW, I don't understand why you first say that "created by tools not necessarily controlled by us we need to avoid changing them", but then you are talking about refactoring these tools. Are we talking about the tools that create these IC configurations? Can you share them?
Regards.
On Mon, Jul 17, 2023 at 8:43 AM Felix Huettner felix.huettner@mail.schwarz wrote:
Hi everyone,
On Thu, Jul 13, 2023 at 07:29:51PM -0300, Roberto Bartzen Acosta wrote:
Hi Rodolfo,
Em qui., 13 de jul. de 2023 às 13:45, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Hello Roberto:
I think you have provided a very detailed explanation of the issue but
you
still didn't present any possible solution. I would ask you at least
for a
proposal and for specific details of what is failing in the OVN sync
tool.
For example, if you add a TS between two clouds, you'll need to create
(1)
routers (NAT registers), (2) router ports (LRP) and (3) some static
routes
(LRSR). All these elements are monitored by the DB sync tool and will
fail
because the counterparts in the Neutron DB don't exist. I guess that
you
have some script or tool or at least a document to manually add these resources; sharing it could help to find a solution.
If these elements are manually created during the deployment phase, you can also control the information provided. I'm thinking about the "external_ids" field; we can push some defined constant that will make these registers transparent to the OVN DB sync tool.
As these resources are also created by tools not necessarily controlled by us we need to avoid changing them. I would therefor propose to not add a "ignore" constant to the "external_ids", to to rather check for the existence of the neutron keys in "external_ids" (i think thats also what is described in the bug below).
Please check this idea and if it is feasible. In that case, please add
a
topic to the Neutron drivers meeting agenda [1] to discuss it.
That sounds good, thanks. https://bugs.launchpad.net/neutron/+bug/2027742
From my perspective there is also an additional step after the above REF. Currently it just makes it possible to use ovn-ic without neutron killing it. However if we could get neutron to treat a Transit Switch as a external network we would skip the need to implement the NAT, LRP and LRSR logic in custom tooling and could use the existing neutron code. This would probably require creating a custom type of provider network and then specifying the transit switch name in the provider options.
Is that something that from your perspective we should take a look at right away, or rather first focus und neutron not burning the transit switch.
Regards Felix
Regards, Roberto
Regards.
[1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers
On Wed, Jul 12, 2023 at 2:12 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
Em qua., 12 de jul. de 2023 às 09:05, Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> escreveu:
Hi Rodolfo,
Thanks for the feedback.
I liked the idea of having a different network type only to hold the Transit Switches and thus the LRP but it depends on some factors like multi-tenancy. From the OVN perspective, the TS is global and will only be linked
to a
tenant when it is plugged into the tenant router port. My concern
here is
that if Neutron manages the TS, it will need to assume the dynamic characteristics of this TS function, ovn-ic creates and removes the
TS in
NB_Global, in addition to plugging and removing ports in this switch according to the network topology. Additionally, Neutron would still
need
to handle the learned remote routes (which are listed as static
routes from
the tenant router perspective).
sorry: NB_Global -> Northbound Database.
This is an open discussion, Felix can add ideas here.
In general, it seems to me that we have no alternatives for this
type of
solution other than OVN-IC (note that ovn-bgp-agent does not learn
remote
routes at the SDN level / OVN). OpenStack seems to be designed to run like a "bubble" and this
traffic
from one VM to another always needs to be routed at the FIP level and external routing layers.
Regards, Roberto
Em qua., 12 de jul. de 2023 às 05:11, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Hello Roberto:
We talked about this possible RFE during the PTG [1] but I don't remember having any proposal. Actually what I remember (and was
written in
the etherpad) is that you were going to present an RFE. Can you
explain it
briefly?
We also talked about the idea, proposed by Felix in the mailing
list,
of having a different network type only to hold the Transit
Switches and
thus the LRP. If you have a proposal, please present it.
Regards.
[1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506
On Tue, Jul 11, 2023 at 10:50 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
> Hello Neutron folks, > > Regarding the conversation started in March [1] about the use of
OVN
> interconnect with Neutron, I would like to evolve the discussion in > relation to resources allocated by OVN-IC and which are not
managed by
> Neutron. In the March PTG [2], the problem related to the db_sync
tool was
> presented, and a proposed solution in which Neutron does not
manage these
> resources. > > After discussing some architectural designs with Felix/StackIT, it > seems to make sense to come up with a proposal to change the
mech_driver to
> validate the external_ids key and not remove resources present in
the OVN
> backend without Neutron "signature". > > The discussion here is more complex than it seems, because > architecturally Neutron does not allow us to interconnect
workloads in
> multiple AZs (with different OpenStacks), but this could be a
requirement
> for a high availability cloud solution (Cloud Region).
Additionally, this
> OVN-IC solution allows interconnecting other cloud backends that
use OVN,
> in the case of kubernetes with ovn-kube. > > We tested an OVN interconnect integrated with 3 OpenStack > installations and it worked very well !!! considering direct L3
traffic at
> the router level between different network infrastructures. > > SYNC_REPAIR - problem > > * Static Routes (learned OVN-IC routes) > * Router Port -> Transit Switches > > Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae > neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING > neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync > [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port
found in
> OVN but not in Neutron, port_id=rt2-admin-tenant1 > Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae > neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING > neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync > [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router > 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes
[{'destination': '
> 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': ' > 10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not in > Neutron > > > Any suggestions on how to resolve this db_sync issue? since all
other
> Neutron modules did not present any problem with OVN-IC (as far as
I
> tested). > I remember Terry was keen to suggest some things to help. I believe > that before proposing some complex mechanism to solve this simple
problem,
> I would like to hear the community opinion. In our use case, a bit
change
> in db_sync with filter by neutron key in external_ids would be
enough.
> > Regards, > Roberto > > > > [1] >
https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.ht...
> [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg > > > > Additional logs: > > OpenStack 1 > > root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# > root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# > root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl > lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 > IPv4 Routes > Route Table <main>: > 20.0.1.0/24 169.254.200.2 dst-ip
(learned)
> 20.0.2.0/24 169.254.200.3 dst-ip
(learned)
> 0.0.0.0/0 200.200.200.1 dst-ip > > IPv6 Routes > Route Table <main>: > ::/0 fc00:ca5a:ca5a:8000:: dst-ip > root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl > lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 > IPv4 Routes > Route Table <main>: > 10.0.1.0/24 169.254.100.2 dst-ip
(learned)
> 10.0.2.0/24 169.254.100.3 dst-ip
(learned)
> 0.0.0.0/0 200.200.200.1 dst-ip > > IPv6 Routes > Route Table <main>: > ::/0 fc00:ca5a:ca5a:8000:: dst-ip > root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# > > > OpenStack 2 > > root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl > lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc > IPv4 Routes > Route Table <main>: > 20.0.0.0/24 169.254.200.1 dst-ip
(learned)
> 20.0.2.0/24 169.254.200.3 dst-ip
(learned)
> 0.0.0.0/0 200.200.200.1 dst-ip > > IPv6 Routes > Route Table <main>: > ::/0 fc00:ca5a:ca5a:8000:: dst-ip > root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl > lr-route-list ce45f681-6454-43fe-974f-81344bb8113a > IPv4 Routes > Route Table <main>: > 10.0.0.0/24 169.254.100.1 dst-ip
(learned)
> 10.0.2.0/24 169.254.100.3 dst-ip
(learned)
> 0.0.0.0/0 200.200.200.1 dst-ip > > IPv6 Routes > Route Table <main>: > ::/0 fc00:ca5a:ca5a:8000:: dst-ip > > > > OpenStack 3 > > root@os-infra-1-neutron-ovn-northd-container-f237db97:~# > root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl > lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 > IPv4 Routes > Route Table <main>: > 20.0.0.0/24 169.254.200.1 dst-ip
(learned)
> 20.0.1.0/24 169.254.200.2 dst-ip
(learned)
> 0.0.0.0/0 200.200.200.1 dst-ip > > IPv6 Routes > Route Table <main>: > ::/0 fc00:ca5a:ca5a:8000:: dst-ip > root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl > lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 > IPv4 Routes > Route Table <main>: > 10.0.0.0/24 169.254.100.1 dst-ip
(learned)
> 10.0.1.0/24 169.254.100.2 dst-ip
(learned)
> 0.0.0.0/0 200.200.200.1 dst-ip > > IPv6 Routes > Route Table <main>: > ::/0 fc00:ca5a:ca5a:8000:: dst-ip > > > OVN-IC Global database > > root@ovn-global-db1:~# ovn-ic-sbctl show > availability-zone osp1 > gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 > hostname: osp1-gwnode1 > type: geneve > ip: 192.168.200.28 > port admin-rt1-tenant1 > transit switch: admin-tenant1 > address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24
fe80::1/64"]
> port admin-rt1-tenant1_1 > transit switch: admin-tenant1_1 > address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] > availability-zone osp2 > gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 > hostname: osp2-gwnode1 > type: geneve > ip: 192.168.200.128 > port admin-rt2-tenant1 > transit switch: admin-tenant1 > address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24
fe80::2/64"]
> port admin-rt2-tenant1_1 > transit switch: admin-tenant1_1 > address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] > availability-zone osp3 > gateway 97595af9-7896-40d0-a883-beadbff1aa5b > hostname: osp3-gwnode1 > type: geneve > ip: 192.168.200.228 > port admin-rt3-tenant1 > transit switch: admin-tenant1 > address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24
fe80::3/64"]
> port admin-rt3-tenant1_1 > transit switch: admin-tenant1_1 > address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"] > > > > > > > > *‘Esta mensagem é direcionada apenas para os endereços constantes
no
> cabeçalho inicial. Se você não está listado nos endereços
constantes no
> cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo
dessa
> mensagem e cuja cópia, encaminhamento e/ou execução das ações
citadas estão
> imediatamente anuladas e proibidas’.* > > *‘Apesar do Magazine Luiza tomar todas as precauções razoáveis
para
> assegurar que nenhum vírus esteja presente nesse e-mail, a empresa
não
> poderá aceitar a responsabilidade por quaisquer perdas ou danos
causados
> por esse e-mail ou por seus anexos’.* >
*‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes
no
cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas
estão
imediatamente anuladas e proibidas’.*
*‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos
causados
por esse e-mail ou por seus anexos’.*
--
_‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas
estão
imediatamente anuladas e proibidas’._
- **‘Apesar do Magazine Luiza tomar
todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade
por
quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail.
Hinweise zum Datenschutz finden Sie hierhttps://www.datenschutz.schwarz.
This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail.
Information on data protection can be found here< https://www.datenschutz.schwarz%3E.
Hi Felix and Rodolfo,
Em seg., 17 de jul. de 2023 às 05:39, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Hello Felix:
That's the point: the launchpad bug describes the registers created by the IC configuration but these registers do not have any identification that informs about their relationship with the IC TS.
Rather than modifying the Neutron resources (a network is a generic resource that is used by multiple backends), I would try to create a NB-IC resource map and the corresponding NB registers (NAT, LRP and LRSR). If you can link a NB-IC TS with the corresponding NB resources, then the problem is fixed because knowing them we can explicitly skip them from the DB sync tool.
If that is not the case, then please present an alternative with the suggested idea of creating an external network type.
BTW, I don't understand why you first say that "created by tools not necessarily controlled by us we need to avoid changing them", but then you are talking about refactoring these tools. Are we talking about the tools that create these IC configurations? Can you share them?
I think we are talking about two different cases.
1 - This proposed RFE intends to solve the problem of removing external resources and that is enough for the OVN-IC to work in generic scenarios: multi tenancy, single external provider, multiple external providers, etc. because the OVN-IC will not relate to Neutron's network topology or capabilities.
2- Felix' suggestion is related to integrating the TS to a network on Neutron (some kind of mapping), and in this case, we have a hybrid solution with the TS managed by an external tool, and the management of the NAT, LRP / TS link with the tenant being done directly on Neutron. For this proposal, I believe that the port filter would not be necessary because Neutron would know the resource mappings in the OVN-NB, but it seems to me a more complex solution.
The question about case 2 is related to which level we are talking about modifying the generic network to map the TS. If it is a network provider level, the discussion is one - this case would solve the schedule/lrp-set-gateway-chassis problem as it would be a gateway port. but if it is a tenant' network level linked to a router / no gateway port (the discussion would be another) - does not solve the gateway port problem, but it seems closer to what the OVN-IC proposes to do at the network L3 interconnect level).
BTW: in case the TS is connected to an external network, we would need an additional router for external/internet connectivity not related to the OVN-IC, or maybe create a router with multiple gateway ports (that doesn't exist yet)...
Regards, Roberto
Regards.
On Mon, Jul 17, 2023 at 8:43 AM Felix Huettner felix.huettner@mail.schwarz wrote:
Hi everyone,
On Thu, Jul 13, 2023 at 07:29:51PM -0300, Roberto Bartzen Acosta wrote:
Hi Rodolfo,
Em qui., 13 de jul. de 2023 às 13:45, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Hello Roberto:
I think you have provided a very detailed explanation of the issue
but you
still didn't present any possible solution. I would ask you at least
for a
proposal and for specific details of what is failing in the OVN sync
tool.
For example, if you add a TS between two clouds, you'll need to
create (1)
routers (NAT registers), (2) router ports (LRP) and (3) some static
routes
(LRSR). All these elements are monitored by the DB sync tool and will
fail
because the counterparts in the Neutron DB don't exist. I guess that
you
have some script or tool or at least a document to manually add these resources; sharing it could help to find a solution.
If these elements are manually created during the deployment phase,
you
can also control the information provided. I'm thinking about the "external_ids" field; we can push some defined constant that will make these registers transparent to the OVN DB sync tool.
As these resources are also created by tools not necessarily controlled by us we need to avoid changing them. I would therefor propose to not add a "ignore" constant to the "external_ids", to to rather check for the existence of the neutron keys in "external_ids" (i think thats also what is described in the bug below).
Thanks for the contribution Felix. The idea is not to add (or at least not expect it to be externally added) any identification in the external_ids, Neutron should only filter for DB consistency validation what it knows and has created by Neuron -> Neutron in the external ids key.
Please check this idea and if it is feasible. In that case, please
add a
topic to the Neutron drivers meeting agenda [1] to discuss it.
That sounds good, thanks. https://bugs.launchpad.net/neutron/+bug/2027742
From my perspective there is also an additional step after the above REF. Currently it just makes it possible to use ovn-ic without neutron killing it. However if we could get neutron to treat a Transit Switch as a external network we would skip the need to implement the NAT, LRP and LRSR logic in custom tooling and could use the existing neutron code. This would probably require creating a custom type of provider network and then specifying the transit switch name in the provider options.
Is that something that from your perspective we should take a look at right away, or rather first focus und neutron not burning the transit switch.
Regards Felix
Regards, Roberto
Regards.
[1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers
On Wed, Jul 12, 2023 at 2:12 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
Em qua., 12 de jul. de 2023 às 09:05, Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> escreveu:
Hi Rodolfo,
Thanks for the feedback.
I liked the idea of having a different network type only to hold the Transit Switches and thus the LRP but it depends on some factors
like
multi-tenancy. From the OVN perspective, the TS is global and will only be linked
to a
tenant when it is plugged into the tenant router port. My concern
here is
that if Neutron manages the TS, it will need to assume the dynamic characteristics of this TS function, ovn-ic creates and removes the
TS in
NB_Global, in addition to plugging and removing ports in this switch according to the network topology. Additionally, Neutron would
still need
to handle the learned remote routes (which are listed as static
routes from
the tenant router perspective).
sorry: NB_Global -> Northbound Database.
This is an open discussion, Felix can add ideas here.
In general, it seems to me that we have no alternatives for this
type of
solution other than OVN-IC (note that ovn-bgp-agent does not learn
remote
routes at the SDN level / OVN). OpenStack seems to be designed to run like a "bubble" and this
traffic
from one VM to another always needs to be routed at the FIP level
and
external routing layers.
Regards, Roberto
Em qua., 12 de jul. de 2023 às 05:11, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
> Hello Roberto: > > We talked about this possible RFE during the PTG [1] but I don't > remember having any proposal. Actually what I remember (and was
written in
> the etherpad) is that you were going to present an RFE. Can you
explain it
> briefly? > > We also talked about the idea, proposed by Felix in the mailing
list,
> of having a different network type only to hold the Transit
Switches and
> thus the LRP. If you have a proposal, please present it. > > Regards. > > [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506 > > On Tue, Jul 11, 2023 at 10:50 PM Roberto Bartzen Acosta < > roberto.acosta@luizalabs.com> wrote: > >> Hello Neutron folks, >> >> Regarding the conversation started in March [1] about the use of
OVN
>> interconnect with Neutron, I would like to evolve the discussion
in
>> relation to resources allocated by OVN-IC and which are not
managed by
>> Neutron. In the March PTG [2], the problem related to the db_sync
tool was
>> presented, and a proposed solution in which Neutron does not
manage these
>> resources. >> >> After discussing some architectural designs with Felix/StackIT, it >> seems to make sense to come up with a proposal to change the
mech_driver to
>> validate the external_ids key and not remove resources present in
the OVN
>> backend without Neutron "signature". >> >> The discussion here is more complex than it seems, because >> architecturally Neutron does not allow us to interconnect
workloads in
>> multiple AZs (with different OpenStacks), but this could be a
requirement
>> for a high availability cloud solution (Cloud Region).
Additionally, this
>> OVN-IC solution allows interconnecting other cloud backends that
use OVN,
>> in the case of kubernetes with ovn-kube. >> >> We tested an OVN interconnect integrated with 3 OpenStack >> installations and it worked very well !!! considering direct L3
traffic at
>> the router level between different network infrastructures. >> >> SYNC_REPAIR - problem >> >> * Static Routes (learned OVN-IC routes) >> * Router Port -> Transit Switches >> >> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port
found in
>> OVN but not in Neutron, port_id=rt2-admin-tenant1 >> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router >> 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes
[{'destination': '
>> 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': ' >> 10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not
in
>> Neutron >> >> >> Any suggestions on how to resolve this db_sync issue? since all
other
>> Neutron modules did not present any problem with OVN-IC (as far
as I
>> tested). >> I remember Terry was keen to suggest some things to help. I
believe
>> that before proposing some complex mechanism to solve this simple
problem,
>> I would like to hear the community opinion. In our use case, a
bit change
>> in db_sync with filter by neutron key in external_ids would be
enough.
>> >> Regards, >> Roberto >> >> >> >> [1] >>
https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.ht...
>> [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg >> >> >> >> Additional logs: >> >> OpenStack 1 >> >> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# >> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# >> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
ovn-nbctl
>> lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 >> IPv4 Routes >> Route Table <main>: >> 20.0.1.0/24 169.254.200.2 dst-ip
(learned)
>> 20.0.2.0/24 169.254.200.3 dst-ip
(learned)
>> 0.0.0.0/0 200.200.200.1 dst-ip >> >> IPv6 Routes >> Route Table <main>: >> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
ovn-nbctl
>> lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 >> IPv4 Routes >> Route Table <main>: >> 10.0.1.0/24 169.254.100.2 dst-ip
(learned)
>> 10.0.2.0/24 169.254.100.3 dst-ip
(learned)
>> 0.0.0.0/0 200.200.200.1 dst-ip >> >> IPv6 Routes >> Route Table <main>: >> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# >> >> >> OpenStack 2 >> >> root@os-infra-1-neutron-ovn-northd-container-30f7e935:~#
ovn-nbctl
>> lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc >> IPv4 Routes >> Route Table <main>: >> 20.0.0.0/24 169.254.200.1 dst-ip
(learned)
>> 20.0.2.0/24 169.254.200.3 dst-ip
(learned)
>> 0.0.0.0/0 200.200.200.1 dst-ip >> >> IPv6 Routes >> Route Table <main>: >> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> root@os-infra-1-neutron-ovn-northd-container-30f7e935:~#
ovn-nbctl
>> lr-route-list ce45f681-6454-43fe-974f-81344bb8113a >> IPv4 Routes >> Route Table <main>: >> 10.0.0.0/24 169.254.100.1 dst-ip
(learned)
>> 10.0.2.0/24 169.254.100.3 dst-ip
(learned)
>> 0.0.0.0/0 200.200.200.1 dst-ip >> >> IPv6 Routes >> Route Table <main>: >> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> >> >> >> OpenStack 3 >> >> root@os-infra-1-neutron-ovn-northd-container-f237db97:~# >> root@os-infra-1-neutron-ovn-northd-container-f237db97:~#
ovn-nbctl
>> lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 >> IPv4 Routes >> Route Table <main>: >> 20.0.0.0/24 169.254.200.1 dst-ip
(learned)
>> 20.0.1.0/24 169.254.200.2 dst-ip
(learned)
>> 0.0.0.0/0 200.200.200.1 dst-ip >> >> IPv6 Routes >> Route Table <main>: >> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> root@os-infra-1-neutron-ovn-northd-container-f237db97:~#
ovn-nbctl
>> lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 >> IPv4 Routes >> Route Table <main>: >> 10.0.0.0/24 169.254.100.1 dst-ip
(learned)
>> 10.0.1.0/24 169.254.100.2 dst-ip
(learned)
>> 0.0.0.0/0 200.200.200.1 dst-ip >> >> IPv6 Routes >> Route Table <main>: >> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> >> >> OVN-IC Global database >> >> root@ovn-global-db1:~# ovn-ic-sbctl show >> availability-zone osp1 >> gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 >> hostname: osp1-gwnode1 >> type: geneve >> ip: 192.168.200.28 >> port admin-rt1-tenant1 >> transit switch: admin-tenant1 >> address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24
fe80::1/64"]
>> port admin-rt1-tenant1_1 >> transit switch: admin-tenant1_1 >> address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] >> availability-zone osp2 >> gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 >> hostname: osp2-gwnode1 >> type: geneve >> ip: 192.168.200.128 >> port admin-rt2-tenant1 >> transit switch: admin-tenant1 >> address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24
fe80::2/64"]
>> port admin-rt2-tenant1_1 >> transit switch: admin-tenant1_1 >> address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] >> availability-zone osp3 >> gateway 97595af9-7896-40d0-a883-beadbff1aa5b >> hostname: osp3-gwnode1 >> type: geneve >> ip: 192.168.200.228 >> port admin-rt3-tenant1 >> transit switch: admin-tenant1 >> address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24
fe80::3/64"]
>> port admin-rt3-tenant1_1 >> transit switch: admin-tenant1_1 >> address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"] >> >> >> >> >> >> >> >> *‘Esta mensagem é direcionada apenas para os endereços constantes
no
>> cabeçalho inicial. Se você não está listado nos endereços
constantes no
>> cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo
dessa
>> mensagem e cuja cópia, encaminhamento e/ou execução das ações
citadas estão
>> imediatamente anuladas e proibidas’.* >> >> *‘Apesar do Magazine Luiza tomar todas as precauções razoáveis
para
>> assegurar que nenhum vírus esteja presente nesse e-mail, a
empresa não
>> poderá aceitar a responsabilidade por quaisquer perdas ou danos
causados
>> por esse e-mail ou por seus anexos’.* >> >
*‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes
no
cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo
dessa
mensagem e cuja cópia, encaminhamento e/ou execução das ações
citadas estão
imediatamente anuladas e proibidas’.*
*‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa
não
poderá aceitar a responsabilidade por quaisquer perdas ou danos
causados
por esse e-mail ou por seus anexos’.*
--
_‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas
estão
imediatamente anuladas e proibidas’._
- **‘Apesar do Magazine Luiza tomar
todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade
por
quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail.
Hinweise zum Datenschutz finden Sie hier<https://www.datenschutz.schwarz
.
This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail.
Information on data protection can be found here< https://www.datenschutz.schwarz%3E.
Ok, I'll ask again for the tool/script/manual used to make the IC connection. I will also ask again for the NB-IC database resources and the NB resources needed, and how are related.
If you want Neutron to handle these resources, then you don't need anything special: you can add an external network, handled only by one project. Inside this project you can create the needed resources: networks, routers, etc. This project/user (controlled by the sysadmin), won't create any VM on these networks/routers. Then you can use these resources to connect the TS.
On Mon, Jul 17, 2023 at 3:10 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
Hi Felix and Rodolfo,
Em seg., 17 de jul. de 2023 às 05:39, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Hello Felix:
That's the point: the launchpad bug describes the registers created by the IC configuration but these registers do not have any identification that informs about their relationship with the IC TS.
Rather than modifying the Neutron resources (a network is a generic resource that is used by multiple backends), I would try to create a NB-IC resource map and the corresponding NB registers (NAT, LRP and LRSR). If you can link a NB-IC TS with the corresponding NB resources, then the problem is fixed because knowing them we can explicitly skip them from the DB sync tool.
If that is not the case, then please present an alternative with the suggested idea of creating an external network type.
BTW, I don't understand why you first say that "created by tools not necessarily controlled by us we need to avoid changing them", but then you are talking about refactoring these tools. Are we talking about the tools that create these IC configurations? Can you share them?
I think we are talking about two different cases.
1 - This proposed RFE intends to solve the problem of removing external resources and that is enough for the OVN-IC to work in generic scenarios: multi tenancy, single external provider, multiple external providers, etc. because the OVN-IC will not relate to Neutron's network topology or capabilities.
2- Felix' suggestion is related to integrating the TS to a network on Neutron (some kind of mapping), and in this case, we have a hybrid solution with the TS managed by an external tool, and the management of the NAT, LRP / TS link with the tenant being done directly on Neutron. For this proposal, I believe that the port filter would not be necessary because Neutron would know the resource mappings in the OVN-NB, but it seems to me a more complex solution.
The question about case 2 is related to which level we are talking about modifying the generic network to map the TS. If it is a network provider level, the discussion is one - this case would solve the schedule/lrp-set-gateway-chassis problem as it would be a gateway port. but if it is a tenant' network level linked to a router / no gateway port (the discussion would be another) - does not solve the gateway port problem, but it seems closer to what the OVN-IC proposes to do at the network L3 interconnect level).
BTW: in case the TS is connected to an external network, we would need an additional router for external/internet connectivity not related to the OVN-IC, or maybe create a router with multiple gateway ports (that doesn't exist yet)...
Regards, Roberto
Regards.
On Mon, Jul 17, 2023 at 8:43 AM Felix Huettner felix.huettner@mail.schwarz wrote:
Hi everyone,
On Thu, Jul 13, 2023 at 07:29:51PM -0300, Roberto Bartzen Acosta wrote:
Hi Rodolfo,
Em qui., 13 de jul. de 2023 às 13:45, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Hello Roberto:
I think you have provided a very detailed explanation of the issue
but you
still didn't present any possible solution. I would ask you at least
for a
proposal and for specific details of what is failing in the OVN sync
tool.
For example, if you add a TS between two clouds, you'll need to
create (1)
routers (NAT registers), (2) router ports (LRP) and (3) some static
routes
(LRSR). All these elements are monitored by the DB sync tool and
will fail
because the counterparts in the Neutron DB don't exist. I guess that
you
have some script or tool or at least a document to manually add these resources; sharing it could help to find a solution.
If these elements are manually created during the deployment phase,
you
can also control the information provided. I'm thinking about the "external_ids" field; we can push some defined constant that will
make
these registers transparent to the OVN DB sync tool.
As these resources are also created by tools not necessarily controlled by us we need to avoid changing them. I would therefor propose to not add a "ignore" constant to the "external_ids", to to rather check for the existence of the neutron keys in "external_ids" (i think thats also what is described in the bug below).
Thanks for the contribution Felix. The idea is not to add (or at least not expect it to be externally added) any identification in the external_ids, Neutron should only filter for DB consistency validation what it knows and has created by Neuron -> Neutron in the external ids key.
Please check this idea and if it is feasible. In that case, please
add a
topic to the Neutron drivers meeting agenda [1] to discuss it.
That sounds good, thanks. https://bugs.launchpad.net/neutron/+bug/2027742
From my perspective there is also an additional step after the above REF. Currently it just makes it possible to use ovn-ic without neutron killing it. However if we could get neutron to treat a Transit Switch as a external network we would skip the need to implement the NAT, LRP and LRSR logic in custom tooling and could use the existing neutron code. This would probably require creating a custom type of provider network and then specifying the transit switch name in the provider options.
Is that something that from your perspective we should take a look at right away, or rather first focus und neutron not burning the transit switch.
Regards Felix
Regards, Roberto
Regards.
[1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers
On Wed, Jul 12, 2023 at 2:12 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
Em qua., 12 de jul. de 2023 às 09:05, Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> escreveu:
> Hi Rodolfo, > > Thanks for the feedback. > > I liked the idea of having a different network type only to hold
the
> Transit Switches and thus the LRP but it depends on some factors
like
> multi-tenancy. > From the OVN perspective, the TS is global and will only be linked
to a
> tenant when it is plugged into the tenant router port. My concern
here is
> that if Neutron manages the TS, it will need to assume the dynamic > characteristics of this TS function, ovn-ic creates and removes
the TS in
> NB_Global, in addition to plugging and removing ports in this
switch
> according to the network topology. Additionally, Neutron would
still need
> to handle the learned remote routes (which are listed as static
routes from
> the tenant router perspective). > sorry: NB_Global -> Northbound Database.
> This is an open discussion, Felix can add ideas here. > > In general, it seems to me that we have no alternatives for this
type of
> solution other than OVN-IC (note that ovn-bgp-agent does not learn
remote
> routes at the SDN level / OVN). > OpenStack seems to be designed to run like a "bubble" and this
traffic
> from one VM to another always needs to be routed at the FIP level
and
> external routing layers. > > Regards, > Roberto > > Em qua., 12 de jul. de 2023 às 05:11, Rodolfo Alonso Hernandez < > ralonsoh@redhat.com> escreveu: > >> Hello Roberto: >> >> We talked about this possible RFE during the PTG [1] but I don't >> remember having any proposal. Actually what I remember (and was
written in
>> the etherpad) is that you were going to present an RFE. Can you
explain it
>> briefly? >> >> We also talked about the idea, proposed by Felix in the mailing
list,
>> of having a different network type only to hold the Transit
Switches and
>> thus the LRP. If you have a proposal, please present it. >> >> Regards. >> >> [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506 >> >> On Tue, Jul 11, 2023 at 10:50 PM Roberto Bartzen Acosta < >> roberto.acosta@luizalabs.com> wrote: >> >>> Hello Neutron folks, >>> >>> Regarding the conversation started in March [1] about the use of
OVN
>>> interconnect with Neutron, I would like to evolve the discussion
in
>>> relation to resources allocated by OVN-IC and which are not
managed by
>>> Neutron. In the March PTG [2], the problem related to the
db_sync tool was
>>> presented, and a proposed solution in which Neutron does not
manage these
>>> resources. >>> >>> After discussing some architectural designs with Felix/StackIT,
it
>>> seems to make sense to come up with a proposal to change the
mech_driver to
>>> validate the external_ids key and not remove resources present
in the OVN
>>> backend without Neutron "signature". >>> >>> The discussion here is more complex than it seems, because >>> architecturally Neutron does not allow us to interconnect
workloads in
>>> multiple AZs (with different OpenStacks), but this could be a
requirement
>>> for a high availability cloud solution (Cloud Region).
Additionally, this
>>> OVN-IC solution allows interconnecting other cloud backends that
use OVN,
>>> in the case of kubernetes with ovn-kube. >>> >>> We tested an OVN interconnect integrated with 3 OpenStack >>> installations and it worked very well !!! considering direct L3
traffic at
>>> the router level between different network infrastructures. >>> >>> SYNC_REPAIR - problem >>> >>> * Static Routes (learned OVN-IC routes) >>> * Router Port -> Transit Switches >>> >>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port
found in
>>> OVN but not in Neutron, port_id=rt2-admin-tenant1 >>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router >>> 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes
[{'destination': '
>>> 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': ' >>> 10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not
in
>>> Neutron >>> >>> >>> Any suggestions on how to resolve this db_sync issue? since all
other
>>> Neutron modules did not present any problem with OVN-IC (as far
as I
>>> tested). >>> I remember Terry was keen to suggest some things to help. I
believe
>>> that before proposing some complex mechanism to solve this
simple problem,
>>> I would like to hear the community opinion. In our use case, a
bit change
>>> in db_sync with filter by neutron key in external_ids would be
enough.
>>> >>> Regards, >>> Roberto >>> >>> >>> >>> [1] >>>
https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.ht...
>>> [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg >>> >>> >>> >>> Additional logs: >>> >>> OpenStack 1 >>> >>> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
ovn-nbctl
>>> lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 >>> IPv4 Routes >>> Route Table <main>: >>> 20.0.1.0/24 169.254.200.2 dst-ip
(learned)
>>> 20.0.2.0/24 169.254.200.3 dst-ip
(learned)
>>> 0.0.0.0/0 200.200.200.1 dst-ip >>> >>> IPv6 Routes >>> Route Table <main>: >>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
ovn-nbctl
>>> lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 >>> IPv4 Routes >>> Route Table <main>: >>> 10.0.1.0/24 169.254.100.2 dst-ip
(learned)
>>> 10.0.2.0/24 169.254.100.3 dst-ip
(learned)
>>> 0.0.0.0/0 200.200.200.1 dst-ip >>> >>> IPv6 Routes >>> Route Table <main>: >>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>> >>> >>> OpenStack 2 >>> >>> root@os-infra-1-neutron-ovn-northd-container-30f7e935:~#
ovn-nbctl
>>> lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc >>> IPv4 Routes >>> Route Table <main>: >>> 20.0.0.0/24 169.254.200.1 dst-ip
(learned)
>>> 20.0.2.0/24 169.254.200.3 dst-ip
(learned)
>>> 0.0.0.0/0 200.200.200.1 dst-ip >>> >>> IPv6 Routes >>> Route Table <main>: >>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>> root@os-infra-1-neutron-ovn-northd-container-30f7e935:~#
ovn-nbctl
>>> lr-route-list ce45f681-6454-43fe-974f-81344bb8113a >>> IPv4 Routes >>> Route Table <main>: >>> 10.0.0.0/24 169.254.100.1 dst-ip
(learned)
>>> 10.0.2.0/24 169.254.100.3 dst-ip
(learned)
>>> 0.0.0.0/0 200.200.200.1 dst-ip >>> >>> IPv6 Routes >>> Route Table <main>: >>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>> >>> >>> >>> OpenStack 3 >>> >>> root@os-infra-1-neutron-ovn-northd-container-f237db97:~# >>> root@os-infra-1-neutron-ovn-northd-container-f237db97:~#
ovn-nbctl
>>> lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 >>> IPv4 Routes >>> Route Table <main>: >>> 20.0.0.0/24 169.254.200.1 dst-ip
(learned)
>>> 20.0.1.0/24 169.254.200.2 dst-ip
(learned)
>>> 0.0.0.0/0 200.200.200.1 dst-ip >>> >>> IPv6 Routes >>> Route Table <main>: >>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>> root@os-infra-1-neutron-ovn-northd-container-f237db97:~#
ovn-nbctl
>>> lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 >>> IPv4 Routes >>> Route Table <main>: >>> 10.0.0.0/24 169.254.100.1 dst-ip
(learned)
>>> 10.0.1.0/24 169.254.100.2 dst-ip
(learned)
>>> 0.0.0.0/0 200.200.200.1 dst-ip >>> >>> IPv6 Routes >>> Route Table <main>: >>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>> >>> >>> OVN-IC Global database >>> >>> root@ovn-global-db1:~# ovn-ic-sbctl show >>> availability-zone osp1 >>> gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 >>> hostname: osp1-gwnode1 >>> type: geneve >>> ip: 192.168.200.28 >>> port admin-rt1-tenant1 >>> transit switch: admin-tenant1 >>> address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24
fe80::1/64"]
>>> port admin-rt1-tenant1_1 >>> transit switch: admin-tenant1_1 >>> address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] >>> availability-zone osp2 >>> gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 >>> hostname: osp2-gwnode1 >>> type: geneve >>> ip: 192.168.200.128 >>> port admin-rt2-tenant1 >>> transit switch: admin-tenant1 >>> address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24
fe80::2/64"]
>>> port admin-rt2-tenant1_1 >>> transit switch: admin-tenant1_1 >>> address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] >>> availability-zone osp3 >>> gateway 97595af9-7896-40d0-a883-beadbff1aa5b >>> hostname: osp3-gwnode1 >>> type: geneve >>> ip: 192.168.200.228 >>> port admin-rt3-tenant1 >>> transit switch: admin-tenant1 >>> address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24
fe80::3/64"]
>>> port admin-rt3-tenant1_1 >>> transit switch: admin-tenant1_1 >>> address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"] >>> >>> >>> >>> >>> >>> >>> >>> *‘Esta mensagem é direcionada apenas para os endereços
constantes no
>>> cabeçalho inicial. Se você não está listado nos endereços
constantes no
>>> cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo
dessa
>>> mensagem e cuja cópia, encaminhamento e/ou execução das ações
citadas estão
>>> imediatamente anuladas e proibidas’.* >>> >>> *‘Apesar do Magazine Luiza tomar todas as precauções razoáveis
para
>>> assegurar que nenhum vírus esteja presente nesse e-mail, a
empresa não
>>> poderá aceitar a responsabilidade por quaisquer perdas ou danos
causados
>>> por esse e-mail ou por seus anexos’.* >>> >>
*‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços
constantes no
cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo
dessa
mensagem e cuja cópia, encaminhamento e/ou execução das ações
citadas estão
imediatamente anuladas e proibidas’.*
*‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa
não
poderá aceitar a responsabilidade por quaisquer perdas ou danos
causados
por esse e-mail ou por seus anexos’.*
--
_‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas
estão
imediatamente anuladas e proibidas’._
- **‘Apesar do Magazine Luiza tomar
todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade
por
quaisquer perdas ou danos causados por esse e-mail ou por seus
anexos’.*
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail.
Hinweise zum Datenschutz finden Sie hier<https://www.datenschutz.schwarz
.
This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail.
Information on data protection can be found here< https://www.datenschutz.schwarz%3E.
*‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.*
*‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
Hi Rodolfo,
Em seg., 17 de jul. de 2023 às 11:30, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Ok, I'll ask again for the tool/script/manual used to make the IC connection. I will also ask again for the NB-IC database resources and the NB resources needed, and how are related.
To be clear, this tool doesn't exist yet and we are working on the design of the software solution [1]. For now, the tests were performed using manual commands, such as those described in the RFE link and very similar to the case applied by the ovn-kube to use OVN-IC with OpenStack - Neutron [2]. The main idea is to develop an external layer interface with an API/SDK and an internal layer interface with the OVN-IC-NB global database and the OpenStack/AZ OVN-NB database [1].
[1] https://drive.google.com/file/d/1yx66YcxNMH_oB11--0-_0eZzmBJHu-Ox/view?usp=s... [2] https://github.com/kubeovn/kube-ovn/blob/v1.11.0/docs/cluster-interconnectio...
If you want Neutron to handle these resources, then you don't need anything special: you can add an external network, handled only by one project. Inside this project you can create the needed resources: networks, routers, etc. This project/user (controlled by the sysadmin), won't create any VM on these networks/routers. Then you can use these resources to connect the TS.
In my opinion the answer would be: yes and no, this approach would work for isolated cases, but think on a larger scale: how many external networks would need to be managed at the IP Fabric level? what is the service type? vlan (we would have segmentation limits) vxlan (we would have to extend the L2 domain across the entire IP fabric) virtual? . I don't know, there are many questions here. Felix can explain better and propose an RFE in the future. Even so, Neutron would need a filter to not delete the static routes learned by the router (the different proposals can be complementary...)
About the proposed RFE, do you see any problem if Neutron deletes from the OVN only what it manages?
On Mon, Jul 17, 2023 at 3:10 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
Hi Felix and Rodolfo,
Em seg., 17 de jul. de 2023 às 05:39, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Hello Felix:
That's the point: the launchpad bug describes the registers created by the IC configuration but these registers do not have any identification that informs about their relationship with the IC TS.
Rather than modifying the Neutron resources (a network is a generic resource that is used by multiple backends), I would try to create a NB-IC resource map and the corresponding NB registers (NAT, LRP and LRSR). If you can link a NB-IC TS with the corresponding NB resources, then the problem is fixed because knowing them we can explicitly skip them from the DB sync tool.
If that is not the case, then please present an alternative with the suggested idea of creating an external network type.
BTW, I don't understand why you first say that "created by tools not necessarily controlled by us we need to avoid changing them", but then you are talking about refactoring these tools. Are we talking about the tools that create these IC configurations? Can you share them?
I think we are talking about two different cases.
1 - This proposed RFE intends to solve the problem of removing external resources and that is enough for the OVN-IC to work in generic scenarios: multi tenancy, single external provider, multiple external providers, etc. because the OVN-IC will not relate to Neutron's network topology or capabilities.
2- Felix' suggestion is related to integrating the TS to a network on Neutron (some kind of mapping), and in this case, we have a hybrid solution with the TS managed by an external tool, and the management of the NAT, LRP / TS link with the tenant being done directly on Neutron. For this proposal, I believe that the port filter would not be necessary because Neutron would know the resource mappings in the OVN-NB, but it seems to me a more complex solution.
The question about case 2 is related to which level we are talking about modifying the generic network to map the TS. If it is a network provider level, the discussion is one - this case would solve the schedule/lrp-set-gateway-chassis problem as it would be a gateway port. but if it is a tenant' network level linked to a router / no gateway port (the discussion would be another) - does not solve the gateway port problem, but it seems closer to what the OVN-IC proposes to do at the network L3 interconnect level).
BTW: in case the TS is connected to an external network, we would need an additional router for external/internet connectivity not related to the OVN-IC, or maybe create a router with multiple gateway ports (that doesn't exist yet)...
Regards, Roberto
Regards.
On Mon, Jul 17, 2023 at 8:43 AM Felix Huettner felix.huettner@mail.schwarz wrote:
Hi everyone,
On Thu, Jul 13, 2023 at 07:29:51PM -0300, Roberto Bartzen Acosta wrote:
Hi Rodolfo,
Em qui., 13 de jul. de 2023 às 13:45, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Hello Roberto:
I think you have provided a very detailed explanation of the issue
but you
still didn't present any possible solution. I would ask you at
least for a
proposal and for specific details of what is failing in the OVN
sync tool.
For example, if you add a TS between two clouds, you'll need to
create (1)
routers (NAT registers), (2) router ports (LRP) and (3) some static
routes
(LRSR). All these elements are monitored by the DB sync tool and
will fail
because the counterparts in the Neutron DB don't exist. I guess
that you
have some script or tool or at least a document to manually add
these
resources; sharing it could help to find a solution.
If these elements are manually created during the deployment phase,
you
can also control the information provided. I'm thinking about the "external_ids" field; we can push some defined constant that will
make
these registers transparent to the OVN DB sync tool.
As these resources are also created by tools not necessarily controlled by us we need to avoid changing them. I would therefor propose to not add a "ignore" constant to the "external_ids", to to rather check for the existence of the neutron keys in "external_ids" (i think thats also what is described in the bug below).
Thanks for the contribution Felix. The idea is not to add (or at least not expect it to be externally added) any identification in the external_ids, Neutron should only filter for DB consistency validation what it knows and has created by Neuron -> Neutron in the external ids key.
Please check this idea and if it is feasible. In that case, please
add a
topic to the Neutron drivers meeting agenda [1] to discuss it.
That sounds good, thanks. https://bugs.launchpad.net/neutron/+bug/2027742
From my perspective there is also an additional step after the above REF. Currently it just makes it possible to use ovn-ic without neutron killing it. However if we could get neutron to treat a Transit Switch as a external network we would skip the need to implement the NAT, LRP and LRSR logic in custom tooling and could use the existing neutron code. This would probably require creating a custom type of provider network and then specifying the transit switch name in the provider options.
Is that something that from your perspective we should take a look at right away, or rather first focus und neutron not burning the transit switch.
Regards Felix
Regards, Roberto
Regards.
[1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers
On Wed, Jul 12, 2023 at 2:12 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
> > > Em qua., 12 de jul. de 2023 às 09:05, Roberto Bartzen Acosta < > roberto.acosta@luizalabs.com> escreveu: > >> Hi Rodolfo, >> >> Thanks for the feedback. >> >> I liked the idea of having a different network type only to hold
the
>> Transit Switches and thus the LRP but it depends on some factors
like
>> multi-tenancy. >> From the OVN perspective, the TS is global and will only be
linked to a
>> tenant when it is plugged into the tenant router port. My concern
here is
>> that if Neutron manages the TS, it will need to assume the dynamic >> characteristics of this TS function, ovn-ic creates and removes
the TS in
>> NB_Global, in addition to plugging and removing ports in this
switch
>> according to the network topology. Additionally, Neutron would
still need
>> to handle the learned remote routes (which are listed as static
routes from
>> the tenant router perspective). >> > sorry: NB_Global -> Northbound Database. > > >> This is an open discussion, Felix can add ideas here. >> >> In general, it seems to me that we have no alternatives for this
type of
>> solution other than OVN-IC (note that ovn-bgp-agent does not
learn remote
>> routes at the SDN level / OVN). >> OpenStack seems to be designed to run like a "bubble" and this
traffic
>> from one VM to another always needs to be routed at the FIP level
and
>> external routing layers. >> >> Regards, >> Roberto >> >> Em qua., 12 de jul. de 2023 às 05:11, Rodolfo Alonso Hernandez < >> ralonsoh@redhat.com> escreveu: >> >>> Hello Roberto: >>> >>> We talked about this possible RFE during the PTG [1] but I don't >>> remember having any proposal. Actually what I remember (and was
written in
>>> the etherpad) is that you were going to present an RFE. Can you
explain it
>>> briefly? >>> >>> We also talked about the idea, proposed by Felix in the mailing
list,
>>> of having a different network type only to hold the Transit
Switches and
>>> thus the LRP. If you have a proposal, please present it. >>> >>> Regards. >>> >>> [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506 >>> >>> On Tue, Jul 11, 2023 at 10:50 PM Roberto Bartzen Acosta < >>> roberto.acosta@luizalabs.com> wrote: >>> >>>> Hello Neutron folks, >>>> >>>> Regarding the conversation started in March [1] about the use
of OVN
>>>> interconnect with Neutron, I would like to evolve the
discussion in
>>>> relation to resources allocated by OVN-IC and which are not
managed by
>>>> Neutron. In the March PTG [2], the problem related to the
db_sync tool was
>>>> presented, and a proposed solution in which Neutron does not
manage these
>>>> resources. >>>> >>>> After discussing some architectural designs with Felix/StackIT,
it
>>>> seems to make sense to come up with a proposal to change the
mech_driver to
>>>> validate the external_ids key and not remove resources present
in the OVN
>>>> backend without Neutron "signature". >>>> >>>> The discussion here is more complex than it seems, because >>>> architecturally Neutron does not allow us to interconnect
workloads in
>>>> multiple AZs (with different OpenStacks), but this could be a
requirement
>>>> for a high availability cloud solution (Cloud Region).
Additionally, this
>>>> OVN-IC solution allows interconnecting other cloud backends
that use OVN,
>>>> in the case of kubernetes with ovn-kube. >>>> >>>> We tested an OVN interconnect integrated with 3 OpenStack >>>> installations and it worked very well !!! considering direct L3
traffic at
>>>> the router level between different network infrastructures. >>>> >>>> SYNC_REPAIR - problem >>>> >>>> * Static Routes (learned OVN-IC routes) >>>> * Router Port -> Transit Switches >>>> >>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router
Port found in
>>>> OVN but not in Neutron, port_id=rt2-admin-tenant1 >>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router >>>> 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes
[{'destination': '
>>>> 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': ' >>>> 10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but
not in
>>>> Neutron >>>> >>>> >>>> Any suggestions on how to resolve this db_sync issue? since all
other
>>>> Neutron modules did not present any problem with OVN-IC (as far
as I
>>>> tested). >>>> I remember Terry was keen to suggest some things to help. I
believe
>>>> that before proposing some complex mechanism to solve this
simple problem,
>>>> I would like to hear the community opinion. In our use case, a
bit change
>>>> in db_sync with filter by neutron key in external_ids would be
enough.
>>>> >>>> Regards, >>>> Roberto >>>> >>>> >>>> >>>> [1] >>>>
https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.ht...
>>>> [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg >>>> >>>> >>>> >>>> Additional logs: >>>> >>>> OpenStack 1 >>>> >>>> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
ovn-nbctl
>>>> lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 >>>> IPv4 Routes >>>> Route Table <main>: >>>> 20.0.1.0/24 169.254.200.2 dst-ip
(learned)
>>>> 20.0.2.0/24 169.254.200.3 dst-ip
(learned)
>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>> >>>> IPv6 Routes >>>> Route Table <main>: >>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
ovn-nbctl
>>>> lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 >>>> IPv4 Routes >>>> Route Table <main>: >>>> 10.0.1.0/24 169.254.100.2 dst-ip
(learned)
>>>> 10.0.2.0/24 169.254.100.3 dst-ip
(learned)
>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>> >>>> IPv6 Routes >>>> Route Table <main>: >>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>> >>>> >>>> OpenStack 2 >>>> >>>> root@os-infra-1-neutron-ovn-northd-container-30f7e935:~#
ovn-nbctl
>>>> lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc >>>> IPv4 Routes >>>> Route Table <main>: >>>> 20.0.0.0/24 169.254.200.1 dst-ip
(learned)
>>>> 20.0.2.0/24 169.254.200.3 dst-ip
(learned)
>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>> >>>> IPv6 Routes >>>> Route Table <main>: >>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>> root@os-infra-1-neutron-ovn-northd-container-30f7e935:~#
ovn-nbctl
>>>> lr-route-list ce45f681-6454-43fe-974f-81344bb8113a >>>> IPv4 Routes >>>> Route Table <main>: >>>> 10.0.0.0/24 169.254.100.1 dst-ip
(learned)
>>>> 10.0.2.0/24 169.254.100.3 dst-ip
(learned)
>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>> >>>> IPv6 Routes >>>> Route Table <main>: >>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>> >>>> >>>> >>>> OpenStack 3 >>>> >>>> root@os-infra-1-neutron-ovn-northd-container-f237db97:~# >>>> root@os-infra-1-neutron-ovn-northd-container-f237db97:~#
ovn-nbctl
>>>> lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 >>>> IPv4 Routes >>>> Route Table <main>: >>>> 20.0.0.0/24 169.254.200.1 dst-ip
(learned)
>>>> 20.0.1.0/24 169.254.200.2 dst-ip
(learned)
>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>> >>>> IPv6 Routes >>>> Route Table <main>: >>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>> root@os-infra-1-neutron-ovn-northd-container-f237db97:~#
ovn-nbctl
>>>> lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 >>>> IPv4 Routes >>>> Route Table <main>: >>>> 10.0.0.0/24 169.254.100.1 dst-ip
(learned)
>>>> 10.0.1.0/24 169.254.100.2 dst-ip
(learned)
>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>> >>>> IPv6 Routes >>>> Route Table <main>: >>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>> >>>> >>>> OVN-IC Global database >>>> >>>> root@ovn-global-db1:~# ovn-ic-sbctl show >>>> availability-zone osp1 >>>> gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 >>>> hostname: osp1-gwnode1 >>>> type: geneve >>>> ip: 192.168.200.28 >>>> port admin-rt1-tenant1 >>>> transit switch: admin-tenant1 >>>> address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24
fe80::1/64"]
>>>> port admin-rt1-tenant1_1 >>>> transit switch: admin-tenant1_1 >>>> address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] >>>> availability-zone osp2 >>>> gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 >>>> hostname: osp2-gwnode1 >>>> type: geneve >>>> ip: 192.168.200.128 >>>> port admin-rt2-tenant1 >>>> transit switch: admin-tenant1 >>>> address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24
fe80::2/64"]
>>>> port admin-rt2-tenant1_1 >>>> transit switch: admin-tenant1_1 >>>> address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] >>>> availability-zone osp3 >>>> gateway 97595af9-7896-40d0-a883-beadbff1aa5b >>>> hostname: osp3-gwnode1 >>>> type: geneve >>>> ip: 192.168.200.228 >>>> port admin-rt3-tenant1 >>>> transit switch: admin-tenant1 >>>> address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24
fe80::3/64"]
>>>> port admin-rt3-tenant1_1 >>>> transit switch: admin-tenant1_1 >>>> address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"] >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> *‘Esta mensagem é direcionada apenas para os endereços
constantes no
>>>> cabeçalho inicial. Se você não está listado nos endereços
constantes no
>>>> cabeçalho, pedimos-lhe que desconsidere completamente o
conteúdo dessa
>>>> mensagem e cuja cópia, encaminhamento e/ou execução das ações
citadas estão
>>>> imediatamente anuladas e proibidas’.* >>>> >>>> *‘Apesar do Magazine Luiza tomar todas as precauções razoáveis
para
>>>> assegurar que nenhum vírus esteja presente nesse e-mail, a
empresa não
>>>> poderá aceitar a responsabilidade por quaisquer perdas ou danos
causados
>>>> por esse e-mail ou por seus anexos’.* >>>> >>> > > *‘Esta mensagem é direcionada apenas para os endereços constantes
no
> cabeçalho inicial. Se você não está listado nos endereços
constantes no
> cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo
dessa
> mensagem e cuja cópia, encaminhamento e/ou execução das ações
citadas estão
> imediatamente anuladas e proibidas’.* > > *‘Apesar do Magazine Luiza tomar todas as precauções razoáveis
para
> assegurar que nenhum vírus esteja presente nesse e-mail, a empresa
não
> poderá aceitar a responsabilidade por quaisquer perdas ou danos
causados
> por esse e-mail ou por seus anexos’.* >
--
_‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes
no
cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas
estão
imediatamente anuladas e proibidas’._
- **‘Apesar do Magazine Luiza tomar
todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a
responsabilidade por
quaisquer perdas ou danos causados por esse e-mail ou por seus
anexos’.*
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail.
Hinweise zum Datenschutz finden Sie hier< https://www.datenschutz.schwarz%3E.
This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail.
Information on data protection can be found here< https://www.datenschutz.schwarz%3E.
*‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.*
*‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
In my opinion the answer would be: yes and no, this approach would work
for isolated cases, but think on a larger scale: how many external networks would need to be managed at the IP Fabric level? This is exactly what you are proposing in your previous mails: "Felix' suggestion is related to integrating the TS to a network on Neutron (some kind of mapping)". So at this point I'm curious about your proposal. And about the static routes, if these routes are created from a Neutron defined router, as I proposed, they won't be deleted.
About the proposed RFE, do you see any problem if Neutron deletes from
the OVN only what it manages? Yes, a huge one: OVN is a service for Neutron, same as RabbitMQ or MySQL. These services are 100% controlled by Neutron because the resources can clash. You can't manually create an IPAM register in the DB because that will disable one IP address. The same for OVN: you can't manually create a LRP in a Logical Switch that provides a Neutron network because this LRP will have an IP address. If a new Neutron port is created and it uses the same IP address of the existing LRP (that doesn't exist in Neutron), the new port will be considered virtual and won't be able to be used in a VM (this is an actual problem I found in a customer last week). So no, Neutron should not allow other resources in OVN apart from what Neutron creates.
On Mon, Jul 17, 2023 at 6:13 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
Hi Rodolfo,
Em seg., 17 de jul. de 2023 às 11:30, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Ok, I'll ask again for the tool/script/manual used to make the IC connection. I will also ask again for the NB-IC database resources and the NB resources needed, and how are related.
To be clear, this tool doesn't exist yet and we are working on the design of the software solution [1]. For now, the tests were performed using manual commands, such as those described in the RFE link and very similar to the case applied by the ovn-kube to use OVN-IC with OpenStack - Neutron [2]. The main idea is to develop an external layer interface with an API/SDK and an internal layer interface with the OVN-IC-NB global database and the OpenStack/AZ OVN-NB database [1].
[1] https://drive.google.com/file/d/1yx66YcxNMH_oB11--0-_0eZzmBJHu-Ox/view?usp=s... [2] https://github.com/kubeovn/kube-ovn/blob/v1.11.0/docs/cluster-interconnectio...
If you want Neutron to handle these resources, then you don't need anything special: you can add an external network, handled only by one project. Inside this project you can create the needed resources: networks, routers, etc. This project/user (controlled by the sysadmin), won't create any VM on these networks/routers. Then you can use these resources to connect the TS.
In my opinion the answer would be: yes and no, this approach would work for isolated cases, but think on a larger scale: how many external networks would need to be managed at the IP Fabric level? what is the service type? vlan (we would have segmentation limits) vxlan (we would have to extend the L2 domain across the entire IP fabric) virtual? . I don't know, there are many questions here. Felix can explain better and propose an RFE in the future. Even so, Neutron would need a filter to not delete the static routes learned by the router (the different proposals can be complementary...)
About the proposed RFE, do you see any problem if Neutron deletes from the OVN only what it manages?
On Mon, Jul 17, 2023 at 3:10 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
Hi Felix and Rodolfo,
Em seg., 17 de jul. de 2023 às 05:39, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Hello Felix:
That's the point: the launchpad bug describes the registers created by the IC configuration but these registers do not have any identification that informs about their relationship with the IC TS.
Rather than modifying the Neutron resources (a network is a generic resource that is used by multiple backends), I would try to create a NB-IC resource map and the corresponding NB registers (NAT, LRP and LRSR). If you can link a NB-IC TS with the corresponding NB resources, then the problem is fixed because knowing them we can explicitly skip them from the DB sync tool.
If that is not the case, then please present an alternative with the suggested idea of creating an external network type.
BTW, I don't understand why you first say that "created by tools not necessarily controlled by us we need to avoid changing them", but then you are talking about refactoring these tools. Are we talking about the tools that create these IC configurations? Can you share them?
I think we are talking about two different cases.
1 - This proposed RFE intends to solve the problem of removing external resources and that is enough for the OVN-IC to work in generic scenarios: multi tenancy, single external provider, multiple external providers, etc. because the OVN-IC will not relate to Neutron's network topology or capabilities.
2- Felix' suggestion is related to integrating the TS to a network on Neutron (some kind of mapping), and in this case, we have a hybrid solution with the TS managed by an external tool, and the management of the NAT, LRP / TS link with the tenant being done directly on Neutron. For this proposal, I believe that the port filter would not be necessary because Neutron would know the resource mappings in the OVN-NB, but it seems to me a more complex solution.
The question about case 2 is related to which level we are talking about modifying the generic network to map the TS. If it is a network provider level, the discussion is one - this case would solve the schedule/lrp-set-gateway-chassis problem as it would be a gateway port. but if it is a tenant' network level linked to a router / no gateway port (the discussion would be another) - does not solve the gateway port problem, but it seems closer to what the OVN-IC proposes to do at the network L3 interconnect level).
BTW: in case the TS is connected to an external network, we would need an additional router for external/internet connectivity not related to the OVN-IC, or maybe create a router with multiple gateway ports (that doesn't exist yet)...
Regards, Roberto
Regards.
On Mon, Jul 17, 2023 at 8:43 AM Felix Huettner felix.huettner@mail.schwarz wrote:
Hi everyone,
On Thu, Jul 13, 2023 at 07:29:51PM -0300, Roberto Bartzen Acosta wrote:
Hi Rodolfo,
Em qui., 13 de jul. de 2023 às 13:45, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
> Hello Roberto: > > I think you have provided a very detailed explanation of the issue
but you
> still didn't present any possible solution. I would ask you at
least for a
> proposal and for specific details of what is failing in the OVN
sync tool.
> > For example, if you add a TS between two clouds, you'll need to
create (1)
> routers (NAT registers), (2) router ports (LRP) and (3) some
static routes
> (LRSR). All these elements are monitored by the DB sync tool and
will fail
> because the counterparts in the Neutron DB don't exist. I guess
that you
> have some script or tool or at least a document to manually add
these
> resources; sharing it could help to find a solution. > > If these elements are manually created during the deployment
phase, you
> can also control the information provided. I'm thinking about the > "external_ids" field; we can push some defined constant that will
make
> these registers transparent to the OVN DB sync tool.
As these resources are also created by tools not necessarily controlled by us we need to avoid changing them. I would therefor propose to not add a "ignore" constant to the "external_ids", to to rather check for the existence of the neutron keys in "external_ids" (i think thats also what is described in the bug below).
Thanks for the contribution Felix. The idea is not to add (or at least not expect it to be externally added) any identification in the external_ids, Neutron should only filter for DB consistency validation what it knows and has created by Neuron -> Neutron in the external ids key.
> Please check this idea and if it is feasible. In that case, please
add a
> topic to the Neutron drivers meeting agenda [1] to discuss it. >
That sounds good, thanks. https://bugs.launchpad.net/neutron/+bug/2027742
From my perspective there is also an additional step after the above REF. Currently it just makes it possible to use ovn-ic without neutron killing it. However if we could get neutron to treat a Transit Switch as a external network we would skip the need to implement the NAT, LRP and LRSR logic in custom tooling and could use the existing neutron code. This would probably require creating a custom type of provider network and then specifying the transit switch name in the provider options.
Is that something that from your perspective we should take a look at right away, or rather first focus und neutron not burning the transit switch.
Regards Felix
Regards, Roberto
> > Regards. > > [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers > > > On Wed, Jul 12, 2023 at 2:12 PM Roberto Bartzen Acosta < > roberto.acosta@luizalabs.com> wrote: > >> >> >> Em qua., 12 de jul. de 2023 às 09:05, Roberto Bartzen Acosta < >> roberto.acosta@luizalabs.com> escreveu: >> >>> Hi Rodolfo, >>> >>> Thanks for the feedback. >>> >>> I liked the idea of having a different network type only to hold
the
>>> Transit Switches and thus the LRP but it depends on some factors
like
>>> multi-tenancy. >>> From the OVN perspective, the TS is global and will only be
linked to a
>>> tenant when it is plugged into the tenant router port. My
concern here is
>>> that if Neutron manages the TS, it will need to assume the
dynamic
>>> characteristics of this TS function, ovn-ic creates and removes
the TS in
>>> NB_Global, in addition to plugging and removing ports in this
switch
>>> according to the network topology. Additionally, Neutron would
still need
>>> to handle the learned remote routes (which are listed as static
routes from
>>> the tenant router perspective). >>> >> sorry: NB_Global -> Northbound Database. >> >> >>> This is an open discussion, Felix can add ideas here. >>> >>> In general, it seems to me that we have no alternatives for this
type of
>>> solution other than OVN-IC (note that ovn-bgp-agent does not
learn remote
>>> routes at the SDN level / OVN). >>> OpenStack seems to be designed to run like a "bubble" and this
traffic
>>> from one VM to another always needs to be routed at the FIP
level and
>>> external routing layers. >>> >>> Regards, >>> Roberto >>> >>> Em qua., 12 de jul. de 2023 às 05:11, Rodolfo Alonso Hernandez < >>> ralonsoh@redhat.com> escreveu: >>> >>>> Hello Roberto: >>>> >>>> We talked about this possible RFE during the PTG [1] but I don't >>>> remember having any proposal. Actually what I remember (and was
written in
>>>> the etherpad) is that you were going to present an RFE. Can you
explain it
>>>> briefly? >>>> >>>> We also talked about the idea, proposed by Felix in the mailing
list,
>>>> of having a different network type only to hold the Transit
Switches and
>>>> thus the LRP. If you have a proposal, please present it. >>>> >>>> Regards. >>>> >>>> [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506 >>>> >>>> On Tue, Jul 11, 2023 at 10:50 PM Roberto Bartzen Acosta < >>>> roberto.acosta@luizalabs.com> wrote: >>>> >>>>> Hello Neutron folks, >>>>> >>>>> Regarding the conversation started in March [1] about the use
of OVN
>>>>> interconnect with Neutron, I would like to evolve the
discussion in
>>>>> relation to resources allocated by OVN-IC and which are not
managed by
>>>>> Neutron. In the March PTG [2], the problem related to the
db_sync tool was
>>>>> presented, and a proposed solution in which Neutron does not
manage these
>>>>> resources. >>>>> >>>>> After discussing some architectural designs with
Felix/StackIT, it
>>>>> seems to make sense to come up with a proposal to change the
mech_driver to
>>>>> validate the external_ids key and not remove resources present
in the OVN
>>>>> backend without Neutron "signature". >>>>> >>>>> The discussion here is more complex than it seems, because >>>>> architecturally Neutron does not allow us to interconnect
workloads in
>>>>> multiple AZs (with different OpenStacks), but this could be a
requirement
>>>>> for a high availability cloud solution (Cloud Region).
Additionally, this
>>>>> OVN-IC solution allows interconnecting other cloud backends
that use OVN,
>>>>> in the case of kubernetes with ovn-kube. >>>>> >>>>> We tested an OVN interconnect integrated with 3 OpenStack >>>>> installations and it worked very well !!! considering direct
L3 traffic at
>>>>> the router level between different network infrastructures. >>>>> >>>>> SYNC_REPAIR - problem >>>>> >>>>> * Static Routes (learned OVN-IC routes) >>>>> * Router Port -> Transit Switches >>>>> >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router
Port found in
>>>>> OVN but not in Neutron, port_id=rt2-admin-tenant1 >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router >>>>> 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes
[{'destination': '
>>>>> 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': ' >>>>> 10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but
not in
>>>>> Neutron >>>>> >>>>> >>>>> Any suggestions on how to resolve this db_sync issue? since
all other
>>>>> Neutron modules did not present any problem with OVN-IC (as
far as I
>>>>> tested). >>>>> I remember Terry was keen to suggest some things to help. I
believe
>>>>> that before proposing some complex mechanism to solve this
simple problem,
>>>>> I would like to hear the community opinion. In our use case, a
bit change
>>>>> in db_sync with filter by neutron key in external_ids would be
enough.
>>>>> >>>>> Regards, >>>>> Roberto >>>>> >>>>> >>>>> >>>>> [1] >>>>>
https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.ht...
>>>>> [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg >>>>> >>>>> >>>>> >>>>> Additional logs: >>>>> >>>>> OpenStack 1 >>>>> >>>>> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>>> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>>> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
ovn-nbctl
>>>>> lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 >>>>> IPv4 Routes >>>>> Route Table <main>: >>>>> 20.0.1.0/24 169.254.200.2 dst-ip
(learned)
>>>>> 20.0.2.0/24 169.254.200.3 dst-ip
(learned)
>>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>> >>>>> IPv6 Routes >>>>> Route Table <main>: >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
ovn-nbctl
>>>>> lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 >>>>> IPv4 Routes >>>>> Route Table <main>: >>>>> 10.0.1.0/24 169.254.100.2 dst-ip
(learned)
>>>>> 10.0.2.0/24 169.254.100.3 dst-ip
(learned)
>>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>> >>>>> IPv6 Routes >>>>> Route Table <main>: >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>>> >>>>> >>>>> OpenStack 2 >>>>> >>>>> root@os-infra-1-neutron-ovn-northd-container-30f7e935:~#
ovn-nbctl
>>>>> lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc >>>>> IPv4 Routes >>>>> Route Table <main>: >>>>> 20.0.0.0/24 169.254.200.1 dst-ip
(learned)
>>>>> 20.0.2.0/24 169.254.200.3 dst-ip
(learned)
>>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>> >>>>> IPv6 Routes >>>>> Route Table <main>: >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>> root@os-infra-1-neutron-ovn-northd-container-30f7e935:~#
ovn-nbctl
>>>>> lr-route-list ce45f681-6454-43fe-974f-81344bb8113a >>>>> IPv4 Routes >>>>> Route Table <main>: >>>>> 10.0.0.0/24 169.254.100.1 dst-ip
(learned)
>>>>> 10.0.2.0/24 169.254.100.3 dst-ip
(learned)
>>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>> >>>>> IPv6 Routes >>>>> Route Table <main>: >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>> >>>>> >>>>> >>>>> OpenStack 3 >>>>> >>>>> root@os-infra-1-neutron-ovn-northd-container-f237db97:~# >>>>> root@os-infra-1-neutron-ovn-northd-container-f237db97:~#
ovn-nbctl
>>>>> lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 >>>>> IPv4 Routes >>>>> Route Table <main>: >>>>> 20.0.0.0/24 169.254.200.1 dst-ip
(learned)
>>>>> 20.0.1.0/24 169.254.200.2 dst-ip
(learned)
>>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>> >>>>> IPv6 Routes >>>>> Route Table <main>: >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>> root@os-infra-1-neutron-ovn-northd-container-f237db97:~#
ovn-nbctl
>>>>> lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 >>>>> IPv4 Routes >>>>> Route Table <main>: >>>>> 10.0.0.0/24 169.254.100.1 dst-ip
(learned)
>>>>> 10.0.1.0/24 169.254.100.2 dst-ip
(learned)
>>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>> >>>>> IPv6 Routes >>>>> Route Table <main>: >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>> >>>>> >>>>> OVN-IC Global database >>>>> >>>>> root@ovn-global-db1:~# ovn-ic-sbctl show >>>>> availability-zone osp1 >>>>> gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 >>>>> hostname: osp1-gwnode1 >>>>> type: geneve >>>>> ip: 192.168.200.28 >>>>> port admin-rt1-tenant1 >>>>> transit switch: admin-tenant1 >>>>> address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24
fe80::1/64"]
>>>>> port admin-rt1-tenant1_1 >>>>> transit switch: admin-tenant1_1 >>>>> address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] >>>>> availability-zone osp2 >>>>> gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 >>>>> hostname: osp2-gwnode1 >>>>> type: geneve >>>>> ip: 192.168.200.128 >>>>> port admin-rt2-tenant1 >>>>> transit switch: admin-tenant1 >>>>> address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24
fe80::2/64"]
>>>>> port admin-rt2-tenant1_1 >>>>> transit switch: admin-tenant1_1 >>>>> address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] >>>>> availability-zone osp3 >>>>> gateway 97595af9-7896-40d0-a883-beadbff1aa5b >>>>> hostname: osp3-gwnode1 >>>>> type: geneve >>>>> ip: 192.168.200.228 >>>>> port admin-rt3-tenant1 >>>>> transit switch: admin-tenant1 >>>>> address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24
fe80::3/64"]
>>>>> port admin-rt3-tenant1_1 >>>>> transit switch: admin-tenant1_1 >>>>> address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"] >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *‘Esta mensagem é direcionada apenas para os endereços
constantes no
>>>>> cabeçalho inicial. Se você não está listado nos endereços
constantes no
>>>>> cabeçalho, pedimos-lhe que desconsidere completamente o
conteúdo dessa
>>>>> mensagem e cuja cópia, encaminhamento e/ou execução das ações
citadas estão
>>>>> imediatamente anuladas e proibidas’.* >>>>> >>>>> *‘Apesar do Magazine Luiza tomar todas as precauções
razoáveis para
>>>>> assegurar que nenhum vírus esteja presente nesse e-mail, a
empresa não
>>>>> poderá aceitar a responsabilidade por quaisquer perdas ou
danos causados
>>>>> por esse e-mail ou por seus anexos’.* >>>>> >>>> >> >> *‘Esta mensagem é direcionada apenas para os endereços constantes
no
>> cabeçalho inicial. Se você não está listado nos endereços
constantes no
>> cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo
dessa
>> mensagem e cuja cópia, encaminhamento e/ou execução das ações
citadas estão
>> imediatamente anuladas e proibidas’.* >> >> *‘Apesar do Magazine Luiza tomar todas as precauções razoáveis
para
>> assegurar que nenhum vírus esteja presente nesse e-mail, a
empresa não
>> poderá aceitar a responsabilidade por quaisquer perdas ou danos
causados
>> por esse e-mail ou por seus anexos’.* >> >
--
_‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes
no
cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo
dessa
mensagem e cuja cópia, encaminhamento e/ou execução das ações
citadas estão
imediatamente anuladas e proibidas’._
- **‘Apesar do Magazine Luiza tomar
todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a
responsabilidade por
quaisquer perdas ou danos causados por esse e-mail ou por seus
anexos’.*
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail.
Hinweise zum Datenschutz finden Sie hier< https://www.datenschutz.schwarz%3E.
This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail.
Information on data protection can be found here< https://www.datenschutz.schwarz%3E.
*‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.*
*‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
*‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.*
*‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
Em seg., 17 de jul. de 2023 às 13:33, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
In my opinion the answer would be: yes and no, this approach would work
for isolated cases, but think on a larger scale: how many external networks would need to be managed at the IP Fabric level? This is exactly what you are proposing in your previous mails: "Felix' suggestion is related to integrating the TS to a network on Neutron (some kind of mapping)". So at this point I'm curious about your proposal. And about the static routes, if these routes are created from a Neutron defined router, as I proposed, they won't be deleted.
The DB sync calls this method [1] to provide a list of the static routes related to some router (Neutron defined router), and then check if this list is consistent with the Neutron static route database [2]. This route will not exist in Neutron (it's a learned route), if this is not made flexible, the OVN-IC cannot be integrated with Neutron because those routes will be deleted. This behavior extends to any other resource dynamically learned by OVN / OVN-IC solution.
[1] https://opendev.org/openstack/neutron/src/branch/master/neutron/plugins/ml2/... [2] https://opendev.org/openstack/neutron/src/branch/master/neutron/plugins/ml2/...
About the proposed RFE, do you see any problem if Neutron deletes from
the OVN only what it manages? Yes, a huge one: OVN is a service for Neutron, same as RabbitMQ or MySQL. These services are 100% controlled by Neutron because the resources can clash. You can't manually create an IPAM register in the DB because that will disable one IP address. The same for OVN: you can't manually create a LRP in a Logical Switch that provides a Neutron network because this LRP will have an IP address. If a new Neutron port is created and it uses the same IP address of the existing LRP (that doesn't exist in Neutron), the new port will be considered virtual and won't be able to be used in a VM (this is an actual problem I found in a customer last week). So no, Neutron should not allow other resources in OVN apart from what Neutron creates.
I understand and it makes sense. However, what would be the options? no other existing solution reinjects routes at the SDN level. The main reason to use OVN-IC is for the router to "learn" the remote routes, if that is not possible from a Neutron perspective and we have to create static route entries manually (via OpenStack) maybe this makes the OVN-IC solution unfeasible.
On Mon, Jul 17, 2023 at 6:13 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
Hi Rodolfo,
Em seg., 17 de jul. de 2023 às 11:30, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Ok, I'll ask again for the tool/script/manual used to make the IC connection. I will also ask again for the NB-IC database resources and the NB resources needed, and how are related.
To be clear, this tool doesn't exist yet and we are working on the design of the software solution [1]. For now, the tests were performed using manual commands, such as those described in the RFE link and very similar to the case applied by the ovn-kube to use OVN-IC with OpenStack - Neutron [2]. The main idea is to develop an external layer interface with an API/SDK and an internal layer interface with the OVN-IC-NB global database and the OpenStack/AZ OVN-NB database [1].
[1] https://drive.google.com/file/d/1yx66YcxNMH_oB11--0-_0eZzmBJHu-Ox/view?usp=s... [2] https://github.com/kubeovn/kube-ovn/blob/v1.11.0/docs/cluster-interconnectio...
If you want Neutron to handle these resources, then you don't need anything special: you can add an external network, handled only by one project. Inside this project you can create the needed resources: networks, routers, etc. This project/user (controlled by the sysadmin), won't create any VM on these networks/routers. Then you can use these resources to connect the TS.
In my opinion the answer would be: yes and no, this approach would work for isolated cases, but think on a larger scale: how many external networks would need to be managed at the IP Fabric level? what is the service type? vlan (we would have segmentation limits) vxlan (we would have to extend the L2 domain across the entire IP fabric) virtual? . I don't know, there are many questions here. Felix can explain better and propose an RFE in the future. Even so, Neutron would need a filter to not delete the static routes learned by the router (the different proposals can be complementary...)
About the proposed RFE, do you see any problem if Neutron deletes from the OVN only what it manages?
On Mon, Jul 17, 2023 at 3:10 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
Hi Felix and Rodolfo,
Em seg., 17 de jul. de 2023 às 05:39, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Hello Felix:
That's the point: the launchpad bug describes the registers created by the IC configuration but these registers do not have any identification that informs about their relationship with the IC TS.
Rather than modifying the Neutron resources (a network is a generic resource that is used by multiple backends), I would try to create a NB-IC resource map and the corresponding NB registers (NAT, LRP and LRSR). If you can link a NB-IC TS with the corresponding NB resources, then the problem is fixed because knowing them we can explicitly skip them from the DB sync tool.
If that is not the case, then please present an alternative with the suggested idea of creating an external network type.
BTW, I don't understand why you first say that "created by tools not necessarily controlled by us we need to avoid changing them", but then you are talking about refactoring these tools. Are we talking about the tools that create these IC configurations? Can you share them?
I think we are talking about two different cases.
1 - This proposed RFE intends to solve the problem of removing external resources and that is enough for the OVN-IC to work in generic scenarios: multi tenancy, single external provider, multiple external providers, etc. because the OVN-IC will not relate to Neutron's network topology or capabilities.
2- Felix' suggestion is related to integrating the TS to a network on Neutron (some kind of mapping), and in this case, we have a hybrid solution with the TS managed by an external tool, and the management of the NAT, LRP / TS link with the tenant being done directly on Neutron. For this proposal, I believe that the port filter would not be necessary because Neutron would know the resource mappings in the OVN-NB, but it seems to me a more complex solution.
The question about case 2 is related to which level we are talking about modifying the generic network to map the TS. If it is a network provider level, the discussion is one - this case would solve the schedule/lrp-set-gateway-chassis problem as it would be a gateway port. but if it is a tenant' network level linked to a router / no gateway port (the discussion would be another) - does not solve the gateway port problem, but it seems closer to what the OVN-IC proposes to do at the network L3 interconnect level).
BTW: in case the TS is connected to an external network, we would need an additional router for external/internet connectivity not related to the OVN-IC, or maybe create a router with multiple gateway ports (that doesn't exist yet)...
Regards, Roberto
Regards.
On Mon, Jul 17, 2023 at 8:43 AM Felix Huettner felix.huettner@mail.schwarz wrote:
Hi everyone,
On Thu, Jul 13, 2023 at 07:29:51PM -0300, Roberto Bartzen Acosta wrote: > Hi Rodolfo, > > Em qui., 13 de jul. de 2023 às 13:45, Rodolfo Alonso Hernandez < > ralonsoh@redhat.com> escreveu: > > > Hello Roberto: > > > > I think you have provided a very detailed explanation of the issue but you > > still didn't present any possible solution. I would ask you at least for a > > proposal and for specific details of what is failing in the OVN sync tool. > > > > For example, if you add a TS between two clouds, you'll need to create (1) > > routers (NAT registers), (2) router ports (LRP) and (3) some static routes > > (LRSR). All these elements are monitored by the DB sync tool and will fail > > because the counterparts in the Neutron DB don't exist. I guess that you > > have some script or tool or at least a document to manually add these > > resources; sharing it could help to find a solution. > > > > If these elements are manually created during the deployment phase, you > > can also control the information provided. I'm thinking about the > > "external_ids" field; we can push some defined constant that will make > > these registers transparent to the OVN DB sync tool.
As these resources are also created by tools not necessarily controlled by us we need to avoid changing them. I would therefor propose to not add a "ignore" constant to the "external_ids", to to rather check for the existence of the neutron keys in "external_ids" (i think thats also what is described in the bug below).
Thanks for the contribution Felix. The idea is not to add (or at least not expect it to be externally added) any identification in the external_ids, Neutron should only filter for DB consistency validation what it knows and has created by Neuron -> Neutron in the external ids key.
> > > Please check this idea and if it is feasible. In that case, please add a > > topic to the Neutron drivers meeting agenda [1] to discuss it. > > > > That sounds good, thanks. > https://bugs.launchpad.net/neutron/+bug/2027742
From my perspective there is also an additional step after the above REF. Currently it just makes it possible to use ovn-ic without neutron killing it. However if we could get neutron to treat a Transit Switch as a external network we would skip the need to implement the NAT, LRP and LRSR logic in custom tooling and could use the existing neutron code. This would probably require creating a custom type of provider network and then specifying the transit switch name in the provider options.
Is that something that from your perspective we should take a look at right away, or rather first focus und neutron not burning the transit switch.
Regards Felix
> > Regards, > Roberto > > > > > > Regards. > > > > [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers > > > > > > On Wed, Jul 12, 2023 at 2:12 PM Roberto Bartzen Acosta < > > roberto.acosta@luizalabs.com> wrote: > > > >> > >> > >> Em qua., 12 de jul. de 2023 às 09:05, Roberto Bartzen Acosta < > >> roberto.acosta@luizalabs.com> escreveu: > >> > >>> Hi Rodolfo, > >>> > >>> Thanks for the feedback. > >>> > >>> I liked the idea of having a different network type only to hold the > >>> Transit Switches and thus the LRP but it depends on some factors like > >>> multi-tenancy. > >>> From the OVN perspective, the TS is global and will only be linked to a > >>> tenant when it is plugged into the tenant router port. My concern here is > >>> that if Neutron manages the TS, it will need to assume the dynamic > >>> characteristics of this TS function, ovn-ic creates and removes the TS in > >>> NB_Global, in addition to plugging and removing ports in this switch > >>> according to the network topology. Additionally, Neutron would still need > >>> to handle the learned remote routes (which are listed as static routes from > >>> the tenant router perspective). > >>> > >> sorry: NB_Global -> Northbound Database. > >> > >> > >>> This is an open discussion, Felix can add ideas here. > >>> > >>> In general, it seems to me that we have no alternatives for this type of > >>> solution other than OVN-IC (note that ovn-bgp-agent does not learn remote > >>> routes at the SDN level / OVN). > >>> OpenStack seems to be designed to run like a "bubble" and this traffic > >>> from one VM to another always needs to be routed at the FIP level and > >>> external routing layers. > >>> > >>> Regards, > >>> Roberto > >>> > >>> Em qua., 12 de jul. de 2023 às 05:11, Rodolfo Alonso Hernandez < > >>> ralonsoh@redhat.com> escreveu: > >>> > >>>> Hello Roberto: > >>>> > >>>> We talked about this possible RFE during the PTG [1] but I don't > >>>> remember having any proposal. Actually what I remember (and was written in > >>>> the etherpad) is that you were going to present an RFE. Can you explain it > >>>> briefly? > >>>> > >>>> We also talked about the idea, proposed by Felix in the mailing list, > >>>> of having a different network type only to hold the Transit Switches and > >>>> thus the LRP. If you have a proposal, please present it. > >>>> > >>>> Regards. > >>>> > >>>> [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506 > >>>> > >>>> On Tue, Jul 11, 2023 at 10:50 PM Roberto Bartzen Acosta < > >>>> roberto.acosta@luizalabs.com> wrote: > >>>> > >>>>> Hello Neutron folks, > >>>>> > >>>>> Regarding the conversation started in March [1] about the use of OVN > >>>>> interconnect with Neutron, I would like to evolve the discussion in > >>>>> relation to resources allocated by OVN-IC and which are not managed by > >>>>> Neutron. In the March PTG [2], the problem related to the db_sync tool was > >>>>> presented, and a proposed solution in which Neutron does not manage these > >>>>> resources. > >>>>> > >>>>> After discussing some architectural designs with Felix/StackIT, it > >>>>> seems to make sense to come up with a proposal to change the mech_driver to > >>>>> validate the external_ids key and not remove resources present in the OVN > >>>>> backend without Neutron "signature". > >>>>> > >>>>> The discussion here is more complex than it seems, because > >>>>> architecturally Neutron does not allow us to interconnect workloads in > >>>>> multiple AZs (with different OpenStacks), but this could be a requirement > >>>>> for a high availability cloud solution (Cloud Region). Additionally, this > >>>>> OVN-IC solution allows interconnecting other cloud backends that use OVN, > >>>>> in the case of kubernetes with ovn-kube. > >>>>> > >>>>> We tested an OVN interconnect integrated with 3 OpenStack > >>>>> installations and it worked very well !!! considering direct L3 traffic at > >>>>> the router level between different network infrastructures. > >>>>> > >>>>> SYNC_REPAIR - problem > >>>>> > >>>>> * Static Routes (learned OVN-IC routes) > >>>>> * Router Port -> Transit Switches > >>>>> > >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae > >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING > >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync > >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in > >>>>> OVN but not in Neutron, port_id=rt2-admin-tenant1 > >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae > >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING > >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync > >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router > >>>>> 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': ' > >>>>> 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': ' > >>>>> 10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not in > >>>>> Neutron > >>>>> > >>>>> > >>>>> Any suggestions on how to resolve this db_sync issue? since all other > >>>>> Neutron modules did not present any problem with OVN-IC (as far as I > >>>>> tested). > >>>>> I remember Terry was keen to suggest some things to help. I believe > >>>>> that before proposing some complex mechanism to solve this simple problem, > >>>>> I would like to hear the community opinion. In our use case, a bit change > >>>>> in db_sync with filter by neutron key in external_ids would be enough. > >>>>> > >>>>> Regards, > >>>>> Roberto > >>>>> > >>>>> > >>>>> > >>>>> [1] > >>>>> https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.ht... > >>>>> [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg > >>>>> > >>>>> > >>>>> > >>>>> Additional logs: > >>>>> > >>>>> OpenStack 1 > >>>>> > >>>>> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# > >>>>> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# > >>>>> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl > >>>>> lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 > >>>>> IPv4 Routes > >>>>> Route Table <main>: > >>>>> 20.0.1.0/24 169.254.200.2 dst-ip (learned) > >>>>> 20.0.2.0/24 169.254.200.3 dst-ip (learned) > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip > >>>>> > >>>>> IPv6 Routes > >>>>> Route Table <main>: > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip > >>>>> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl > >>>>> lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 > >>>>> IPv4 Routes > >>>>> Route Table <main>: > >>>>> 10.0.1.0/24 169.254.100.2 dst-ip (learned) > >>>>> 10.0.2.0/24 169.254.100.3 dst-ip (learned) > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip > >>>>> > >>>>> IPv6 Routes > >>>>> Route Table <main>: > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip > >>>>> root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# > >>>>> > >>>>> > >>>>> OpenStack 2 > >>>>> > >>>>> root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl > >>>>> lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc > >>>>> IPv4 Routes > >>>>> Route Table <main>: > >>>>> 20.0.0.0/24 169.254.200.1 dst-ip (learned) > >>>>> 20.0.2.0/24 169.254.200.3 dst-ip (learned) > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip > >>>>> > >>>>> IPv6 Routes > >>>>> Route Table <main>: > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip > >>>>> root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl > >>>>> lr-route-list ce45f681-6454-43fe-974f-81344bb8113a > >>>>> IPv4 Routes > >>>>> Route Table <main>: > >>>>> 10.0.0.0/24 169.254.100.1 dst-ip (learned) > >>>>> 10.0.2.0/24 169.254.100.3 dst-ip (learned) > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip > >>>>> > >>>>> IPv6 Routes > >>>>> Route Table <main>: > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip > >>>>> > >>>>> > >>>>> > >>>>> OpenStack 3 > >>>>> > >>>>> root@os-infra-1-neutron-ovn-northd-container-f237db97:~# > >>>>> root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl > >>>>> lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 > >>>>> IPv4 Routes > >>>>> Route Table <main>: > >>>>> 20.0.0.0/24 169.254.200.1 dst-ip (learned) > >>>>> 20.0.1.0/24 169.254.200.2 dst-ip (learned) > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip > >>>>> > >>>>> IPv6 Routes > >>>>> Route Table <main>: > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip > >>>>> root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl > >>>>> lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 > >>>>> IPv4 Routes > >>>>> Route Table <main>: > >>>>> 10.0.0.0/24 169.254.100.1 dst-ip (learned) > >>>>> 10.0.1.0/24 169.254.100.2 dst-ip (learned) > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip > >>>>> > >>>>> IPv6 Routes > >>>>> Route Table <main>: > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip > >>>>> > >>>>> > >>>>> OVN-IC Global database > >>>>> > >>>>> root@ovn-global-db1:~# ovn-ic-sbctl show > >>>>> availability-zone osp1 > >>>>> gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 > >>>>> hostname: osp1-gwnode1 > >>>>> type: geneve > >>>>> ip: 192.168.200.28 > >>>>> port admin-rt1-tenant1 > >>>>> transit switch: admin-tenant1 > >>>>> address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"] > >>>>> port admin-rt1-tenant1_1 > >>>>> transit switch: admin-tenant1_1 > >>>>> address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] > >>>>> availability-zone osp2 > >>>>> gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 > >>>>> hostname: osp2-gwnode1 > >>>>> type: geneve > >>>>> ip: 192.168.200.128 > >>>>> port admin-rt2-tenant1 > >>>>> transit switch: admin-tenant1 > >>>>> address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"] > >>>>> port admin-rt2-tenant1_1 > >>>>> transit switch: admin-tenant1_1 > >>>>> address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] > >>>>> availability-zone osp3 > >>>>> gateway 97595af9-7896-40d0-a883-beadbff1aa5b > >>>>> hostname: osp3-gwnode1 > >>>>> type: geneve > >>>>> ip: 192.168.200.228 > >>>>> port admin-rt3-tenant1 > >>>>> transit switch: admin-tenant1 > >>>>> address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"] > >>>>> port admin-rt3-tenant1_1 > >>>>> transit switch: admin-tenant1_1 > >>>>> address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"] > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> *‘Esta mensagem é direcionada apenas para os endereços constantes no > >>>>> cabeçalho inicial. Se você não está listado nos endereços constantes no > >>>>> cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa > >>>>> mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão > >>>>> imediatamente anuladas e proibidas’.* > >>>>> > >>>>> *‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para > >>>>> assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não > >>>>> poderá aceitar a responsabilidade por quaisquer perdas ou danos causados > >>>>> por esse e-mail ou por seus anexos’.* > >>>>> > >>>> > >> > >> *‘Esta mensagem é direcionada apenas para os endereços constantes no > >> cabeçalho inicial. Se você não está listado nos endereços constantes no > >> cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa > >> mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão > >> imediatamente anuladas e proibidas’.* > >> > >> *‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para > >> assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não > >> poderá aceitar a responsabilidade por quaisquer perdas ou danos causados > >> por esse e-mail ou por seus anexos’.* > >> > > > > -- > > > > > _‘Esta mensagem é direcionada apenas para os endereços constantes no > cabeçalho inicial. Se você não está listado nos endereços constantes no > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão > imediatamente anuladas e proibidas’._ > > > * **‘Apesar do Magazine Luiza tomar > todas as precauções razoáveis para assegurar que nenhum vírus esteja > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.* > > > Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail.
Hinweise zum Datenschutz finden Sie hier< https://www.datenschutz.schwarz%3E.
This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail.
Information on data protection can be found here< https://www.datenschutz.schwarz%3E.
*‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.*
*‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
*‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.*
*‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
Ok, this is being tortuous. First of all: define a strategy. If you are going to create the resources in Neutron, define how. I've provided a way to do this, find a formal strategy to ground it.
Second: (again) try to find a connection between resources, if you are going to use the strategy of creating the resources in Neutron. The "Logical_Router_Static_Route" belongs to a router univocally. If that router has been created by OpenStack, then you can modify the DB sync method to consider learned routes too.
In order to do this, you'll need a set of resources that are going to be needed in Neutron, the OVN counterparts and other resources (like "Logical_Router_Static_Route") that will be added and will be present in OVN and not in Neutron DB. Also you'll need to know how to relate all of them.
Hi everyone,
we will validate in the next weeks the idea i'll describe below. If it sounds feasible to us i would write up a spec for this for later implementation.
Setup: 1. Two neutron setups with OVN 2. A OVN-IC setup not managed by any tooling (used manually)
Target setup for a individual project
+-------------------------------------+ | Transit Switch (ts1) | | 172.24.0.0/24 | +-------------------------------------+ | | +----------------+ +----------------+ | 172.24.0.10 | | 172.24.0.11 | | Router | | Router | | 192.168.0.1 | | 192.168.1.1 | +----------------+ +----------------+ | | +----------------+ +----------------+ | Network | | Network | | 192.168.0.0/24 | | 192.168.1.0/24 | +----------------+ +----------------+ | | +----------------+ +----------------+ | VM | | VM | | 192.168.0.10 | | 192.168.1.10 | +----------------+ +----------------+
Main creation workflow of a interconnected network: 1. Create a transit switch in the OVN-IC-NB 2. In each neutron: a. Create a new provider network that maps to the transit switch. E.g. using `--provider-network-type ovn-ic --provider-phyiscal-network ts1` b. Create the internal network and router as always and attach the router to the internal network and the provider network
Changes needed in Neutron (from my perspective): 1. Neutron must not touch the resources created by OVN-IC. This might be done using two options: a. Ignore all resources where `external_ids` does not contain any key starting with `neutron`. b. Ignore specific resources as defined by individual rules: I. ignore all Logical_Router that contain `other_config:interconn-ts` II. ignore all Logical_Router_Static_Route that contain `external_ids:ic-learned-route` 2. Neutron must allow OVN-IC transit switches to be treated as external networks. AFAIK this should work in the same way as creating any kind of OVN network (so Logical_Switch), but just importing an existing Logical_Switch.
From my perspective with these two changes we should be good to
interconnect two neutron setups with probably minimal changes needed. This is because OVN-IC would only touch the transit switch (which looks in the Northbound like a regular Logical_Switch). All other resources would still be managed by neutron and work as they do now.
I will get back to this discussion and (if appropriate) provide a spec or this idea, once we had time to validate this on our side (which will take some weeks).
Thanks Felix
On Tue, Jul 18, 2023 at 01:23:27PM +0200, Rodolfo Alonso Hernandez wrote:
Ok, this is being tortuous. First of all: define a strategy. If you are going to create the resources in Neutron, define how. I've provided a way to do this, find a formal strategy to ground it.
Second: (again) try to find a connection between resources, if you are going to use the strategy of creating the resources in Neutron. The "Logical_Router_Static_Route" belongs to a router univocally. If that router has been created by OpenStack, then you can modify the DB sync method to consider learned routes too.
In order to do this, you'll need a set of resources that are going to be needed in Neutron, the OVN counterparts and other resources (like "Logical_Router_Static_Route") that will be added and will be present in OVN and not in Neutron DB. Also you'll need to know how to relate all of them.
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail.
Hinweise zum Datenschutz finden Sie hierhttps://www.datenschutz.schwarz.
This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail.
Information on data protection can be found herehttps://www.datenschutz.schwarz.
Hi Rodolfo,
I agree with you, my initial idea with this thread was exactly what we are doing, debating problems and possible solutions for interconnecting workloads using unmanaged dynamic router resources. Thank you for being engaged with this! I'm sorry if the strategy wasn't presented clearly, let me talk about it then.
Imagine we have a tenant on OpenStack, and this client has all the resources created by OpenStack itself 1 - Tenant OpenStack project 2 - Tenant self-service network 3 - Tenant self-service subnet - e.g. 10.0.0.0/24 4 - Security groups 5 - Maybe FIPs 6 - Tenant Logical Router 7 - Tenant Logical Router Ports: 1. external network (provider/default gw) ; 2. internal network (self-service subnet) 8 - Tenant VMs - some VMs on the self-service subnet 10.0.0.0/24
Now that same tenant expects to connect these VMs with another deployment in the same region - High Availability.
Option 1 - Floating IPs -> it's not a good option Option 2 - OVN-IC - L3 routing between self-service networks :)
What's the strategy? connect the Tenant Logical Router with the TS of the OVN-IC.
Creation steps: 1. Create a transit switch in the OVN-IC-NB 2. In each OpenStack / Neutron: a. Create a new Logical_Router_Port (externally managed) on the Tenant Logical Router and link this port with the TS.
What is the point here, as Felix comments: even with the creation of the special network type, it will still be necessary to filter learned resources. " a. Ignore all resources where `external_ids` does not contain any key starting with `neutron`."
So, from my perspective: Neutron changes for the [RFE] unmanaged dynamic router resources - OVN b. Ignore specific resources as defined by individual rules: I. ignore all Logical_Router_Ports where `external_ids` does not contain any key starting with `neutron`. II. ignore all Logical_Router_Static_Route where `external_ids` does not contain any key starting with `neutron`.
We have already tested this and it is enough for interconnect to work in multi-tenant scenarios, and with the router connected to a generic external network provider (where all other tenants are connected).
That is, the Logical Router was created by Neutron, so it can decide to learn dynamic routes and ignore external interconnect Logical_Router_Ports.
BTW: Felix, the transit switch will learn remote Logical_Switch_Ports ports automatically via OVN-NB, in this case these ports were not created by Neutron and would need more special treatment to skip these resources.
Note: about creating a new external provider network type, I already expressed my arguments about the multi tenant case and the need to create more resources to have this together with the default gw. If it were possible to have this in the same Logical_router tenant, I mean, use two external networks on the same Router: here we would have a lot of gain in my perspective
Regards, Roberto
Em ter., 18 de jul. de 2023 às 09:40, Felix Huettner felix.huettner@mail.schwarz escreveu:
Hi everyone,
we will validate in the next weeks the idea i'll describe below. If it sounds feasible to us i would write up a spec for this for later implementation.
Setup:
- Two neutron setups with OVN
- A OVN-IC setup not managed by any tooling (used manually)
Target setup for a individual project
+-------------------------------------+ | Transit Switch (ts1) | | 172.24.0.0/24 | +-------------------------------------+ | | +----------------+ +----------------+ | 172.24.0.10 | | 172.24.0.11 | | Router | | Router | | 192.168.0.1 | | 192.168.1.1 | +----------------+ +----------------+ | | +----------------+ +----------------+ | Network | | Network | | 192.168.0.0/24 | | 192.168.1.0/24 | +----------------+ +----------------+ | | +----------------+ +----------------+ | VM | | VM | | 192.168.0.10 | | 192.168.1.10 | +----------------+ +----------------+
Main creation workflow of a interconnected network:
- Create a transit switch in the OVN-IC-NB
- In each neutron: a. Create a new provider network that maps to the transit switch. E.g. using `--provider-network-type ovn-ic
--provider-phyiscal-network ts1` b. Create the internal network and router as always and attach the router to the internal network and the provider network
Changes needed in Neutron (from my perspective):
- Neutron must not touch the resources created by OVN-IC. This might be done using two options: a. Ignore all resources where `external_ids` does not contain any key starting with `neutron`. b. Ignore specific resources as defined by individual rules: I. ignore all Logical_Router that contain `other_config:interconn-ts` II. ignore all Logical_Router_Static_Route that contain `external_ids:ic-learned-route`
- Neutron must allow OVN-IC transit switches to be treated as external networks. AFAIK this should work in the same way as creating any kind of OVN network (so Logical_Switch), but just importing an existing Logical_Switch.
From my perspective with these two changes we should be good to interconnect two neutron setups with probably minimal changes needed. This is because OVN-IC would only touch the transit switch (which looks in the Northbound like a regular Logical_Switch). All other resources would still be managed by neutron and work as they do now.
I will get back to this discussion and (if appropriate) provide a spec or this idea, once we had time to validate this on our side (which will take some weeks).
Thanks Felix
On Tue, Jul 18, 2023 at 01:23:27PM +0200, Rodolfo Alonso Hernandez wrote:
Ok, this is being tortuous. First of all: define a strategy. If you are going to create the resources in Neutron, define how. I've provided a way to do this, find a formal strategy to ground it.
Second: (again) try to find a connection between resources, if you are going to use the strategy of creating the resources in Neutron. The "Logical_Router_Static_Route" belongs to a router univocally. If that router has been created by OpenStack, then you can modify the DB sync method to consider learned routes too.
In order to do this, you'll need a set of resources that are going to be needed in Neutron, the OVN counterparts and other resources (like "Logical_Router_Static_Route") that will be added and will be present in OVN and not in Neutron DB. Also you'll need to know how to relate all of them.
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail.
Hinweise zum Datenschutz finden Sie hierhttps://www.datenschutz.schwarz.
This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail.
Information on data protection can be found here< https://www.datenschutz.schwarz%3E.
Would it be possible to implement multiple external network ports on the same router?
It is not the same case of multi-homing [1], in the case of OVN-IC there are no symmetric internet routes.... [1] https://review.opendev.org/c/openstack/neutron/+/874199
Basically an external port for ovn-ic doesn't have SNAT!
Could this be done Rodolfo? What would be the effort/implications?
If possible, we can integrate everything we are discussing in terms of solutions into a single reference design.
Em ter., 18 de jul. de 2023 às 10:20, Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> escreveu:
Hi Rodolfo,
I agree with you, my initial idea with this thread was exactly what we are doing, debating problems and possible solutions for interconnecting workloads using unmanaged dynamic router resources. Thank you for being engaged with this! I'm sorry if the strategy wasn't presented clearly, let me talk about it then.
Imagine we have a tenant on OpenStack, and this client has all the resources created by OpenStack itself 1 - Tenant OpenStack project 2 - Tenant self-service network 3 - Tenant self-service subnet - e.g. 10.0.0.0/24 4 - Security groups 5 - Maybe FIPs 6 - Tenant Logical Router 7 - Tenant Logical Router Ports: 1. external network (provider/default gw) ; 2. internal network (self-service subnet) 8 - Tenant VMs - some VMs on the self-service subnet 10.0.0.0/24
Now that same tenant expects to connect these VMs with another deployment in the same region - High Availability.
Option 1 - Floating IPs -> it's not a good option Option 2 - OVN-IC - L3 routing between self-service networks :)
What's the strategy? connect the Tenant Logical Router with the TS of the OVN-IC.
Creation steps:
- Create a transit switch in the OVN-IC-NB
- In each OpenStack / Neutron: a. Create a new Logical_Router_Port (externally managed) on the Tenant
Logical Router and link this port with the TS.
What is the point here, as Felix comments: even with the creation of the special network type, it will still be necessary to filter learned resources. " a. Ignore all resources where `external_ids` does not contain any key starting with `neutron`."
So, from my perspective: Neutron changes for the [RFE] unmanaged dynamic router resources - OVN b. Ignore specific resources as defined by individual rules: I. ignore all Logical_Router_Ports where `external_ids` does not contain any key starting with `neutron`. II. ignore all Logical_Router_Static_Route where `external_ids` does not contain any key starting with `neutron`.
We have already tested this and it is enough for interconnect to work in multi-tenant scenarios, and with the router connected to a generic external network provider (where all other tenants are connected).
That is, the Logical Router was created by Neutron, so it can decide to learn dynamic routes and ignore external interconnect Logical_Router_Ports.
BTW: Felix, the transit switch will learn remote Logical_Switch_Ports ports automatically via OVN-NB, in this case these ports were not created by Neutron and would need more special treatment to skip these resources.
Note: about creating a new external provider network type, I already expressed my arguments about the multi tenant case and the need to create more resources to have this together with the default gw. If it were possible to have this in the same Logical_router tenant, I mean, use two external networks on the same Router: here we would have a lot of gain in my perspective
Regards, Roberto
Em ter., 18 de jul. de 2023 às 09:40, Felix Huettner felix.huettner@mail.schwarz escreveu:
Hi everyone,
we will validate in the next weeks the idea i'll describe below. If it sounds feasible to us i would write up a spec for this for later implementation.
Setup:
- Two neutron setups with OVN
- A OVN-IC setup not managed by any tooling (used manually)
Target setup for a individual project
+-------------------------------------+ | Transit Switch (ts1) | | 172.24.0.0/24 | +-------------------------------------+ | | +----------------+ +----------------+ | 172.24.0.10 | | 172.24.0.11 | | Router | | Router | | 192.168.0.1 | | 192.168.1.1 | +----------------+ +----------------+ | | +----------------+ +----------------+ | Network | | Network | | 192.168.0.0/24 | | 192.168.1.0/24 | +----------------+ +----------------+ | | +----------------+ +----------------+ | VM | | VM | | 192.168.0.10 | | 192.168.1.10 | +----------------+ +----------------+
Main creation workflow of a interconnected network:
- Create a transit switch in the OVN-IC-NB
- In each neutron: a. Create a new provider network that maps to the transit switch. E.g. using `--provider-network-type ovn-ic
--provider-phyiscal-network ts1` b. Create the internal network and router as always and attach the router to the internal network and the provider network
Changes needed in Neutron (from my perspective):
- Neutron must not touch the resources created by OVN-IC. This might be done using two options: a. Ignore all resources where `external_ids` does not contain any key starting with `neutron`. b. Ignore specific resources as defined by individual rules: I. ignore all Logical_Router that contain `other_config:interconn-ts` II. ignore all Logical_Router_Static_Route that contain `external_ids:ic-learned-route`
- Neutron must allow OVN-IC transit switches to be treated as external networks. AFAIK this should work in the same way as creating any kind of OVN network (so Logical_Switch), but just importing an existing Logical_Switch.
From my perspective with these two changes we should be good to interconnect two neutron setups with probably minimal changes needed. This is because OVN-IC would only touch the transit switch (which looks in the Northbound like a regular Logical_Switch). All other resources would still be managed by neutron and work as they do now.
I will get back to this discussion and (if appropriate) provide a spec or this idea, once we had time to validate this on our side (which will take some weeks).
Thanks Felix
On Tue, Jul 18, 2023 at 01:23:27PM +0200, Rodolfo Alonso Hernandez wrote:
Ok, this is being tortuous. First of all: define a strategy. If you are going to create the resources in Neutron, define how. I've provided a
way
to do this, find a formal strategy to ground it.
Second: (again) try to find a connection between resources, if you are going to use the strategy of creating the resources in Neutron. The "Logical_Router_Static_Route" belongs to a router univocally. If that router has been created by OpenStack, then you can modify the DB sync method to consider learned routes too.
In order to do this, you'll need a set of resources that are going to be needed in Neutron, the OVN counterparts and other resources (like "Logical_Router_Static_Route") that will be added and will be present in OVN and not in Neutron DB. Also you'll need to know how to relate all of them.
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail.
Hinweise zum Datenschutz finden Sie hier<https://www.datenschutz.schwarz
.
This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail.
Information on data protection can be found here< https://www.datenschutz.schwarz%3E.
Before that you need to explain why you need multiple external network ports and why cannot be solved with the current architecture Neutron is providing.
On Tue, Jul 18, 2023 at 3:52 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
Would it be possible to implement multiple external network ports on the same router?
It is not the same case of multi-homing [1], in the case of OVN-IC there are no symmetric internet routes.... [1] https://review.opendev.org/c/openstack/neutron/+/874199
Basically an external port for ovn-ic doesn't have SNAT!
Could this be done Rodolfo? What would be the effort/implications?
If possible, we can integrate everything we are discussing in terms of solutions into a single reference design.
Em ter., 18 de jul. de 2023 às 10:20, Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> escreveu:
Hi Rodolfo,
I agree with you, my initial idea with this thread was exactly what we are doing, debating problems and possible solutions for interconnecting workloads using unmanaged dynamic router resources. Thank you for being engaged with this! I'm sorry if the strategy wasn't presented clearly, let me talk about it then.
Imagine we have a tenant on OpenStack, and this client has all the resources created by OpenStack itself 1 - Tenant OpenStack project 2 - Tenant self-service network 3 - Tenant self-service subnet - e.g. 10.0.0.0/24 4 - Security groups 5 - Maybe FIPs 6 - Tenant Logical Router 7 - Tenant Logical Router Ports: 1. external network (provider/default gw) ; 2. internal network (self-service subnet) 8 - Tenant VMs - some VMs on the self-service subnet 10.0.0.0/24
Now that same tenant expects to connect these VMs with another deployment in the same region - High Availability.
Option 1 - Floating IPs -> it's not a good option Option 2 - OVN-IC - L3 routing between self-service networks :)
What's the strategy? connect the Tenant Logical Router with the TS of the OVN-IC.
Creation steps:
- Create a transit switch in the OVN-IC-NB
- In each OpenStack / Neutron: a. Create a new Logical_Router_Port (externally managed) on the Tenant
Logical Router and link this port with the TS.
What is the point here, as Felix comments: even with the creation of the special network type, it will still be necessary to filter learned resources. " a. Ignore all resources where `external_ids` does not contain any key starting with `neutron`."
So, from my perspective: Neutron changes for the [RFE] unmanaged dynamic router resources - OVN b. Ignore specific resources as defined by individual rules: I. ignore all Logical_Router_Ports where `external_ids` does not contain any key starting with `neutron`. II. ignore all Logical_Router_Static_Route where `external_ids` does not contain any key starting with `neutron`.
We have already tested this and it is enough for interconnect to work in multi-tenant scenarios, and with the router connected to a generic external network provider (where all other tenants are connected).
That is, the Logical Router was created by Neutron, so it can decide to learn dynamic routes and ignore external interconnect Logical_Router_Ports.
BTW: Felix, the transit switch will learn remote Logical_Switch_Ports ports automatically via OVN-NB, in this case these ports were not created by Neutron and would need more special treatment to skip these resources.
Note: about creating a new external provider network type, I already expressed my arguments about the multi tenant case and the need to create more resources to have this together with the default gw. If it were possible to have this in the same Logical_router tenant, I mean, use two external networks on the same Router: here we would have a lot of gain in my perspective
Regards, Roberto
Em ter., 18 de jul. de 2023 às 09:40, Felix Huettner felix.huettner@mail.schwarz escreveu:
Hi everyone,
we will validate in the next weeks the idea i'll describe below. If it sounds feasible to us i would write up a spec for this for later implementation.
Setup:
- Two neutron setups with OVN
- A OVN-IC setup not managed by any tooling (used manually)
Target setup for a individual project
+-------------------------------------+ | Transit Switch (ts1) | | 172.24.0.0/24 | +-------------------------------------+ | | +----------------+ +----------------+ | 172.24.0.10 | | 172.24.0.11 | | Router | | Router | | 192.168.0.1 | | 192.168.1.1 | +----------------+ +----------------+ | | +----------------+ +----------------+ | Network | | Network | | 192.168.0.0/24 | | 192.168.1.0/24 | +----------------+ +----------------+ | | +----------------+ +----------------+ | VM | | VM | | 192.168.0.10 | | 192.168.1.10 | +----------------+ +----------------+
Main creation workflow of a interconnected network:
- Create a transit switch in the OVN-IC-NB
- In each neutron: a. Create a new provider network that maps to the transit switch. E.g. using `--provider-network-type ovn-ic
--provider-phyiscal-network ts1` b. Create the internal network and router as always and attach the router to the internal network and the provider network
Changes needed in Neutron (from my perspective):
- Neutron must not touch the resources created by OVN-IC. This might be done using two options: a. Ignore all resources where `external_ids` does not contain any key starting with `neutron`. b. Ignore specific resources as defined by individual rules: I. ignore all Logical_Router that contain `other_config:interconn-ts` II. ignore all Logical_Router_Static_Route that contain `external_ids:ic-learned-route`
- Neutron must allow OVN-IC transit switches to be treated as external networks. AFAIK this should work in the same way as creating any kind of OVN network (so Logical_Switch), but just importing an existing Logical_Switch.
From my perspective with these two changes we should be good to interconnect two neutron setups with probably minimal changes needed. This is because OVN-IC would only touch the transit switch (which looks in the Northbound like a regular Logical_Switch). All other resources would still be managed by neutron and work as they do now.
I will get back to this discussion and (if appropriate) provide a spec or this idea, once we had time to validate this on our side (which will take some weeks).
Thanks Felix
On Tue, Jul 18, 2023 at 01:23:27PM +0200, Rodolfo Alonso Hernandez wrote:
Ok, this is being tortuous. First of all: define a strategy. If you are going to create the resources in Neutron, define how. I've provided a
way
to do this, find a formal strategy to ground it.
Second: (again) try to find a connection between resources, if you are going to use the strategy of creating the resources in Neutron. The "Logical_Router_Static_Route" belongs to a router univocally. If that router has been created by OpenStack, then you can modify the DB sync method to consider learned routes too.
In order to do this, you'll need a set of resources that are going to
be
needed in Neutron, the OVN counterparts and other resources (like "Logical_Router_Static_Route") that will be added and will be present
in
OVN and not in Neutron DB. Also you'll need to know how to relate all
of
them.
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail.
Hinweise zum Datenschutz finden Sie hier<https://www.datenschutz.schwarz
.
This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail.
Information on data protection can be found here< https://www.datenschutz.schwarz%3E.
*‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.*
*‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
Em qua., 19 de jul. de 2023 às 04:57, Rodolfo Alonso Hernandez < ralonsoh@redhat.com> escreveu:
Before that you need to explain why you need multiple external network ports and why cannot be solved with the current architecture Neutron is providing.
I and maybe others need this because Neutron's networking backend doesn't have infinite resources. Although the architecture allows for adding a new Logical_Router and configuring static routes, etc... This means consuming more resources on the SDN backend.
So the problem here would be scalability: each tenant would need 2 routers to do the work of only 1. In addition to the set of additional LRP's, LSP's, Static Routes that the backend would need to configure. For a few tenants this would be fine, but when we talk about thousands, this question is very relevant.
ML2/OVN, for example, limits backend resources according to the OVN version (each one has its scale limitations), in Ussuri's OVN 20.03 (we can't create more than 600 routers and 400k openflow flows without triggering config timeouts - ovn-controller/openvswitch recompute snowball)... in Yoga's OVN 22.03 the numbers are a little better, but we also have limits... Dedicating more resources to doing this would be a waste of infrastructure.
On Tue, Jul 18, 2023 at 3:52 PM Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> wrote:
Would it be possible to implement multiple external network ports on the same router?
It is not the same case of multi-homing [1], in the case of OVN-IC there are no symmetric internet routes.... [1] https://review.opendev.org/c/openstack/neutron/+/874199
Basically an external port for ovn-ic doesn't have SNAT!
Could this be done Rodolfo? What would be the effort/implications?
If possible, we can integrate everything we are discussing in terms of solutions into a single reference design.
Em ter., 18 de jul. de 2023 às 10:20, Roberto Bartzen Acosta < roberto.acosta@luizalabs.com> escreveu:
Hi Rodolfo,
I agree with you, my initial idea with this thread was exactly what we are doing, debating problems and possible solutions for interconnecting workloads using unmanaged dynamic router resources. Thank you for being engaged with this! I'm sorry if the strategy wasn't presented clearly, let me talk about it then.
Imagine we have a tenant on OpenStack, and this client has all the resources created by OpenStack itself 1 - Tenant OpenStack project 2 - Tenant self-service network 3 - Tenant self-service subnet - e.g. 10.0.0.0/24 4 - Security groups 5 - Maybe FIPs 6 - Tenant Logical Router 7 - Tenant Logical Router Ports: 1. external network (provider/default gw) ; 2. internal network (self-service subnet) 8 - Tenant VMs - some VMs on the self-service subnet 10.0.0.0/24
Now that same tenant expects to connect these VMs with another deployment in the same region - High Availability.
Option 1 - Floating IPs -> it's not a good option Option 2 - OVN-IC - L3 routing between self-service networks :)
What's the strategy? connect the Tenant Logical Router with the TS of the OVN-IC.
Creation steps:
- Create a transit switch in the OVN-IC-NB
- In each OpenStack / Neutron: a. Create a new Logical_Router_Port (externally managed) on the
Tenant Logical Router and link this port with the TS.
What is the point here, as Felix comments: even with the creation of the special network type, it will still be necessary to filter learned resources. " a. Ignore all resources where `external_ids` does not contain any key starting with `neutron`."
So, from my perspective: Neutron changes for the [RFE] unmanaged dynamic router resources - OVN b. Ignore specific resources as defined by individual rules: I. ignore all Logical_Router_Ports where `external_ids` does not contain any key starting with `neutron`. II. ignore all Logical_Router_Static_Route where `external_ids` does not contain any key starting with `neutron`.
We have already tested this and it is enough for interconnect to work in multi-tenant scenarios, and with the router connected to a generic external network provider (where all other tenants are connected).
That is, the Logical Router was created by Neutron, so it can decide to learn dynamic routes and ignore external interconnect Logical_Router_Ports.
BTW: Felix, the transit switch will learn remote Logical_Switch_Ports ports automatically via OVN-NB, in this case these ports were not created by Neutron and would need more special treatment to skip these resources.
Note: about creating a new external provider network type, I already expressed my arguments about the multi tenant case and the need to create more resources to have this together with the default gw. If it were possible to have this in the same Logical_router tenant, I mean, use two external networks on the same Router: here we would have a lot of gain in my perspective
Regards, Roberto
Em ter., 18 de jul. de 2023 às 09:40, Felix Huettner felix.huettner@mail.schwarz escreveu:
Hi everyone,
we will validate in the next weeks the idea i'll describe below. If it sounds feasible to us i would write up a spec for this for later implementation.
Setup:
- Two neutron setups with OVN
- A OVN-IC setup not managed by any tooling (used manually)
Target setup for a individual project
+-------------------------------------+ | Transit Switch (ts1) | | 172.24.0.0/24 | +-------------------------------------+ | | +----------------+ +----------------+ | 172.24.0.10 | | 172.24.0.11 | | Router | | Router | | 192.168.0.1 | | 192.168.1.1 | +----------------+ +----------------+ | | +----------------+ +----------------+ | Network | | Network | | 192.168.0.0/24 | | 192.168.1.0/24 | +----------------+ +----------------+ | | +----------------+ +----------------+ | VM | | VM | | 192.168.0.10 | | 192.168.1.10 | +----------------+ +----------------+
Main creation workflow of a interconnected network:
- Create a transit switch in the OVN-IC-NB
- In each neutron: a. Create a new provider network that maps to the transit switch. E.g. using `--provider-network-type ovn-ic
--provider-phyiscal-network ts1` b. Create the internal network and router as always and attach the router to the internal network and the provider network
Changes needed in Neutron (from my perspective):
- Neutron must not touch the resources created by OVN-IC. This might be done using two options: a. Ignore all resources where `external_ids` does not contain any key starting with `neutron`. b. Ignore specific resources as defined by individual rules: I. ignore all Logical_Router that contain `other_config:interconn-ts` II. ignore all Logical_Router_Static_Route that contain `external_ids:ic-learned-route`
- Neutron must allow OVN-IC transit switches to be treated as external networks. AFAIK this should work in the same way as creating any kind of OVN network (so Logical_Switch), but just importing an existing Logical_Switch.
From my perspective with these two changes we should be good to interconnect two neutron setups with probably minimal changes needed. This is because OVN-IC would only touch the transit switch (which looks in the Northbound like a regular Logical_Switch). All other resources would still be managed by neutron and work as they do now.
I will get back to this discussion and (if appropriate) provide a spec or this idea, once we had time to validate this on our side (which will take some weeks).
Thanks Felix
On Tue, Jul 18, 2023 at 01:23:27PM +0200, Rodolfo Alonso Hernandez wrote:
Ok, this is being tortuous. First of all: define a strategy. If you
are
going to create the resources in Neutron, define how. I've provided a
way
to do this, find a formal strategy to ground it.
Second: (again) try to find a connection between resources, if you are going to use the strategy of creating the resources in Neutron. The "Logical_Router_Static_Route" belongs to a router univocally. If that router has been created by OpenStack, then you can modify the DB sync method to consider learned routes too.
In order to do this, you'll need a set of resources that are going to
be
needed in Neutron, the OVN counterparts and other resources (like "Logical_Router_Static_Route") that will be added and will be present
in
OVN and not in Neutron DB. Also you'll need to know how to relate all
of
them.
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail.
Hinweise zum Datenschutz finden Sie hier< https://www.datenschutz.schwarz%3E.
This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail.
Information on data protection can be found here< https://www.datenschutz.schwarz%3E.
*‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.*
*‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
participants (3)
-
Felix Huettner
-
Roberto Bartzen Acosta
-
Rodolfo Alonso Hernandez