Hey,
we are using OVN 22.03 and face an issue where a VM that is directly connected to the provider network won't be accessible, because it cannot arp for the Gateway IP. OVN routers do reply to the arp request though. We know that this exact scenario works as we have it running in our staging environment.
Oddly enough if the right MAC-IP Binding is manually defined within the VM and the Gateway, the traffic will begin to flow correctly according to the right SGs.
I did an ovn-trace and were able to see that the traffic is supposed to be flooded to the right ports. The ovs-trace on the other hand did not show the same picture. It just did 4k recirculations and then dropped the packet. I already restarted the ovn-controller on the right hv, but that did not do anything.
The LSP:
$ ovn-nbctl list Logical_Switch_Port cfce175b-9d88-4c2e-a5cc-d76cd5c71deb _uuid : c5dfb248-941e-4d4e-af1a-9ccafc22db70 addresses : ["fa:16:3e:a2:d7:1a 2a02:ed80:0:3::341 91.198.2.33"fa:16:3e:a2:d7:1a2a02:ed80:0:3::34191.198.2.33] dhcpv4_options : 1922ee38-282f-4f5c-ade8-6cd157ee52e9 dhcpv6_options : [] dynamic_addresses : [] enabled : true external_ids : {"neutron:cidrs"neutron:cidrs="2a02:ed80:0:3::341/64 91.198.2.33/24", "neutron:device_id"neutron:device_id="8062ec61-0c68-41dd-b77c-e8b72ad16a88", "neutron:device_owner"neutron:device_owner="compute:AZ1"compute:AZ1, "neutron:network_name"neutron:network_name=neutron-210e26d7-942f-4e17-89b2-571eee87d7e4, "neutron:port_name"neutron:port_name="", "neutron:project_id"neutron:project_id="99fb21796a8f4cbda42ba5b9d1e307dd", "neutron:revision_number"neutron:revision_number="16", "neutron:security_group_ids"neutron:security_group_ids="3e41777f-7aa4-4368-9992-5ca7cc2a5372 873b3b62-0918-4b1e-be73-fdbed50d2ac2"} ha_chassis_group : [] name : "cfce175b-9d88-4c2e-a5cc-d76cd5c71deb" options : {mcast_flood_reports="true", requested-chassis=net-openstack-hv31} parent_name : [] port_security : ["fa:16:3e:a2:d7:1a 2a02:ed80:0:3::341 91.198.2.33"fa:16:3e:a2:d7:1a2a02:ed80:0:3::34191.198.2.33] tag : [] tag_request : [] type : "" up : true
The PB:
$ ovn-sbctl find Port_Binding logical_port=cfce175b-9d88-4c2e-a5cc-d76cd5c71deb _uuid : e9e5ce44-698f-4a29-acd1-2f24cc1d1950 chassis : c944c21a-3344-4fda-ab4e-a4cc07403125 datapath : 993b44d5-1629-4e9b-b44e-24096d8b3959 encap : [] external_ids : {"neutron:cidrs"neutron:cidrs="2a02:ed80:0:3::341/64 91.198.2.33/24", "neutron:device_id"neutron:device_id="8062ec61-0c68-41dd-b77c-e8b72ad16a88", "neutron:device_owner"neutron:device_owner="compute:AZ1"compute:AZ1, "neutron:network_name"neutron:network_name=neutron-210e26d7-942f-4e17-89b2-571eee87d7e4, "neutron:port_name"neutron:port_name="", "neutron:project_id"neutron:project_id="99fb21796a8f4cbda42ba5b9d1e307dd", "neutron:revision_number"neutron:revision_number="16", "neutron:security_group_ids"neutron:security_group_ids="3e41777f-7aa4-4368-9992-5ca7cc2a5372 873b3b62-0918-4b1e-be73-fdbed50d2ac2"} gateway_chassis : [] ha_chassis_group : [] logical_port : "cfce175b-9d88-4c2e-a5cc-d76cd5c71deb" mac : ["fa:16:3e:a2:d7:1a 2a02:ed80:0:3::341 91.198.2.33"fa:16:3e:a2:d7:1a2a02:ed80:0:3::34191.198.2.33] nat_addresses : [] options : {mcast_flood_reports="true", requested-chassis=net-openstack-hv31} parent_port : [] requested_chassis : c944c21a-3344-4fda-ab4e-a4cc07403125 tag : [] tunnel_key : 344 type : "" up : true virtual_parent : []
The LS:
$ ovn-nbctl list Logical_Switch public-network _uuid : 56d8be55-462a-4b93-8710-3c79ca386213 acls : [] copp : [] dns_records : [] external_ids : {"neutron:mtu"neutron:mtu="1500", "neutron:network_name"neutron:network_name=public-network, "neutron:revision_number"neutron:revision_number="21"} forwarding_groups : [] load_balancer : [] load_balancer_group : [] name : neutron-210e26d7-942f-4e17-89b2-571eee87d7e4 other_config : {mcast_flood_unregistered="false", mcast_snoop="false"} ports : [00225774-8fbc-473f-ae5e-d486c54212c8, ..., c5dfb248-941e-4d4e-af1a-9ccafc22db70, ... qos_rules : []
The patchport:
$ ovn-nbctl list Logical_Switch_Port provnet-aa35051c-6fc0-463a-8807-0cb28903be14 _uuid : f7259aeb-0e63-4d20-8a8e-54ebf454a524 addresses : [unknown] dhcpv4_options : [] dhcpv6_options : [] dynamic_addresses : [] enabled : [] external_ids : {} ha_chassis_group : [] name : provnet-aa35051c-6fc0-463a-8807-0cb28903be14 options : {mcast_flood="false", mcast_flood_reports="true", network_name=physnet1} parent_name : [] port_security : [] tag : [] tag_request : [] type : localnet up : false
I hope I provided the needed context! Thanks in advance!
Best regards, Justin Lamp
-- Justin Lamp Systems Engineer
NETWAYS Managed Services GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg Tel: +49 911 92885-0 | Fax: +49 911 92885-77 CEO: Julian Hein, Bernd Erk, Sebastian Saemann | AG Nuernberg HRB25207 https://www.netways.de | justin.lamp@netways.de
** stackconf 2023 - September - https://stackconf.eu ** ** OSMC 2023 - November - https://osmc.de ** ** NETWAYS Web Services - https://nws.netways.de ** ** NETWAYS Trainings - https://netways.de/trainings **
Hi Justin,
It's a shot in the dark, but one scenario that I experienced was related to the selected TC qdisc. The behavior (ARP not getting through despite what ovn-trace suggests) showed with `fq` TC qdisc used, and switching to `fq_codel` fixed the problem. You may want to try another TC discipline.
Something like: tc qdisc replace dev eth0 root fq_codel
Where eth0 is your NIC for the provider network.
Ihar
On Fri, Aug 4, 2023 at 2:38 PM Justin Lamp justin.lamp@netways.de wrote:
Hey,
we are using OVN 22.03 and face an issue where a VM that is directly connected to the provider network won't be accessible, because it cannot arp for the Gateway IP. OVN routers do reply to the arp request though. We know that this exact scenario works as we have it running in our staging environment.
Oddly enough if the right MAC-IP Binding is manually defined within the VM and the Gateway, the traffic will begin to flow correctly according to the right SGs.
I did an ovn-trace and were able to see that the traffic is supposed to be flooded to the right ports. The ovs-trace on the other hand did not show the same picture. It just did 4k recirculations and then dropped the packet. I already restarted the ovn-controller on the right hv, but that did not do anything.
The LSP:
$ ovn-nbctl list Logical_Switch_Port cfce175b-9d88-4c2e-a5cc-d76cd5c71deb _uuid : c5dfb248-941e-4d4e-af1a-9ccafc22db70 addresses : ["fa:16:3e:a2:d7:1a 2a02:ed80:0:3::341 91.198.2.33"] dhcpv4_options : 1922ee38-282f-4f5c-ade8-6cd157ee52e9 dhcpv6_options : [] dynamic_addresses : [] enabled : true external_ids : {"neutron:cidrs"="2a02:ed80:0:3::341/64 91.198.2.33/24", "neutron:device_id"="8062ec61-0c68-41dd-b77c-e8b72ad16a88", "neutron:device_owner"="compute:AZ1", "neutron:network_name"=neutron-210e26d7-942f-4e17-89b2-571eee87d7e4, "neutron:port_name"="", "neutron:project_id"="99fb21796a8f4cbda42ba5b9d1e307dd", "neutron:revision_number"="16", "neutron:security_group_ids"="3e41777f-7aa4-4368-9992-5ca7cc2a5372 873b3b62-0918-4b1e-be73-fdbed50d2ac2"} ha_chassis_group : [] name : "cfce175b-9d88-4c2e-a5cc-d76cd5c71deb" options : {mcast_flood_reports="true", requested-chassis=net-openstack-hv31} parent_name : [] port_security : ["fa:16:3e:a2:d7:1a 2a02:ed80:0:3::341 91.198.2.33"] tag : [] tag_request : [] type : "" up : true
The PB:
$ ovn-sbctl find Port_Binding logical_port=cfce175b-9d88-4c2e-a5cc-d76cd5c71deb _uuid : e9e5ce44-698f-4a29-acd1-2f24cc1d1950 chassis : c944c21a-3344-4fda-ab4e-a4cc07403125 datapath : 993b44d5-1629-4e9b-b44e-24096d8b3959 encap : [] external_ids : {"neutron:cidrs"="2a02:ed80:0:3::341/64 91.198.2.33/24", "neutron:device_id"="8062ec61-0c68-41dd-b77c-e8b72ad16a88", "neutron:device_owner"="compute:AZ1", "neutron:network_name"=neutron-210e26d7-942f-4e17-89b2-571eee87d7e4, "neutron:port_name"="", "neutron:project_id"="99fb21796a8f4cbda42ba5b9d1e307dd", "neutron:revision_number"="16", "neutron:security_group_ids"="3e41777f-7aa4-4368-9992-5ca7cc2a5372 873b3b62-0918-4b1e-be73-fdbed50d2ac2"} gateway_chassis : [] ha_chassis_group : [] logical_port : "cfce175b-9d88-4c2e-a5cc-d76cd5c71deb" mac : ["fa:16:3e:a2:d7:1a 2a02:ed80:0:3::341 91.198.2.33"] nat_addresses : [] options : {mcast_flood_reports="true", requested-chassis=net-openstack-hv31} parent_port : [] requested_chassis : c944c21a-3344-4fda-ab4e-a4cc07403125 tag : [] tunnel_key : 344 type : "" up : true virtual_parent : []
The LS:
$ ovn-nbctl list Logical_Switch public-network _uuid : 56d8be55-462a-4b93-8710-3c79ca386213 acls : [] copp : [] dns_records : [] external_ids : {"neutron:mtu"="1500", "neutron:network_name"=public-network, "neutron:revision_number"="21"} forwarding_groups : [] load_balancer : [] load_balancer_group : [] name : neutron-210e26d7-942f-4e17-89b2-571eee87d7e4 other_config : {mcast_flood_unregistered="false", mcast_snoop="false"} ports : [00225774-8fbc-473f-ae5e-d486c54212c8, ..., c5dfb248-941e-4d4e-af1a-9ccafc22db70, ... qos_rules : []
The patchport:
$ ovn-nbctl list Logical_Switch_Port provnet-aa35051c-6fc0-463a-8807-0cb28903be14 _uuid : f7259aeb-0e63-4d20-8a8e-54ebf454a524 addresses : [unknown] dhcpv4_options : [] dhcpv6_options : [] dynamic_addresses : [] enabled : [] external_ids : {} ha_chassis_group : [] name : provnet-aa35051c-6fc0-463a-8807-0cb28903be14 options : {mcast_flood="false", mcast_flood_reports="true", network_name=physnet1} parent_name : [] port_security : [] tag : [] tag_request : [] type : localnet up : false
I hope I provided the needed context! Thanks in advance!
Best regards, Justin Lamp
-- Justin Lamp Systems Engineer
NETWAYS Managed Services GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg Tel: +49 911 92885-0 | Fax: +49 911 92885-77 CEO: Julian Hein, Bernd Erk, Sebastian Saemann | AG Nuernberg HRB25207 https://www.netways.de | justin.lamp@netways.de
** stackconf 2023 - September - https://stackconf.eu ** ** OSMC 2023 - November - https://osmc.de ** ** NETWAYS Web Services - https://nws.netways.de ** ** NETWAYS Trainings - https://netways.de/trainings **
Hi Justin,
i guess your "public-network" logical switch has a large number of ports connected to it (somewhere between 200 and 300 or more).
In this case you might be interested in this fix [1]. It allows you to configure the "public-network" logical switch to not broadcast arp requests to all routers and instead only send it to non-router ports (Arp requests for LRPs will still be forwarded to that individual router).
Note that this setting will break GARPs on this network. Not sure if that would be an issue for you.
I have never tested this with a VM that is connected directly on a provider network but i would guess it should still work.
Regards Felix
[1]: https://github.com/ovn-org/ovn/commit/37d308a2074515834692d442475a8e05310a15... On Fri, Aug 04, 2023 at 01:30:45PM +0000, Justin Lamp wrote:
Hey,
we are using OVN 22.03 and face an issue where a VM that is directly connected to the provider network won't be accessible, because it cannot arp for the Gateway IP. OVN routers do reply to the arp request though. We know that this exact scenario works as we have it running in our staging environment.
Oddly enough if the right MAC-IP Binding is manually defined within the VM and the Gateway, the traffic will begin to flow correctly according to the right SGs.
I did an ovn-trace and were able to see that the traffic is supposed to be flooded to the right ports. The ovs-trace on the other hand did not show the same picture. It just did 4k recirculations and then dropped the packet. I already restarted the ovn-controller on the right hv, but that did not do anything.
The LSP:
$ ovn-nbctl list Logical_Switch_Port cfce175b-9d88-4c2e-a5cc-d76cd5c71deb _uuid : c5dfb248-941e-4d4e-af1a-9ccafc22db70 addresses : ["fa:16:3e:a2:d7:1a 2a02:ed80:0:3::341 91.198.2.33"fa:16:3e:a2:d7:1a2a02:ed80:0:3::34191.198.2.33] dhcpv4_options : 1922ee38-282f-4f5c-ade8-6cd157ee52e9 dhcpv6_options : [] dynamic_addresses : [] enabled : true external_ids : {"neutron:cidrs"neutron:cidrs="2a02:ed80:0:3::341/64 91.198.2.33/24", "neutron:device_id"neutron:device_id="8062ec61-0c68-41dd-b77c-e8b72ad16a88", "neutron:device_owner"neutron:device_owner="compute:AZ1"compute:AZ1, "neutron:network_name"neutron:network_name=neutron-210e26d7-942f-4e17-89b2-571eee87d7e4, "neutron:port_name"neutron:port_name="", "neutron:project_id"neutron:project_id="99fb21796a8f4cbda42ba5b9d1e307dd", "neutron:revision_number"neutron:revision_number="16", "neutron:security_group_ids"neutron:security_group_ids="3e41777f-7aa4-4368-9992-5ca7cc2a5372 873b3b62-0918-4b1e-be73-fdbed50d2ac2"} ha_chassis_group : [] name : "cfce175b-9d88-4c2e-a5cc-d76cd5c71deb" options : {mcast_flood_reports="true", requested-chassis=net-openstack-hv31} parent_name : [] port_security : ["fa:16:3e:a2:d7:1a 2a02:ed80:0:3::341 91.198.2.33"fa:16:3e:a2:d7:1a2a02:ed80:0:3::34191.198.2.33] tag : [] tag_request : [] type : "" up : true
The PB:
$ ovn-sbctl find Port_Binding logical_port=cfce175b-9d88-4c2e-a5cc-d76cd5c71deb _uuid : e9e5ce44-698f-4a29-acd1-2f24cc1d1950 chassis : c944c21a-3344-4fda-ab4e-a4cc07403125 datapath : 993b44d5-1629-4e9b-b44e-24096d8b3959 encap : [] external_ids : {"neutron:cidrs"neutron:cidrs="2a02:ed80:0:3::341/64 91.198.2.33/24", "neutron:device_id"neutron:device_id="8062ec61-0c68-41dd-b77c-e8b72ad16a88", "neutron:device_owner"neutron:device_owner="compute:AZ1"compute:AZ1, "neutron:network_name"neutron:network_name=neutron-210e26d7-942f-4e17-89b2-571eee87d7e4, "neutron:port_name"neutron:port_name="", "neutron:project_id"neutron:project_id="99fb21796a8f4cbda42ba5b9d1e307dd", "neutron:revision_number"neutron:revision_number="16", "neutron:security_group_ids"neutron:security_group_ids="3e41777f-7aa4-4368-9992-5ca7cc2a5372 873b3b62-0918-4b1e-be73-fdbed50d2ac2"} gateway_chassis : [] ha_chassis_group : [] logical_port : "cfce175b-9d88-4c2e-a5cc-d76cd5c71deb" mac : ["fa:16:3e:a2:d7:1a 2a02:ed80:0:3::341 91.198.2.33"fa:16:3e:a2:d7:1a2a02:ed80:0:3::34191.198.2.33] nat_addresses : [] options : {mcast_flood_reports="true", requested-chassis=net-openstack-hv31} parent_port : [] requested_chassis : c944c21a-3344-4fda-ab4e-a4cc07403125 tag : [] tunnel_key : 344 type : "" up : true virtual_parent : []
The LS:
$ ovn-nbctl list Logical_Switch public-network _uuid : 56d8be55-462a-4b93-8710-3c79ca386213 acls : [] copp : [] dns_records : [] external_ids : {"neutron:mtu"neutron:mtu="1500", "neutron:network_name"neutron:network_name=public-network, "neutron:revision_number"neutron:revision_number="21"} forwarding_groups : [] load_balancer : [] load_balancer_group : [] name : neutron-210e26d7-942f-4e17-89b2-571eee87d7e4 other_config : {mcast_flood_unregistered="false", mcast_snoop="false"} ports : [00225774-8fbc-473f-ae5e-d486c54212c8, ..., c5dfb248-941e-4d4e-af1a-9ccafc22db70, ... qos_rules : []
The patchport:
$ ovn-nbctl list Logical_Switch_Port provnet-aa35051c-6fc0-463a-8807-0cb28903be14 _uuid : f7259aeb-0e63-4d20-8a8e-54ebf454a524 addresses : [unknown] dhcpv4_options : [] dhcpv6_options : [] dynamic_addresses : [] enabled : [] external_ids : {} ha_chassis_group : [] name : provnet-aa35051c-6fc0-463a-8807-0cb28903be14 options : {mcast_flood="false", mcast_flood_reports="true", network_name=physnet1} parent_name : [] port_security : [] tag : [] tag_request : [] type : localnet up : false
I hope I provided the needed context! Thanks in advance!
Best regards, Justin Lamp
-- Justin Lamp Systems Engineer
NETWAYS Managed Services GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg Tel: +49 911 92885-0 | Fax: +49 911 92885-77 CEO: Julian Hein, Bernd Erk, Sebastian Saemann | AG Nuernberg HRB25207 https://www.netways.de | justin.lamp@netways.de
** stackconf 2023 - September - https://stackconf.eu ** ** OSMC 2023 - November - https://osmc.de ** ** NETWAYS Web Services - https://nws.netways.de ** ** NETWAYS Trainings - https://netways.de/trainings **
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail.
Hinweise zum Datenschutz finden Sie hierhttps://www.datenschutz.schwarz.
This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail.
Information on data protection can be found herehttps://www.datenschutz.schwarz.
Hi Felix,
thank you so much for pointing me to your MR. We finally got around to updating OVN and that fixes it!
Best regards, Justin Lamp
Am 08.08.23 um 14:18 schrieb Felix Huettner:
Hi Justin,
i guess your "public-network" logical switch has a large number of ports connected to it (somewhere between 200 and 300 or more).
In this case you might be interested in this fix [1]. It allows you to configure the "public-network" logical switch to not broadcast arp requests to all routers and instead only send it to non-router ports (Arp requests for LRPs will still be forwarded to that individual router).
Note that this setting will break GARPs on this network. Not sure if that would be an issue for you.
I have never tested this with a VM that is connected directly on a provider network but i would guess it should still work.
Regards Felix
On Fri, Aug 04, 2023 at 01:30:45PM +0000, Justin Lamp wrote:
Hey,
we are using OVN 22.03 and face an issue where a VM that is directly connected to the provider network won't be accessible, because it cannot arp for the Gateway IP. OVN routers do reply to the arp request though. We know that this exact scenario works as we have it running in our staging environment.
Oddly enough if the right MAC-IP Binding is manually defined within the VM and the Gateway, the traffic will begin to flow correctly according to the right SGs.
I did an ovn-trace and were able to see that the traffic is supposed to be flooded to the right ports. The ovs-trace on the other hand did not show the same picture. It just did 4k recirculations and then dropped the packet. I already restarted the ovn-controller on the right hv, but that did not do anything.
The LSP:
$ ovn-nbctl list Logical_Switch_Port cfce175b-9d88-4c2e-a5cc-d76cd5c71deb _uuid : c5dfb248-941e-4d4e-af1a-9ccafc22db70 addresses : ["fa:16:3e:a2:d7:1a 2a02:ed80:0:3::341 91.198.2.33"fa:16:3e:a2:d7:1a2a02:ed80:0:3::34191.198.2.33] dhcpv4_options : 1922ee38-282f-4f5c-ade8-6cd157ee52e9 dhcpv6_options : [] dynamic_addresses : [] enabled : true external_ids : {"neutron:cidrs"neutron:cidrs="2a02:ed80:0:3::341/64 91.198.2.33/24", "neutron:device_id"neutron:device_id="8062ec61-0c68-41dd-b77c-e8b72ad16a88", "neutron:device_owner"neutron:device_owner="compute:AZ1"compute:AZ1, "neutron:network_name"neutron:network_name=neutron-210e26d7-942f-4e17-89b2-571eee87d7e4, "neutron:port_name"neutron:port_name="", "neutron:project_id"neutron:project_id="99fb21796a8f4cbda42ba5b9d1e307dd", "neutron:revision_number"neutron:revision_number="16", "neutron:security_group_ids"neutron:security_group_ids="3e41777f-7aa4-4368-9992-5ca7cc2a5372 873b3b62-0918-4b1e-be73-fdbed50d2ac2"} ha_chassis_group : [] name : "cfce175b-9d88-4c2e-a5cc-d76cd5c71deb" options : {mcast_flood_reports="true", requested-chassis=net-openstack-hv31} parent_name : [] port_security : ["fa:16:3e:a2:d7:1a 2a02:ed80:0:3::341 91.198.2.33"fa:16:3e:a2:d7:1a2a02:ed80:0:3::34191.198.2.33] tag : [] tag_request : [] type : "" up : true
The PB:
$ ovn-sbctl find Port_Binding logical_port=cfce175b-9d88-4c2e-a5cc-d76cd5c71deb _uuid : e9e5ce44-698f-4a29-acd1-2f24cc1d1950 chassis : c944c21a-3344-4fda-ab4e-a4cc07403125 datapath : 993b44d5-1629-4e9b-b44e-24096d8b3959 encap : [] external_ids : {"neutron:cidrs"neutron:cidrs="2a02:ed80:0:3::341/64 91.198.2.33/24", "neutron:device_id"neutron:device_id="8062ec61-0c68-41dd-b77c-e8b72ad16a88", "neutron:device_owner"neutron:device_owner="compute:AZ1"compute:AZ1, "neutron:network_name"neutron:network_name=neutron-210e26d7-942f-4e17-89b2-571eee87d7e4, "neutron:port_name"neutron:port_name="", "neutron:project_id"neutron:project_id="99fb21796a8f4cbda42ba5b9d1e307dd", "neutron:revision_number"neutron:revision_number="16", "neutron:security_group_ids"neutron:security_group_ids="3e41777f-7aa4-4368-9992-5ca7cc2a5372 873b3b62-0918-4b1e-be73-fdbed50d2ac2"} gateway_chassis : [] ha_chassis_group : [] logical_port : "cfce175b-9d88-4c2e-a5cc-d76cd5c71deb" mac : ["fa:16:3e:a2:d7:1a 2a02:ed80:0:3::341 91.198.2.33"fa:16:3e:a2:d7:1a2a02:ed80:0:3::34191.198.2.33] nat_addresses : [] options : {mcast_flood_reports="true", requested-chassis=net-openstack-hv31} parent_port : [] requested_chassis : c944c21a-3344-4fda-ab4e-a4cc07403125 tag : [] tunnel_key : 344 type : "" up : true virtual_parent : []
The LS:
$ ovn-nbctl list Logical_Switch public-network _uuid : 56d8be55-462a-4b93-8710-3c79ca386213 acls : [] copp : [] dns_records : [] external_ids : {"neutron:mtu"neutron:mtu="1500", "neutron:network_name"neutron:network_name=public-network, "neutron:revision_number"neutron:revision_number="21"} forwarding_groups : [] load_balancer : [] load_balancer_group : [] name : neutron-210e26d7-942f-4e17-89b2-571eee87d7e4 other_config : {mcast_flood_unregistered="false", mcast_snoop="false"} ports : [00225774-8fbc-473f-ae5e-d486c54212c8, ..., c5dfb248-941e-4d4e-af1a-9ccafc22db70, ... qos_rules : []
The patchport:
$ ovn-nbctl list Logical_Switch_Port provnet-aa35051c-6fc0-463a-8807-0cb28903be14 _uuid : f7259aeb-0e63-4d20-8a8e-54ebf454a524 addresses : [unknown] dhcpv4_options : [] dhcpv6_options : [] dynamic_addresses : [] enabled : [] external_ids : {} ha_chassis_group : [] name : provnet-aa35051c-6fc0-463a-8807-0cb28903be14 options : {mcast_flood="false", mcast_flood_reports="true", network_name=physnet1} parent_name : [] port_security : [] tag : [] tag_request : [] type : localnet up : false
I hope I provided the needed context! Thanks in advance!
Best regards, Justin Lamp
-- Justin Lamp Systems Engineer
NETWAYS Managed Services GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg Tel: +49 911 92885-0 | Fax: +49 911 92885-77 CEO: Julian Hein, Bernd Erk, Sebastian Saemann | AG Nuernberg HRB25207 https://www.netways.de | justin.lamp@netways.de
** stackconf 2023 - September - https://stackconf.eu ** ** OSMC 2023 - November - https://osmc.de ** ** NETWAYS Web Services - https://nws.netways.de ** ** NETWAYS Trainings - https://netways.de/trainings **
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail.
Hinweise zum Datenschutz finden Sie hierhttps://www.datenschutz.schwarz.
This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail.
Information on data protection can be found herehttps://www.datenschutz.schwarz.
-- Justin Lamp Systems Engineer
NETWAYS Managed Services GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg Tel: +49 911 92885-0 | Fax: +49 911 92885-77 CEO: Julian Hein, Bernd Erk, Sebastian Saemann | AG Nuernberg HRB25207 https://www.netways.de | justin.lamp@netways.de
** stackconf 2023 - September - https://stackconf.eu ** ** OSMC 2023 - November - https://osmc.de ** ** NETWAYS Web Services - https://nws.netways.de ** ** NETWAYS Trainings - https://netways.de/trainings **
Interesting, updated OVN from what to what?
On Tue, Aug 22, 2023 at 2:45 AM Justin Lamp justin.lamp@netways.de wrote:
Hi Felix,
thank you so much for pointing me to your MR. We finally got around to updating OVN and that fixes it!
Best regards, Justin Lamp
Am 08.08.23 um 14:18 schrieb Felix Huettner:
Hi Justin,
i guess your "public-network" logical switch has a large number of ports connected to it (somewhere between 200 and 300 or more).
In this case you might be interested in this fix [1]. It allows you to configure the "public-network" logical switch to not broadcast arp requests to all routers and instead only send it to non-router ports (Arp requests for LRPs will still be forwarded to that individual
router).
Note that this setting will break GARPs on this network. Not sure if that would be an issue for you.
I have never tested this with a VM that is connected directly on a provider network but i would guess it should still work.
Regards Felix
[1]:
https://github.com/ovn-org/ovn/commit/37d308a2074515834692d442475a8e05310a15...
On Fri, Aug 04, 2023 at 01:30:45PM +0000, Justin Lamp wrote:
Hey,
we are using OVN 22.03 and face an issue where a VM that is directly
connected to the provider network won't be accessible, because it cannot arp for the Gateway IP. OVN routers do reply to the arp request though. We know that this exact scenario works as we have it running in our staging environment.
Oddly enough if the right MAC-IP Binding is manually defined within the
VM and the Gateway, the traffic will begin to flow correctly according to the right SGs.
I did an ovn-trace and were able to see that the traffic is supposed to
be flooded to the right ports. The ovs-trace on the other hand did not show the same picture. It just did 4k recirculations and then dropped the packet. I already restarted the ovn-controller on the right hv, but that did not do anything.
The LSP:
$ ovn-nbctl list Logical_Switch_Port
cfce175b-9d88-4c2e-a5cc-d76cd5c71deb
_uuid : c5dfb248-941e-4d4e-af1a-9ccafc22db70 addresses : ["fa:16:3e:a2:d7:1a 2a02:ed80:0:3::341
91.198.2.33"fa:16:3e:a2:d7:1a2a02:ed80:0:3::34191.198.2.33]
dhcpv4_options : 1922ee38-282f-4f5c-ade8-6cd157ee52e9 dhcpv6_options : [] dynamic_addresses : [] enabled : true external_ids :
{"neutron:cidrs"neutron:cidrs="2a02:ed80:0:3::341/64 91.198.2.33/24", "neutron:device_id"neutron:device_id="8062ec61-0c68-41dd-b77c-e8b72ad16a88", "neutron:device_owner"neutron:device_owner="compute:AZ1"compute:AZ1, "neutron:network_name"neutron:network_name=neutron-210e26d7-942f-4e17-89b2-571eee87d7e4, "neutron:port_name"neutron:port_name="", "neutron:project_id"neutron:project_id="99fb21796a8f4cbda42ba5b9d1e307dd", "neutron:revision_number"neutron:revision_number="16", "neutron:security_group_ids"neutron:security_group_ids="3e41777f-7aa4-4368-9992-5ca7cc2a5372 873b3b62-0918-4b1e-be73-fdbed50d2ac2"}
ha_chassis_group : [] name : "cfce175b-9d88-4c2e-a5cc-d76cd5c71deb" options : {mcast_flood_reports="true",
requested-chassis=net-openstack-hv31}
parent_name : [] port_security : ["fa:16:3e:a2:d7:1a 2a02:ed80:0:3::341
91.198.2.33"fa:16:3e:a2:d7:1a2a02:ed80:0:3::34191.198.2.33]
tag : [] tag_request : [] type : "" up : true
The PB:
$ ovn-sbctl find Port_Binding
logical_port=cfce175b-9d88-4c2e-a5cc-d76cd5c71deb
_uuid : e9e5ce44-698f-4a29-acd1-2f24cc1d1950 chassis : c944c21a-3344-4fda-ab4e-a4cc07403125 datapath : 993b44d5-1629-4e9b-b44e-24096d8b3959 encap : [] external_ids :
{"neutron:cidrs"neutron:cidrs="2a02:ed80:0:3::341/64 91.198.2.33/24", "neutron:device_id"neutron:device_id="8062ec61-0c68-41dd-b77c-e8b72ad16a88", "neutron:device_owner"neutron:device_owner="compute:AZ1"compute:AZ1, "neutron:network_name"neutron:network_name=neutron-210e26d7-942f-4e17-89b2-571eee87d7e4, "neutron:port_name"neutron:port_name="", "neutron:project_id"neutron:project_id="99fb21796a8f4cbda42ba5b9d1e307dd", "neutron:revision_number"neutron:revision_number="16", "neutron:security_group_ids"neutron:security_group_ids="3e41777f-7aa4-4368-9992-5ca7cc2a5372 873b3b62-0918-4b1e-be73-fdbed50d2ac2"}
gateway_chassis : [] ha_chassis_group : [] logical_port : "cfce175b-9d88-4c2e-a5cc-d76cd5c71deb" mac : ["fa:16:3e:a2:d7:1a 2a02:ed80:0:3::341
91.198.2.33"fa:16:3e:a2:d7:1a2a02:ed80:0:3::34191.198.2.33]
nat_addresses : [] options : {mcast_flood_reports="true",
requested-chassis=net-openstack-hv31}
parent_port : [] requested_chassis : c944c21a-3344-4fda-ab4e-a4cc07403125 tag : [] tunnel_key : 344 type : "" up : true virtual_parent : []
The LS:
$ ovn-nbctl list Logical_Switch public-network _uuid : 56d8be55-462a-4b93-8710-3c79ca386213 acls : [] copp : [] dns_records : [] external_ids : {"neutron:mtu"neutron:mtu="1500",
"neutron:network_name"neutron:network_name=public-network, "neutron:revision_number"neutron:revision_number="21"}
forwarding_groups : [] load_balancer : [] load_balancer_group : [] name : neutron-210e26d7-942f-4e17-89b2-571eee87d7e4 other_config : {mcast_flood_unregistered="false",
mcast_snoop="false"}
ports : [00225774-8fbc-473f-ae5e-d486c54212c8, ...,
c5dfb248-941e-4d4e-af1a-9ccafc22db70, ...
qos_rules : []
The patchport:
$ ovn-nbctl list Logical_Switch_Port
provnet-aa35051c-6fc0-463a-8807-0cb28903be14
_uuid : f7259aeb-0e63-4d20-8a8e-54ebf454a524 addresses : [unknown] dhcpv4_options : [] dhcpv6_options : [] dynamic_addresses : [] enabled : [] external_ids : {} ha_chassis_group : [] name : provnet-aa35051c-6fc0-463a-8807-0cb28903be14 options : {mcast_flood="false", mcast_flood_reports="true",
network_name=physnet1}
parent_name : [] port_security : [] tag : [] tag_request : [] type : localnet up : false
I hope I provided the needed context! Thanks in advance!
Best regards, Justin Lamp
-- Justin Lamp Systems Engineer
NETWAYS Managed Services GmbH | Deutschherrnstr. 15-19 | D-90429
Nuernberg
Tel: +49 911 92885-0 | Fax: +49 911 92885-77 CEO: Julian Hein, Bernd Erk, Sebastian Saemann | AG Nuernberg HRB25207 https://www.netways.de | justin.lamp@netways.de
** stackconf 2023 - September - https://stackconf.eu ** ** OSMC 2023 - November - https://osmc.de ** ** NETWAYS Web Services - https://nws.netways.de ** ** NETWAYS Trainings - https://netways.de/trainings **
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für
die Verwertung durch den vorgesehenen Empfänger bestimmt.
Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den
Absender bitte unverzüglich in Kenntnis und löschen diese E Mail.
Hinweise zum Datenschutz finden Sie hier<https://www.datenschutz.schwarz .
This e-mail may contain confidential content and is intended only for
the specified recipient/s.
If you are not the intended recipient, please inform the sender
immediately and delete this e-mail.
Information on data protection can be found here<
https://www.datenschutz.schwarz%3E.
-- Justin Lamp Systems Engineer
NETWAYS Managed Services GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg Tel: +49 911 92885-0 | Fax: +49 911 92885-77 CEO: Julian Hein, Bernd Erk, Sebastian Saemann | AG Nuernberg HRB25207 https://www.netways.de | justin.lamp@netways.de
** stackconf 2023 - September - https://stackconf.eu ** ** OSMC 2023 - November - https://osmc.de ** ** NETWAYS Web Services - https://nws.netways.de ** ** NETWAYS Trainings - https://netways.de/trainings **
I updated our 22.03 installation directly to 23.06, although we propably could have just used the patch on 22.03 directly. But we wanted those nice Live-Migration improvements!
Am 22.08.23 um 14:21 schrieb Satish Patel: Interesting, updated OVN from what to what?
On Tue, Aug 22, 2023 at 2:45 AM Justin Lamp <justin.lamp@netways.demailto:justin.lamp@netways.de> wrote: Hi Felix,
thank you so much for pointing me to your MR. We finally got around to updating OVN and that fixes it!
Best regards, Justin Lamp
Am 08.08.23 um 14:18 schrieb Felix Huettner:
Hi Justin,
i guess your "public-network" logical switch has a large number of ports connected to it (somewhere between 200 and 300 or more).
In this case you might be interested in this fix [1]. It allows you to configure the "public-network" logical switch to not broadcast arp requests to all routers and instead only send it to non-router ports (Arp requests for LRPs will still be forwarded to that individual router).
Note that this setting will break GARPs on this network. Not sure if that would be an issue for you.
I have never tested this with a VM that is connected directly on a provider network but i would guess it should still work.
Regards Felix
On Fri, Aug 04, 2023 at 01:30:45PM +0000, Justin Lamp wrote:
Hey,
we are using OVN 22.03 and face an issue where a VM that is directly connected to the provider network won't be accessible, because it cannot arp for the Gateway IP. OVN routers do reply to the arp request though. We know that this exact scenario works as we have it running in our staging environment.
Oddly enough if the right MAC-IP Binding is manually defined within the VM and the Gateway, the traffic will begin to flow correctly according to the right SGs.
I did an ovn-trace and were able to see that the traffic is supposed to be flooded to the right ports. The ovs-trace on the other hand did not show the same picture. It just did 4k recirculations and then dropped the packet. I already restarted the ovn-controller on the right hv, but that did not do anything.
The LSP:
$ ovn-nbctl list Logical_Switch_Port cfce175b-9d88-4c2e-a5cc-d76cd5c71deb _uuid : c5dfb248-941e-4d4e-af1a-9ccafc22db70 addresses : ["fa:16:3e:a2:d7:1a 2a02:ed80:0:3::341 91.198.2.33"fa:16:3e:a2:d7:1a2a02:ed80:0:3::34191.198.2.33fa:16:3e:a2:d7:1a2a02:ed80:0:3::34191.198.2.33fa:16:3e:a2:d7:1a2a02:ed80:0:3::34191.198.2.33] dhcpv4_options : 1922ee38-282f-4f5c-ade8-6cd157ee52e9 dhcpv6_options : [] dynamic_addresses : [] enabled : true external_ids : {"neutron:cidrs"neutron:cidrsneutron:cidrsneutron:cidrs="2a02:ed80:0:3::341/64 91.198.2.33/24http://91.198.2.33/24", "neutron:device_id"neutron:device_idneutron:device_idneutron:device_id="8062ec61-0c68-41dd-b77c-e8b72ad16a88", "neutron:device_owner"neutron:device_ownerneutron:device_ownerneutron:device_owner="compute:AZ1"compute:AZ1compute:AZ1compute:AZ1, "neutron:network_name"neutron:network_nameneutron:network_nameneutron:network_name=neutron-210e26d7-942f-4e17-89b2-571eee87d7e4, "neutron:port_name"neutron:port_nameneutron:port_nameneutron:port_name="", "neutron:project_id"neutron:project_idneutron:project_idneutron:project_id="99fb21796a8f4cbda42ba5b9d1e307dd", "neutron:revision_number"neutron:revision_numberneutron:revision_numberneutron:revision_number="16", "neutron:security_group_ids"neutron:security_group_idsneutron:security_group_idsneutron:security_group_ids="3e41777f-7aa4-4368-9992-5ca7cc2a5372 873b3b62-0918-4b1e-be73-fdbed50d2ac2"} ha_chassis_group : [] name : "cfce175b-9d88-4c2e-a5cc-d76cd5c71deb" options : {mcast_flood_reports="true", requested-chassis=net-openstack-hv31} parent_name : [] port_security : ["fa:16:3e:a2:d7:1a 2a02:ed80:0:3::341 91.198.2.33"fa:16:3e:a2:d7:1a2a02:ed80:0:3::34191.198.2.33fa:16:3e:a2:d7:1a2a02:ed80:0:3::34191.198.2.33fa:16:3e:a2:d7:1a2a02:ed80:0:3::34191.198.2.33] tag : [] tag_request : [] type : "" up : true
The PB:
$ ovn-sbctl find Port_Binding logical_port=cfce175b-9d88-4c2e-a5cc-d76cd5c71deb _uuid : e9e5ce44-698f-4a29-acd1-2f24cc1d1950 chassis : c944c21a-3344-4fda-ab4e-a4cc07403125 datapath : 993b44d5-1629-4e9b-b44e-24096d8b3959 encap : [] external_ids : {"neutron:cidrs"neutron:cidrsneutron:cidrsneutron:cidrs="2a02:ed80:0:3::341/64 91.198.2.33/24http://91.198.2.33/24", "neutron:device_id"neutron:device_idneutron:device_idneutron:device_id="8062ec61-0c68-41dd-b77c-e8b72ad16a88", "neutron:device_owner"neutron:device_ownerneutron:device_ownerneutron:device_owner="compute:AZ1"compute:AZ1compute:AZ1compute:AZ1, "neutron:network_name"neutron:network_nameneutron:network_nameneutron:network_name=neutron-210e26d7-942f-4e17-89b2-571eee87d7e4, "neutron:port_name"neutron:port_nameneutron:port_nameneutron:port_name="", "neutron:project_id"neutron:project_idneutron:project_idneutron:project_id="99fb21796a8f4cbda42ba5b9d1e307dd", "neutron:revision_number"neutron:revision_numberneutron:revision_numberneutron:revision_number="16", "neutron:security_group_ids"neutron:security_group_idsneutron:security_group_idsneutron:security_group_ids="3e41777f-7aa4-4368-9992-5ca7cc2a5372 873b3b62-0918-4b1e-be73-fdbed50d2ac2"} gateway_chassis : [] ha_chassis_group : [] logical_port : "cfce175b-9d88-4c2e-a5cc-d76cd5c71deb" mac : ["fa:16:3e:a2:d7:1a 2a02:ed80:0:3::341 91.198.2.33"fa:16:3e:a2:d7:1a2a02:ed80:0:3::34191.198.2.33fa:16:3e:a2:d7:1a2a02:ed80:0:3::34191.198.2.33fa:16:3e:a2:d7:1a2a02:ed80:0:3::34191.198.2.33] nat_addresses : [] options : {mcast_flood_reports="true", requested-chassis=net-openstack-hv31} parent_port : [] requested_chassis : c944c21a-3344-4fda-ab4e-a4cc07403125 tag : [] tunnel_key : 344 type : "" up : true virtual_parent : []
The LS:
$ ovn-nbctl list Logical_Switch public-network _uuid : 56d8be55-462a-4b93-8710-3c79ca386213 acls : [] copp : [] dns_records : [] external_ids : {"neutron:mtu"neutron:mtuneutron:mtuneutron:mtu="1500", "neutron:network_name"neutron:network_nameneutron:network_nameneutron:network_name=public-network, "neutron:revision_number"neutron:revision_numberneutron:revision_numberneutron:revision_number="21"} forwarding_groups : [] load_balancer : [] load_balancer_group : [] name : neutron-210e26d7-942f-4e17-89b2-571eee87d7e4 other_config : {mcast_flood_unregistered="false", mcast_snoop="false"} ports : [00225774-8fbc-473f-ae5e-d486c54212c8, ..., c5dfb248-941e-4d4e-af1a-9ccafc22db70, ... qos_rules : []
The patchport:
$ ovn-nbctl list Logical_Switch_Port provnet-aa35051c-6fc0-463a-8807-0cb28903be14 _uuid : f7259aeb-0e63-4d20-8a8e-54ebf454a524 addresses : [unknown] dhcpv4_options : [] dhcpv6_options : [] dynamic_addresses : [] enabled : [] external_ids : {} ha_chassis_group : [] name : provnet-aa35051c-6fc0-463a-8807-0cb28903be14 options : {mcast_flood="false", mcast_flood_reports="true", network_name=physnet1} parent_name : [] port_security : [] tag : [] tag_request : [] type : localnet up : false
I hope I provided the needed context! Thanks in advance!
Best regards, Justin Lamp
-- Justin Lamp Systems Engineer
NETWAYS Managed Services GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg Tel: +49 911 92885-0 | Fax: +49 911 92885-77 CEO: Julian Hein, Bernd Erk, Sebastian Saemann | AG Nuernberg HRB25207 https://www.netways.de | justin.lamp@netways.demailto:justin.lamp@netways.de
** stackconf 2023 - September - https://stackconf.eu ** ** OSMC 2023 - November - https://osmc.de ** ** NETWAYS Web Services - https://nws.netways.de ** ** NETWAYS Trainings - https://netways.de/trainings **
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail.
Hinweise zum Datenschutz finden Sie hierhttps://www.datenschutz.schwarz.
This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail.
Information on data protection can be found herehttps://www.datenschutz.schwarz.
-- Justin Lamp Systems Engineer
NETWAYS Managed Services GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg Tel: +49 911 92885-0 | Fax: +49 911 92885-77 CEO: Julian Hein, Bernd Erk, Sebastian Saemann | AG Nuernberg HRB25207 https://www.netways.de | justin.lamp@netways.demailto:justin.lamp@netways.de
** stackconf 2023 - September - https://stackconf.eu ** ** OSMC 2023 - November - https://osmc.de ** ** NETWAYS Web Services - https://nws.netways.de ** ** NETWAYS Trainings - https://netways.de/trainings **
-- Justin Lamp Systems Engineer
NETWAYS Managed Services GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg Tel: +49 911 92885-0 | Fax: +49 911 92885-77 CEO: Julian Hein, Bernd Erk, Sebastian Saemann | AG Nuernberg HRB25207 https://www.netways.de | justin.lamp@netways.de
** stackconf 2023 - September - https://stackconf.eu ** ** OSMC 2023 - November - https://osmc.de ** ** NETWAYS Web Services - https://nws.netways.de ** ** NETWAYS Trainings - https://netways.de/trainings **
participants (4)
-
Felix Huettner
-
Ihar Hrachyshka
-
Justin Lamp
-
Satish Patel