<div dir="ltr"><div>Hi Gary,</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Em ter., 27 de jun. de 2023 às 11:47, Yatin Karel <<a href="mailto:ykarel@redhat.com">ykarel@redhat.com</a>> escreveu:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>Hi Gary,</div><div><br></div><div>On top what Rodolfo said<br></div></div>On Tue, Jun 27, 2023 at 5:15 PM Gary Molenkamp <<a href="mailto:molenkam@uwo.ca" target="_blank">molenkam@uwo.ca</a>> wrote:<div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Good morning, I'm having a problem with snat routing under OVN but I'm <br>
not sure if something is mis-configured or just my understanding of how <br>
OVN is architected is wrong.<br>
<br>
I've built a Zed cloud, since upgraded to Antelope, using the Neutron <br>
Manual install method here: <br>
<a href="https://docs.openstack.org/neutron/latest/install/ovn/manual_install.html" rel="noreferrer" target="_blank">https://docs.openstack.org/neutron/latest/install/ovn/manual_install.html</a><br>
I'm using a multi-tenent configuration using geneve and the flat <br>
provider network is present on each hypervisor. Each hypervisor is <br>
connected to the physical provider network, along with the tenent <br>
network and is tagged as an external chassis under OVN.<br>
br-int exists, as does br-provider<br>
ovs-vsctl set open . <br>
external-ids:ovn-cms-options=enable-chassis-as-gw<br></blockquote><div><br></div><div>Any specific reason to enable gateway on compute nodes? Generally it's recommended to use controller/network nodes as gateway. What's your env(number of controllers, network, compute nodes)?<br></div></div></div></blockquote><div><br></div><div>Wouldn't it be interesting to enable-chassis-as-gw on the compute nodes, just in case you want to use DVR: If that's the case, you need to map the external bridge (<span style="background-color:rgb(244,245,247);color:rgb(23,43,77);font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Consolas,Courier,monospace;font-size:14px;white-space:pre">ovs-vsctl set open . external-ids:ovn-bridge-mappings=...</span>) via ansible this is created automatically, but in the manual installation I didn't see any mention of it.</div><div> </div><div>The problem is basically that the port of the OVN LRP may not be in the same chassis as the VM that failed (since the CR-LRP will be where the first VM of that network will be created). The suggestion is to remove the enable-chassis-as-gw from the compute nodes to allow the VM to forward traffic via tunneling/Geneve to the chassis where the LRP resides.<br></div><div><br></div><div><span style="color:rgb(23,43,77);font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Consolas,Courier,monospace;font-size:14px;white-space:pre;background-color:rgb(244,245,247)">ovs-vsctl remove open . external-ids ovn-cms-options="enable-chassis-as-gw"
</span><span class="gmail-comment gmail-linenumber gmail-ds-line-number" style="box-sizing:border-box;padding-left:8px;margin-right:8px;text-align:right;float:left;font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Consolas,Courier,monospace;font-size:14px;white-space:pre;background-color:rgb(244,245,247);display:inline-block;padding-right:8px"></span><span style="color:rgb(23,43,77);font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Consolas,Courier,monospace;font-size:14px;white-space:pre;background-color:rgb(244,245,247)">ovs-vsctl remove open . external-ids ovn-bridge-mappings
</span><span class="gmail-comment gmail-linenumber gmail-ds-line-number" style="box-sizing:border-box;padding-left:8px;margin-right:8px;text-align:right;float:left;font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Consolas,Courier,monospace;font-size:14px;white-space:pre;background-color:rgb(244,245,247);display:inline-block;padding-right:8px"></span><span style="color:rgb(23,43,77);font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Consolas,Courier,monospace;font-size:14px;white-space:pre;background-color:rgb(244,245,247)">ip link set br-provider-name down
</span><span class="gmail-comment gmail-linenumber gmail-ds-line-number" style="box-sizing:border-box;padding-left:8px;margin-right:8px;text-align:right;float:left;font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Consolas,Courier,monospace;font-size:14px;white-space:pre;background-color:rgb(244,245,247);display:inline-block;padding-right:8px"></span><span style="color:rgb(23,43,77);font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Consolas,Courier,monospace;font-size:14px;white-space:pre;background-color:rgb(244,245,247)">ovs-vsctl del-br </span><span style="color:rgb(23,43,77);font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Consolas,Courier,monospace;font-size:14px;white-space:pre;background-color:rgb(244,245,247)">br-provider-name</span><span style="color:rgb(23,43,77);font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Consolas,Courier,monospace;font-size:14px;white-space:pre;background-color:rgb(244,245,247)">
</span><span class="gmail-comment gmail-linenumber gmail-ds-line-number" style="box-sizing:border-box;padding-left:8px;margin-right:8px;text-align:right;float:left;font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Consolas,Courier,monospace;font-size:14px;white-space:pre;background-color:rgb(244,245,247);display:inline-block;padding-right:8px"></span><span style="color:rgb(23,43,77);font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Consolas,Courier,monospace;font-size:14px;white-space:pre;background-color:rgb(244,245,247)">systemctl restart ovn-controller
</span><span class="gmail-comment gmail-linenumber gmail-ds-line-number" style="box-sizing:border-box;padding-left:8px;margin-right:8px;text-align:right;float:left;font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Consolas,Courier,monospace;font-size:14px;white-space:pre;background-color:rgb(244,245,247);display:inline-block;padding-right:8px"></span><span style="color:rgb(23,43,77);font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Consolas,Courier,monospace;font-size:14px;white-space:pre;background-color:rgb(244,245,247)">systemctl restart openvswitch-switch
</span><span class="gmail-comment gmail-linenumber gmail-ds-line-number" style="box-sizing:border-box;padding-left:8px;margin-right:8px;text-align:right;float:left;font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Consolas,Courier,monospace;font-size:14px;white-space:pre;background-color:rgb(244,245,247);display:inline-block;padding-right:8px"></span></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
For most cases, distributed FIP based connectivity is working without <br>
issue, but I'm having an issue where VMs without a FIP are not always <br>
able to use the SNAT services of the tenent network router.<br>
Scenario:<br>
Internal network named cs3319: with subnet <a href="http://172.31.100.0/23" rel="noreferrer" target="_blank">172.31.100.0/23</a><br>
Has a router named cs3319_router with external gateway set (snat <br>
enabled)<br>
<br>
This network has 3 vms:<br>
- #1 has a FIP and can be accessed externally<br>
- #2 has no FIP, can be accessed via VM1 and can access <br>
external resources via SNAT (ie OS repos, DNS, etc)<br>
- #3 has no FIP, can be accessed via VM1 but has no external <br>
SNAT connectivity<br>
<br></blockquote><div>Considering it works for some vm but for some not, the above point for enable-chassis-as-gw could be related.<br></div><div>The working vm is hosted on compute05 or some other compute node? Where is the gateway router port scheduled(can check ovn-sbctl show for cr-lrp-<router gateway port id>)?<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
From what I can tell, the chassis config is correct, compute05 is the <br>
hypervisor and the faulty VM has a port binding on this hypervisor:<br>
<br>
ovn-sbctl show<br>
...<br>
Chassis "8e0fa17c-e480-4b60-9015-bd8833412561"<br>
hostname: <a href="http://compute05.cloud.sci.uwo.ca" rel="noreferrer" target="_blank">compute05.cloud.sci.uwo.ca</a><br>
Encap geneve<br>
ip: "192.168.0.105"<br>
options: {csum="true"}<br>
Port_Binding "7a5257eb-caea-45bf-b48c-620c5dff4b39"<br>
Port_Binding "50e16602-78e6-429b-8c2f-e7e838ece1b4"<br>
Port_Binding "f121c9f4-c3fe-4ea9-b754-a809be95a3fd"<br>
<br>
The router has the candidate gateways, and the snat set:<br>
<br>
ovn-nbctl show 92df19a7-4ebe-43ea-b233-f4e9f5a46e7c<br>
router 92df19a7-4ebe-43ea-b233-f4e9f5a46e7c <br>
(neutron-389439b5-07f8-44b6-a35b-c76651b48be5) (aka cs3319_public_router)<br>
port lrp-44ae1753-845e-4822-9e3d-a41e0469e257<br>
mac: "fa:16:3e:9a:db:d8"<br>
networks: ["<a href="http://129.100.21.94/22" rel="noreferrer" target="_blank">129.100.21.94/22</a>"]<br>
gateway chassis: [5c039d38-70b2-4ee6-9df1-596f82c68106 <br>
99facd23-ad17-4b68-a8c2-1ff6da15ac5f <br>
1694116c-6d30-4c31-b5ea-0f411878316e <br>
2a4bbaf9-228a-462e-8970-0cdbf59086e6 9332c61b-93e1-4a70-9547-701a014bfd98]<br>
port lrp-509bba37-fa06-42d6-9210-2342045490db<br>
mac: "fa:16:3e:ff:0f:3b"<br>
networks: ["<a href="http://172.31.100.1/23" rel="noreferrer" target="_blank">172.31.100.1/23</a>"]<br>
nat 11e0565a-4695-4f67-b4ee-101f1b1b9a4f<br>
external ip: "129.100.21.94"<br>
logical ip: "<a href="http://172.31.100.0/23" rel="noreferrer" target="_blank">172.31.100.0/23</a>"<br>
type: "snat"<br>
nat 21e4be02-d81c-46e8-8fa8-3f94edb4aed1<br>
external ip: "129.100.21.87"<br>
logical ip: "172.31.100.49"<br>
type: "dnat_and_snat"<br>
<br>
Each network agent on the hypervisors shows the ovn controller up :<br>
OVN Controller Gateway agent | <a href="http://compute05.cloud.sci.uwo.ca" rel="noreferrer" target="_blank">compute05.cloud.sci.uwo.ca</a> <br>
| | :-) | UP | ovn-controller<br>
<br>
The ovs vswitch on the hypervisor looks correct afaict and ovn ports bfd <br>
status are all forwarding to other hypervisors. ie:<br>
Port ovn-2a4bba-0<br>
Interface ovn-2a4bba-0<br>
type: geneve<br>
options: {csum="true", key=flow, remote_ip="192.168.0.106"}<br>
bfd_status: {diagnostic="No Diagnostic", <br>
flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic", <br>
remote_state=up, state=up}<br>
<br>
<br>
Any advice on where to look would be appreciated.<br>
<br></blockquote><div>I have seen mtu specific issues in the past, would be good to rule out any mtu issue with working and non working cases.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
PS. Version info:<br>
Neutron 22.0.0-1<br>
OVN 22.12<br>
<br>
neutron options:<br>
enable_distributed_floating_ip = true<br>
ovn_l3_scheduler = leastloaded<br>
<br>
<br>
<br>
Thanks<br>
Gary<br>
<br>
<br>
<br>
-- <br>
Gary Molenkamp Science Technology Services<br>
Systems/Cloud Administrator University of Western Ontario<br>
<a href="mailto:molenkam@uwo.ca" target="_blank">molenkam@uwo.ca</a> <a href="http://sts.sci.uwo.ca" rel="noreferrer" target="_blank">http://sts.sci.uwo.ca</a><br>
(519) 661-2111 x86882 (519) 661-3566<br>
<br>
<br></blockquote><div>Thanks and Regards</div><div>Yatin Karel <br></div></div>
</div>
</blockquote></div></div>
<br>
<div style="font-family:Arial,Helvetica,sans-serif;font-size:1.3em"><div style="color:rgb(97,97,97);font-family:'Open Sans';font-size:14px;line-height:21px;background-color:rgb(255,255,255)"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8000001907349px;line-height:normal"><div style="font-family:Arial,Helvetica,sans-serif;font-size:1.3em"><div style="color:rgb(97,97,97);font-family:'Open Sans';font-size:14px;line-height:21px"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8000001907349px;line-height:normal"><br></div></div></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:1.3em"><i style="font-family:arial,sans-serif;font-size:x-small">‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.</i></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:1.3em"><p style="font-family:arial,sans-serif;text-align:justify"><i><font size="1"> </font></i><i><font size="1">‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.</font></i></p></div></div></div></div>