<div dir="ltr"><div>Hello Harshit:</div><div><br></div><div>You need first to fix your underlying network configuration. If the interfaces created on "sriovnet1" can't talk to each other this is because your network is blocking this traffic. Check your TOR switches to enable/allow this VLAN range.</div><div><br></div><div>Regards.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 23, 2022 at 9:42 AM Harshit Mishra <<a href="mailto:harshit@voereir.com">harshit@voereir.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hello Rodolfo,<br>
<br>
Thanks for your support.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<blockquote style="border-color:rgb(200,200,200);border-left:3px solid rgb(200,200,200);padding-left:1ex;margin-left:0.8ex;color:rgb(102,102,102)">
<div><span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline"><sub><span style="font-size:15px;background-color:rgb(255,255,255);display:inline">If
the VMs are spawned without any issue, that could be a problem in the underlying network deployment. When sending traffic from a VM with an SR-IOV port, try to capture the traffic in the physical function. Please check the physical network connected to the
SR-IOV network cards allow VLAN traffic in 1000:1009 (according to your configuration).</span><br>
</sub></span></div>
</blockquote>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Network I have configured of VLAN type in a given range.<br>
<blockquote style="border-color:rgb(200,200,200);border-left:3px solid rgb(200,200,200);padding-left:1ex;margin-left:0.8ex;color:rgb(102,102,102)">
<div>openstack network show 4cacd198-29c3-47b3-bf77-39f1d01c9a22 -c provider:network_type -c provider:physical_network -c provider:segmentation_id
<div>+---------------------------+-----------+</div>
<div>| Field | Value |</div>
<div>+---------------------------+-----------+</div>
<div>| provider:network_type | vlan |</div>
<div>| provider:physical_network | sriovnet1 |</div>
<div>| provider:segmentation_id | 1002 |</div>
+---------------------------+-----------+<br>
</div>
</blockquote>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
Note: I forget to add that Intra-communication between the VM's working fine deployed on same HOST but inter-communication not working. </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
One more thing IP address is not assigning to interfaces of VM. We have to configure static IP address manually.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
Please find below incoming traffic flow on PF. <br>
<br>
<br>
<blockquote style="border-color:rgb(200,200,200);border-left:3px solid rgb(200,200,200);padding-left:1ex;margin-left:0.8ex;color:rgb(102,102,102)">
# tcpdump -i enp8s0f1
<div>tcpdump: verbose output suppressed, use -v or -vv for full protocol decode</div>
<div>listening on enp8s0f1, link-type EN10MB (Ethernet), capture size 262144 bytes</div>
<div>12:39:28.584000 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:88:f5:3e (oui Unknown), length 285</div>
<div>12:39:39.183491 IP6 fe80::a236:9fff:fe24:803a > ip6-allrouters: ICMP6, router solicitation, length 16</div>
<div>12:39:49.660269 IP6 fe80::a236:9fff:fe53:e598.5240 > ff02::15a.5240: UDP, length 221</div>
<div>12:39:49.660897 IP6 fe80::a236:9fff:fe53:e59a.5240 > ff02::15a.5240: UDP, length 221</div>
<div>12:39:54.105381 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from a0:36:9f:d8:0d:a8 (oui Unknown), length 293</div>
<div>12:39:54.897051 LLDP, length 46</div>
<div>12:39:56.281671 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 3c:fd:fe:cd:06:71 (oui Unknown), length 293</div>
<div>12:40:06.286310 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from a0:36:9f:d8:0d:aa (oui Unknown), length 293</div>
<div>12:40:15.039416 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 3c:fd:fe:cd:06:70 (oui Unknown), length 293</div>
<div>12:40:19.675902 IP6 fe80::a236:9fff:fe53:e598.5240 > ff02::15a.5240: UDP, length 221</div>
<div>12:40:19.676390 IP6 fe80::a236:9fff:fe53:e59a.5240 > ff02::15a.5240: UDP, length 221</div>
<div>12:40:25.708972 LLDP, length 46</div>
<div>12:40:29.288242 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:c4:3f:4f (oui Unknown), length 285</div>
12:40:33.008653 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:88:f5:3e (oui Unknown), length 285 <br>
</blockquote>
<div><span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"></span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<blockquote style="border-color:rgb(200,200,200);border-left:3px solid rgb(200,200,200);padding-left:1ex;margin-left:0.8ex;color:rgb(102,102,102)">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline">Another question:
do the VMs with SR-IOV port receive the DHCP reply? If so, at least you know the compute to controller communication is working (assuming the DHCP agent is on the controller).</span><br>
</div>
</blockquote>
No, VMs not receiving DHCP reply, you can refer above logs. Yes, DHCP agent is on controller.<br>
<br>
Kindly let me know anything else is required.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
Thanks!</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Regards,<br>
Harshit Mishra <br>
<br>
</div>
<div id="gmail-m_1389149178264763167appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_1389149178264763167divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Rodolfo Alonso Hernandez <<a href="mailto:ralonsoh@redhat.com" target="_blank">ralonsoh@redhat.com</a>><br>
<b>Sent:</b> Wednesday, June 22, 2022 6:46 PM<br>
<b>To:</b> Harshit Mishra <<a href="mailto:harshit@voereir.com" target="_blank">harshit@voereir.com</a>><br>
<b>Subject:</b> Re: [kolla-ansible][xena] Facing network issue when providing SR-IOV support</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>Hello Harshit:</div>
<div><br>
</div>
<div>If the VMs are spawned without any issue, that could be a problem in the underlying network deployment. When sending traffic from a VM with an SR-IOV port, try to capture the traffic in the physical function. Please check the physical network connected
to the SR-IOV network cards allow VLAN traffic in 1000:1009 (according to your configuration).</div>
<div><br>
</div>
<div>Another question: do the VMs with SR-IOV port receive the DHCP reply? If so, at least you know the compute to controller communication is working (assuming the DHCP agent is on the controller).</div>
<div><br>
</div>
<div>Regards.<br>
</div>
</div>
<br>
<div>
<div dir="ltr">On Wed, Jun 22, 2022 at 2:17 PM Harshit Mishra <<a href="mailto:harshit@voereir.com" target="_blank">harshit@voereir.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">Hi!<br>
</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"><br>
I have deployed Openstack Xena using kolla-ansible (v13.0.1) on a multi-node setup (1 controller+network, multiple computes).</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"> </span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">On this cluster I would like to have support for normal, as well as direct (SR-IOV) vNIC types.</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"> </span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">I have done all the pre-requirement like configure VF on SR-IOV network interface.</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"> </span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">In the current deployment, I have created two physnets, one for flat network (called physnet1 on br-ex), and one for SR-IOV (called sriovnet1 on Intel 10G 2P X520 card). I am creating one network of VXLAN
type for normal vNIC, and one of VLAN type on sriovnet1 (called sriov_network) for direct vNIC.</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"> </span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">Mapping of PF with provider in sriov-agent.ini:</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"> </span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"># cat /etc/kolla/neutron-sriov-agent/sriov_agent.ini</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">[sriov_nic]</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">physical_device_mappings = sriovnet1:enp8s0f1</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">exclude_devices =</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"> </span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">[securitygroup]</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">firewall_driver = neutron.agent.firewall.NoopFirewallDriver</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"> </span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">----------------------------------------------</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">ml2_conf.ini on controller node:</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"> </span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"># cat /etc/kolla/neutron-server/ml2_conf.ini</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">[ml2]</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">type_drivers = flat,vlan,vxlan</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">tenant_network_types = vxlan,vlan,flat</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">mechanism_drivers = openvswitch,l2population,sriovnicswitch</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">extension_drivers = port_security</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"> </span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">[ml2_type_vlan]</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">network_vlan_ranges = physnet1,sriovnet1:1000:1009</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"> </span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">[ml2_type_flat]</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">flat_networks = physnet1,sriovnet1</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"> </span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">[ml2_sriov]</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">agent_required = False</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">supported_pci_vendor_devs = 8086:10ed</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"> </span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">[ml2_type_vxlan]</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">vni_ranges = 1:1000</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"> </span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">----------------------------------------------</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"> </span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"># grep -nr enabled_filters /etc/kolla/nova-scheduler/</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">/etc/kolla/nova-scheduler/nova.conf:13:enabled_filters = ComputeFilter,AvailabilityZoneFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,SameHostFilter,DifferentHostFilter,AggregateInstanceExtraSpecsFilter,PciPassthroughFilter</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"> </span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">----------------------------------------------</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"> </span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"># grep passthrough /etc/kolla/nova-compute/nova.conf</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">passthrough_whitelist = [{"physical_network": "sriovnet1", "devname": "enp8s0f1"}]</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"> </span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"> </span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">The interfaces on VMs from normal vNICs are able to communicate inter-compute. The same is not working for the direct vNICs belonging to the sriov_network. I have tried multiple config changes, but none
seem to work.</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black"> </span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:12pt;color:black">Please help in solving this issue, or suggest a different way of achieving this if this is wrong or not optimal.</span></p>
<p style="color:rgb(32,31,30);background-color:rgb(255,255,255);font-size:10pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:11pt"> </span></p>
<div style="margin:0px;font-size:15px;color:rgb(32,31,30);background-color:rgb(255,255,255)">
<div style="margin:0px">
<div style="margin:0px">
<p style="font-size:10pt;font-family:Calibri,sans-serif;margin:0px"><span style="margin:0px;font-size:11pt">Thanks!</span></p>
</div>
</div>
</div>
<br>
Regards,
<div>Harshit Mishra</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote></div>