[nova] failing OVS offload on mellanox connectx-4 (RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7)

5 Mar 2019

      Dear Openstack community,

first let me apologise for my ignorance in this topic, this is the first time I try to setup an environment like this and I don't know anyone who has done it before.

I am trying to setup OVS offload following this document https://docs.openstack.org/neutron/rocky/admin/config-ovs-offload.html

My Openstack is Rocky deployed through kolla-ansible.

I have a feeling my OVS offload is not setup properly as vm creation fails and the error "RepresentorNotFound: Failed getting representor port for PF" can be see in nova-compute logs.

# Nova compute logs:

2019-03-05 11:24:46.392 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Attempting claim on node zeus-54.localdomain: memory 8192 MB, disk 80 GB, vcpus 4 CPU
2019-03-05 11:24:46.393 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Total memory: 524173 MB, used: 8704.00 MB
2019-03-05 11:24:46.394 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] memory limit not specified, defaulting to unlimited
2019-03-05 11:24:46.394 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Total disk: 18630 GB, used: 80.00 GB
2019-03-05 11:24:46.395 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] disk limit not specified, defaulting to unlimited
2019-03-05 11:24:46.396 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Total vcpu: 56 VCPU, used: 4.00 VCPU
2019-03-05 11:24:46.396 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] vcpu limit not specified, defaulting to unlimited
2019-03-05 11:24:46.408 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Claim successful on node zeus-54.localdomain
2019-03-05 11:24:47.144 7 INFO nova.virt.libvirt.driver [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Creating image
2019-03-05 11:24:48.957 7 ERROR os_vif [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] Failed to plug vif VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True): RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7
2019-03-05 11:24:48.957 7 ERROR os_vif Traceback (most recent call last):
2019-03-05 11:24:48.957 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/os_vif/__init__.py", line 77, in plug
2019-03-05 11:24:48.957 7 ERROR os_vif     plugin.plug(vif, instance_info)
2019-03-05 11:24:48.957 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 215, in plug
2019-03-05 11:24:48.957 7 ERROR os_vif     self._plug_vf_passthrough(vif, instance_info)
2019-03-05 11:24:48.957 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 189, in _plug_vf_passthrough
2019-03-05 11:24:48.957 7 ERROR os_vif     representor = linux_net.get_representor_port(pf_ifname, vf_num)
2019-03-05 11:24:48.957 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py", line 381, in get_representor_port
2019-03-05 11:24:48.957 7 ERROR os_vif     raise exception.RepresentorNotFound(ifname=pf_ifname, vf_num=vf_num)
2019-03-05 11:24:48.957 7 ERROR os_vif RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7
2019-03-05 11:24:48.957 7 ERROR os_vif
2019-03-05 11:24:48.960 7 ERROR nova.virt.libvirt.driver [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Failed to start libvirt guest: InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: Failed getting representor port for PF ens2f1 with 7
2019-03-05 11:24:48.969 7 ERROR os_vif [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] Failed to unplug vif VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True): RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7
2019-03-05 11:24:48.969 7 ERROR os_vif Traceback (most recent call last):
2019-03-05 11:24:48.969 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/os_vif/__init__.py", line 110, in unplug
2019-03-05 11:24:48.969 7 ERROR os_vif     plugin.unplug(vif, instance_info)
2019-03-05 11:24:48.969 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 277, in unplug
2019-03-05 11:24:48.969 7 ERROR os_vif     self._unplug_vf_passthrough(vif, instance_info)
2019-03-05 11:24:48.969 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 250, in _unplug_vf_passthrough
2019-03-05 11:24:48.969 7 ERROR os_vif     representor = linux_net.get_representor_port(pf_ifname, vf_num)
2019-03-05 11:24:48.969 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py", line 381, in get_representor_port
2019-03-05 11:24:48.969 7 ERROR os_vif     raise exception.RepresentorNotFound(ifname=pf_ifname, vf_num=vf_num)
2019-03-05 11:24:48.969 7 ERROR os_vif RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7
2019-03-05 11:24:48.969 7 ERROR os_vif
2019-03-05 11:24:48.971 7 INFO nova.virt.libvirt.driver [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Deleting instance files /var/lib/nova/instances/1b03f2e7-0c44-46c2-87f5-d68f93654c58_del
2019-03-05 11:24:48.972 7 INFO nova.virt.libvirt.driver [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Deletion of /var/lib/nova/instances/1b03f2e7-0c44-46c2-87f5-d68f93654c58_del complete
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Instance failed to spawn: InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: Failed getting representor port for PF ens2f1 with 7
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Traceback (most recent call last):
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2368, in _build_resources
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]     yield resources
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2132, in _build_and_run_instance
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]     block_device_info=block_device_info)
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3080, in spawn
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]     destroy_disks_on_failure=True)
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5646, in _create_domain_and_network
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]     destroy_disks_on_failure)
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]     self.force_reraise()
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]     six.reraise(self.type_, self.value, self.tb)
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5605, in _create_domain_and_network
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]     self.plug_vifs(instance, network_info)
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 819, in plug_vifs
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]     self.vif_driver.plug(instance, vif)
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 807, in plug
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]     self._plug_os_vif(instance, vif_obj)
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 787, in _plug_os_vif
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]     raise exception.InternalError(msg)
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: Failed getting representor port for PF ens2f1 with 7
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]
2019-03-05 11:24:49.085 7 INFO nova.compute.manager [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Terminating instance
2019-03-05 11:24:49.091 7 INFO nova.virt.libvirt.driver [-] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Instance destroyed successfully.
2019-03-05 11:24:49.099 7 ERROR os_vif [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] Failed to unplug vif VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True): RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7
2019-03-05 11:24:49.099 7 ERROR os_vif Traceback (most recent call last):
2019-03-05 11:24:49.099 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/os_vif/__init__.py", line 110, in unplug
2019-03-05 11:24:49.099 7 ERROR os_vif     plugin.unplug(vif, instance_info)
2019-03-05 11:24:49.099 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 277, in unplug
2019-03-05 11:24:49.099 7 ERROR os_vif     self._unplug_vf_passthrough(vif, instance_info)
2019-03-05 11:24:49.099 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 250, in _unplug_vf_passthrough
2019-03-05 11:24:49.099 7 ERROR os_vif     representor = linux_net.get_representor_port(pf_ifname, vf_num)
2019-03-05 11:24:49.099 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py", line 381, in get_representor_port
2019-03-05 11:24:49.099 7 ERROR os_vif     raise exception.RepresentorNotFound(ifname=pf_ifname, vf_num=vf_num)
2019-03-05 11:24:49.099 7 ERROR os_vif RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7
2019-03-05 11:24:49.099 7 ERROR os_vif
2019-03-05 11:24:49.101 7 INFO nova.virt.libvirt.driver [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Deletion of /var/lib/nova/instances/1b03f2e7-0c44-46c2-87f5-d68f93654c58_del complete
2019-03-05 11:24:49.198 7 INFO nova.compute.manager [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Took 0.11 seconds to destroy the instance on the hypervisor.
2019-03-05 11:24:50.384 7 INFO nova.compute.manager [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Took 0.77 seconds to deallocate network for instance.
2019-03-05 11:24:50.624 7 INFO nova.scheduler.client.report [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] Deleted allocation for instance 1b03f2e7-0c44-46c2-87f5-d68f93654c58
2019-03-05 11:25:07.884 7 INFO nova.compute.resource_tracker [req-1b32a794-0bf0-4abe-a024-699920f791d2 - - - - -] Final resource view: name=zeus-54.localdomain phys_ram=524173MB used_ram=8704MB phys_disk=18630GB used_disk=80GB total_vcpus=56 used_vcpus=4 pci_stats=[PciDevicePool(count=15,numa_node=1,product_id='1016',tags={dev_type='type-VF',physical_network='physnet1'},vendor_id='15b3')]

### This is my environment/configuration

# OVS version:
docker exec -it openvswitch_vswitchd ovs-vswitchd --version
ovs-vswitchd (Open vSwitch) 2.10.1

# Mellanox firmware version:
[root@zeus-54 ~]# ibv_devinfo
hca_id: mlx5_9
        transport:                      InfiniBand (0)
        fw_ver:                         14.24.1000

# Kernel version:
[root@zeus-54 ~]# uname -r
3.10.0-862.el7.x86_64

# OS distro:
[root@zeus-54 ~]# cat /etc/*release
CentOS Linux release 7.6.1810 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.6.1810 (Core)
CentOS Linux release 7.6.1810 (Core)

# SR-IOV devices:
[root@zeus-54 ~]# lspci -D -nn | grep -i mel
0000:88:00.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015]
0000:88:00.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015]
0000:88:00.2 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:00.3 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:00.4 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:00.5 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:00.6 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:00.7 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:01.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:01.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:01.2 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:01.3 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:01.4 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:01.5 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:01.6 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:01.7 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:02.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:02.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

[root@zeus-54 ~]# ibdev2netdev -v
0000:88:00.0 mlx5_0 (MT4117 - MT1611X09936) CX4121A - ConnectX-4 LX SFP28 fw 14.24.1000 port 1 (ACTIVE) ==> ens2f0 (Up)
0000:88:00.1 mlx5_1 (MT4117 - MT1611X09936) CX4121A - ConnectX-4 LX SFP28 fw 14.24.1000 port 1 (ACTIVE) ==> ens2f1 (Up)
0000:88:01.2 mlx5_10 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f2 (Up)
0000:88:01.3 mlx5_11 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f3 (Up)
0000:88:01.4 mlx5_12 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f4 (Up)
0000:88:01.5 mlx5_13 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f5 (Up)
0000:88:01.6 mlx5_14 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f6 (Up)
0000:88:01.7 mlx5_15 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f7 (Up)
0000:88:02.0 mlx5_16 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> enp136s2 (Up)
0000:88:02.1 mlx5_17 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> enp136s2f1 (Up)
0000:88:00.2 mlx5_2 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> ens2f2 (Up)
0000:88:00.3 mlx5_3 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> ens2f3 (Up)
0000:88:00.4 mlx5_4 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> ens2f4 (Up)
0000:88:00.5 mlx5_5 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> ens2f5 (Up)
0000:88:00.6 mlx5_6 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> ens2f6 (Up)
0000:88:00.7 mlx5_7 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> ens2f7 (Up)
0000:88:01.0 mlx5_8 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1 (Up)
0000:88:01.1 mlx5_9 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f1 (Up)

# Embeded switch configuration:
[root@zeus-54 ~]# devlink dev eswitch show pci/0000:88:00.0
pci/0000:88:00.0: mode legacy inline-mode none encap enable
[root@zeus-54 ~]# devlink dev eswitch show pci/0000:88:00.1
pci/0000:88:00.1: mode legacy inline-mode none encap enable

[root@zeus-54 ~]# sudo ethtool -k ens2f0 | grep hw-tc-offload
hw-tc-offload: on
[root@zeus-54 ~]# sudo ethtool -k ens2f1 | grep hw-tc-offload
hw-tc-offload: on

[root@zeus-54 ~]# docker exec -it openvswitch_vswitchd sudo ovs-vsctl get Open_vSwitch . other_config
{hw-offload="true", max-idle="30000"}
[root@zeus-54 ~]# docker exec -it openvswitch_vswitchd sudo ovs-vsctl get Open_vSwitch . other_config
{hw-offload="true", max-idle="30000"}

# Check ovs offload
[root@zeus-54 ~]# docker exec -it openvswitch_vswitchd sudo ovs-dpctl show
2019-03-05T00:50:42Z|00001|dpif_netlink|INFO|The kernel module does not support meters.
system@ovs-system:
  lookups: hit:0 missed:0 lost:0
  flows: 0
  masks: hit:0 total:0 hit/pkt:0.00
  port 0: ovs-system (internal)
  port 1: br-ex (internal)
  port 2: br-int (internal)
  port 3: br-tun (internal)

QUESTIONS:
Is my kernel version right?
Am I supposed to see the PFs attached to a port in the ovs-dpctl show output?
What else can I check?

Thank you very much

NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.

Manuel Sopena Ballesteros

Moshe Levi

Hamdy Khader

Waleed Musa

Manuel Sopena Ballesteros

Moshe Levi

tags

participants (4)