Hi Manuel, I see form the configuration below that the mode is still legacy (so for sure the representors will not be created) # Embeded switch configuration: [root@zeus-54 ~]# devlink dev eswitch show pci/0000:88:00.0 pci/0000:88:00.0: mode legacy inline-mode none encap enable [root@zeus-54 ~]# devlink dev eswitch show pci/0000:88:00.1 pci/0000:88:00.1: mode legacy inline-mode none encap enable Are you sure you configured it correctly? https://docs.openstack.org/neutron/rocky/admin/config-ovs-offload.html#confi... Please take care that you need to unbind all the vfs before changing to switchdev mode Regards Waleed Mousa SW Engineer at Mellanox ________________________________ From: Hamdy Khader Sent: 06 March 2019 09:25:39 To: Manuel Sopena Ballesteros; openstack@lists.openstack.org; Moshe Levi; Waleed Musa Subject: Re: [nova] failing OVS offload on mellanox connectx-4 (RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7) Hi, @Waleed can you help configure offloading for CX4 ? Regards, Hamdy ________________________________ From: Manuel Sopena Ballesteros <manuel.sb@garvan.org.au> Sent: Tuesday, March 5, 2019 7:52 AM To: openstack@lists.openstack.org Subject: [nova] failing OVS offload on mellanox connectx-4 (RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7) Dear Openstack community, first let me apologise for my ignorance in this topic, this is the first time I try to setup an environment like this and I don't know anyone who has done it before. I am trying to setup OVS offload following this document https://docs.openstack.org/neutron/rocky/admin/config-ovs-offload.html My Openstack is Rocky deployed through kolla-ansible. I have a feeling my OVS offload is not setup properly as vm creation fails and the error "RepresentorNotFound: Failed getting representor port for PF" can be see in nova-compute logs. # Nova compute logs: 2019-03-05 11:24:46.392 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Attempting claim on node zeus-54.localdomain: memory 8192 MB, disk 80 GB, vcpus 4 CPU 2019-03-05 11:24:46.393 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Total memory: 524173 MB, used: 8704.00 MB 2019-03-05 11:24:46.394 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] memory limit not specified, defaulting to unlimited 2019-03-05 11:24:46.394 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Total disk: 18630 GB, used: 80.00 GB 2019-03-05 11:24:46.395 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] disk limit not specified, defaulting to unlimited 2019-03-05 11:24:46.396 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Total vcpu: 56 VCPU, used: 4.00 VCPU 2019-03-05 11:24:46.396 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] vcpu limit not specified, defaulting to unlimited 2019-03-05 11:24:46.408 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Claim successful on node zeus-54.localdomain 2019-03-05 11:24:47.144 7 INFO nova.virt.libvirt.driver [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Creating image 2019-03-05 11:24:48.957 7 ERROR os_vif [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] Failed to plug vif VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True): RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7 2019-03-05 11:24:48.957 7 ERROR os_vif Traceback (most recent call last): 2019-03-05 11:24:48.957 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/os_vif/__init__.py", line 77, in plug 2019-03-05 11:24:48.957 7 ERROR os_vif plugin.plug(vif, instance_info) 2019-03-05 11:24:48.957 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 215, in plug 2019-03-05 11:24:48.957 7 ERROR os_vif self._plug_vf_passthrough(vif, instance_info) 2019-03-05 11:24:48.957 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 189, in _plug_vf_passthrough 2019-03-05 11:24:48.957 7 ERROR os_vif representor = linux_net.get_representor_port(pf_ifname, vf_num) 2019-03-05 11:24:48.957 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py", line 381, in get_representor_port 2019-03-05 11:24:48.957 7 ERROR os_vif raise exception.RepresentorNotFound(ifname=pf_ifname, vf_num=vf_num) 2019-03-05 11:24:48.957 7 ERROR os_vif RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7 2019-03-05 11:24:48.957 7 ERROR os_vif 2019-03-05 11:24:48.960 7 ERROR nova.virt.libvirt.driver [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Failed to start libvirt guest: InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: Failed getting representor port for PF ens2f1 with 7 2019-03-05 11:24:48.969 7 ERROR os_vif [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] Failed to unplug vif VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True): RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7 2019-03-05 11:24:48.969 7 ERROR os_vif Traceback (most recent call last): 2019-03-05 11:24:48.969 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/os_vif/__init__.py", line 110, in unplug 2019-03-05 11:24:48.969 7 ERROR os_vif plugin.unplug(vif, instance_info) 2019-03-05 11:24:48.969 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 277, in unplug 2019-03-05 11:24:48.969 7 ERROR os_vif self._unplug_vf_passthrough(vif, instance_info) 2019-03-05 11:24:48.969 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 250, in _unplug_vf_passthrough 2019-03-05 11:24:48.969 7 ERROR os_vif representor = linux_net.get_representor_port(pf_ifname, vf_num) 2019-03-05 11:24:48.969 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py", line 381, in get_representor_port 2019-03-05 11:24:48.969 7 ERROR os_vif raise exception.RepresentorNotFound(ifname=pf_ifname, vf_num=vf_num) 2019-03-05 11:24:48.969 7 ERROR os_vif RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7 2019-03-05 11:24:48.969 7 ERROR os_vif 2019-03-05 11:24:48.971 7 INFO nova.virt.libvirt.driver [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Deleting instance files /var/lib/nova/instances/1b03f2e7-0c44-46c2-87f5-d68f93654c58_del 2019-03-05 11:24:48.972 7 INFO nova.virt.libvirt.driver [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Deletion of /var/lib/nova/instances/1b03f2e7-0c44-46c2-87f5-d68f93654c58_del complete 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Instance failed to spawn: InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: Failed getting representor port for PF ens2f1 with 7 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Traceback (most recent call last): 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2368, in _build_resources 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] yield resources 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2132, in _build_and_run_instance 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] block_device_info=block_device_info) 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3080, in spawn 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] destroy_disks_on_failure=True) 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5646, in _create_domain_and_network 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] destroy_disks_on_failure) 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] self.force_reraise() 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] six.reraise(self.type_, self.value, self.tb) 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5605, in _create_domain_and_network 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] self.plug_vifs(instance, network_info) 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 819, in plug_vifs 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] self.vif_driver.plug(instance, vif) 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 807, in plug 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] self._plug_os_vif(instance, vif_obj) 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 787, in _plug_os_vif 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] raise exception.InternalError(msg) 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: Failed getting representor port for PF ens2f1 with 7 2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] 2019-03-05 11:24:49.085 7 INFO nova.compute.manager [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Terminating instance 2019-03-05 11:24:49.091 7 INFO nova.virt.libvirt.driver [-] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Instance destroyed successfully. 2019-03-05 11:24:49.099 7 ERROR os_vif [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] Failed to unplug vif VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True): RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7 2019-03-05 11:24:49.099 7 ERROR os_vif Traceback (most recent call last): 2019-03-05 11:24:49.099 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/os_vif/__init__.py", line 110, in unplug 2019-03-05 11:24:49.099 7 ERROR os_vif plugin.unplug(vif, instance_info) 2019-03-05 11:24:49.099 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 277, in unplug 2019-03-05 11:24:49.099 7 ERROR os_vif self._unplug_vf_passthrough(vif, instance_info) 2019-03-05 11:24:49.099 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 250, in _unplug_vf_passthrough 2019-03-05 11:24:49.099 7 ERROR os_vif representor = linux_net.get_representor_port(pf_ifname, vf_num) 2019-03-05 11:24:49.099 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py", line 381, in get_representor_port 2019-03-05 11:24:49.099 7 ERROR os_vif raise exception.RepresentorNotFound(ifname=pf_ifname, vf_num=vf_num) 2019-03-05 11:24:49.099 7 ERROR os_vif RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7 2019-03-05 11:24:49.099 7 ERROR os_vif 2019-03-05 11:24:49.101 7 INFO nova.virt.libvirt.driver [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Deletion of /var/lib/nova/instances/1b03f2e7-0c44-46c2-87f5-d68f93654c58_del complete 2019-03-05 11:24:49.198 7 INFO nova.compute.manager [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Took 0.11 seconds to destroy the instance on the hypervisor. 2019-03-05 11:24:50.384 7 INFO nova.compute.manager [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Took 0.77 seconds to deallocate network for instance. 2019-03-05 11:24:50.624 7 INFO nova.scheduler.client.report [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] Deleted allocation for instance 1b03f2e7-0c44-46c2-87f5-d68f93654c58 2019-03-05 11:25:07.884 7 INFO nova.compute.resource_tracker [req-1b32a794-0bf0-4abe-a024-699920f791d2 - - - - -] Final resource view: name=zeus-54.localdomain phys_ram=524173MB used_ram=8704MB phys_disk=18630GB used_disk=80GB total_vcpus=56 used_vcpus=4 pci_stats=[PciDevicePool(count=15,numa_node=1,product_id='1016',tags={dev_type='type-VF',physical_network='physnet1'},vendor_id='15b3')] ### This is my environment/configuration # OVS version: docker exec -it openvswitch_vswitchd ovs-vswitchd --version ovs-vswitchd (Open vSwitch) 2.10.1 # Mellanox firmware version: [root@zeus-54 ~]# ibv_devinfo hca_id: mlx5_9 transport: InfiniBand (0) fw_ver: 14.24.1000 # Kernel version: [root@zeus-54 ~]# uname -r 3.10.0-862.el7.x86_64 # OS distro: [root@zeus-54 ~]# cat /etc/*release CentOS Linux release 7.6.1810 (Core) NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7" CentOS Linux release 7.6.1810 (Core) CentOS Linux release 7.6.1810 (Core) # SR-IOV devices: [root@zeus-54 ~]# lspci -D -nn | grep -i mel 0000:88:00.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015] 0000:88:00.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015] 0000:88:00.2 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016] 0000:88:00.3 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016] 0000:88:00.4 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016] 0000:88:00.5 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016] 0000:88:00.6 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016] 0000:88:00.7 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016] 0000:88:01.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016] 0000:88:01.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016] 0000:88:01.2 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016] 0000:88:01.3 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016] 0000:88:01.4 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016] 0000:88:01.5 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016] 0000:88:01.6 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016] 0000:88:01.7 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016] 0000:88:02.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016] 0000:88:02.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016] [root@zeus-54 ~]# ibdev2netdev -v 0000:88:00.0 mlx5_0 (MT4117 - MT1611X09936) CX4121A - ConnectX-4 LX SFP28 fw 14.24.1000 port 1 (ACTIVE) ==> ens2f0 (Up) 0000:88:00.1 mlx5_1 (MT4117 - MT1611X09936) CX4121A - ConnectX-4 LX SFP28 fw 14.24.1000 port 1 (ACTIVE) ==> ens2f1 (Up) 0000:88:01.2 mlx5_10 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f2 (Up) 0000:88:01.3 mlx5_11 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f3 (Up) 0000:88:01.4 mlx5_12 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f4 (Up) 0000:88:01.5 mlx5_13 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f5 (Up) 0000:88:01.6 mlx5_14 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f6 (Up) 0000:88:01.7 mlx5_15 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f7 (Up) 0000:88:02.0 mlx5_16 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> enp136s2 (Up) 0000:88:02.1 mlx5_17 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> enp136s2f1 (Up) 0000:88:00.2 mlx5_2 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> ens2f2 (Up) 0000:88:00.3 mlx5_3 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> ens2f3 (Up) 0000:88:00.4 mlx5_4 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> ens2f4 (Up) 0000:88:00.5 mlx5_5 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> ens2f5 (Up) 0000:88:00.6 mlx5_6 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> ens2f6 (Up) 0000:88:00.7 mlx5_7 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> ens2f7 (Up) 0000:88:01.0 mlx5_8 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1 (Up) 0000:88:01.1 mlx5_9 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f1 (Up) # Embeded switch configuration: [root@zeus-54 ~]# devlink dev eswitch show pci/0000:88:00.0 pci/0000:88:00.0: mode legacy inline-mode none encap enable [root@zeus-54 ~]# devlink dev eswitch show pci/0000:88:00.1 pci/0000:88:00.1: mode legacy inline-mode none encap enable [root@zeus-54 ~]# sudo ethtool -k ens2f0 | grep hw-tc-offload hw-tc-offload: on [root@zeus-54 ~]# sudo ethtool -k ens2f1 | grep hw-tc-offload hw-tc-offload: on [root@zeus-54 ~]# docker exec -it openvswitch_vswitchd sudo ovs-vsctl get Open_vSwitch . other_config {hw-offload="true", max-idle="30000"} [root@zeus-54 ~]# docker exec -it openvswitch_vswitchd sudo ovs-vsctl get Open_vSwitch . other_config {hw-offload="true", max-idle="30000"} # Check ovs offload [root@zeus-54 ~]# docker exec -it openvswitch_vswitchd sudo ovs-dpctl show 2019-03-05T00:50:42Z|00001|dpif_netlink|INFO|The kernel module does not support meters. system@ovs-system: lookups: hit:0 missed:0 lost:0 flows: 0 masks: hit:0 total:0 hit/pkt:0.00 port 0: ovs-system (internal) port 1: br-ex (internal) port 2: br-int (internal) port 3: br-tun (internal) QUESTIONS: Is my kernel version right? Am I supposed to see the PFs attached to a port in the ovs-dpctl show output? What else can I check? Thank you very much NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.