[nova] failing OVS offload on mellanox connectx-4 (RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7)

Manuel Sopena Ballesteros manuel.sb at garvan.org.au
Wed Mar 6 13:59:24 UTC 2019


Hi Waleed,

Thank you for your email.

You are right the switch was in legacy mode. I changed it and now I can create vms and ports, however the vms does not have any eth interface and I can't see the pci devices.

I open a case with Mellanox Case 00578559 but nobody has replied yet.

I can copy and paste the content of the case here if it is easier for you?

Please let me know

Thank you very much

Manuel

From: Waleed Musa [mailto:waleedm at mellanox.com]
Sent: Wednesday, March 6, 2019 7:14 PM
To: Hamdy Khader; Manuel Sopena Ballesteros; openstack at lists.openstack.org; Moshe Levi
Subject: Re: [nova] failing OVS offload on mellanox connectx-4 (RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7)


Hi Manuel,



I see form the configuration below that the mode is still legacy (so for sure the representors will not be created)

# Embeded switch configuration:

[root at zeus-54 ~]# devlink dev eswitch show pci/0000:88:00.0

pci/0000:88:00.0: mode legacy inline-mode none encap enable

[root at zeus-54 ~]# devlink dev eswitch show pci/0000:88:00.1

pci/0000:88:00.1: mode legacy inline-mode none encap enable


Are you sure you configured it correctly?

https://docs.openstack.org/neutron/rocky/admin/config-ovs-offload.html#configure-open-vswitch-hardware-offloading



Please take care that you need to unbind all the vfs before changing to switchdev mode



Regards

Waleed Mousa

SW Engineer at Mellanox

________________________________
From: Hamdy Khader
Sent: 06 March 2019 09:25:39
To: Manuel Sopena Ballesteros; openstack at lists.openstack.org; Moshe Levi; Waleed Musa
Subject: Re: [nova] failing OVS offload on mellanox connectx-4 (RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7)

Hi,

@Waleed can you help configure offloading for CX4 ?

Regards,
Hamdy
________________________________
From: Manuel Sopena Ballesteros <manuel.sb at garvan.org.au>
Sent: Tuesday, March 5, 2019 7:52 AM
To: openstack at lists.openstack.org
Subject: [nova] failing OVS offload on mellanox connectx-4 (RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7)


Dear Openstack community,



first let me apologise for my ignorance in this topic, this is the first time I try to setup an environment like this and I don't know anyone who has done it before.



I am trying to setup OVS offload following this document https://docs.openstack.org/neutron/rocky/admin/config-ovs-offload.html



My Openstack is Rocky deployed through kolla-ansible.



I have a feeling my OVS offload is not setup properly as vm creation fails and the error "RepresentorNotFound: Failed getting representor port for PF" can be see in nova-compute logs.







# Nova compute logs:



2019-03-05 11:24:46.392 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Attempting claim on node zeus-54.localdomain: memory 8192 MB, disk 80 GB, vcpus 4 CPU

2019-03-05 11:24:46.393 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Total memory: 524173 MB, used: 8704.00 MB

2019-03-05 11:24:46.394 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] memory limit not specified, defaulting to unlimited

2019-03-05 11:24:46.394 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Total disk: 18630 GB, used: 80.00 GB

2019-03-05 11:24:46.395 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] disk limit not specified, defaulting to unlimited

2019-03-05 11:24:46.396 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Total vcpu: 56 VCPU, used: 4.00 VCPU

2019-03-05 11:24:46.396 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] vcpu limit not specified, defaulting to unlimited

2019-03-05 11:24:46.408 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Claim successful on node zeus-54.localdomain

2019-03-05 11:24:47.144 7 INFO nova.virt.libvirt.driver [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Creating image

2019-03-05 11:24:48.957 7 ERROR os_vif [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] Failed to plug vif VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True): RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7

2019-03-05 11:24:48.957 7 ERROR os_vif Traceback (most recent call last):

2019-03-05 11:24:48.957 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/os_vif/__init__.py", line 77, in plug

2019-03-05 11:24:48.957 7 ERROR os_vif     plugin.plug(vif, instance_info)

2019-03-05 11:24:48.957 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 215, in plug

2019-03-05 11:24:48.957 7 ERROR os_vif     self._plug_vf_passthrough(vif, instance_info)

2019-03-05 11:24:48.957 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 189, in _plug_vf_passthrough

2019-03-05 11:24:48.957 7 ERROR os_vif     representor = linux_net.get_representor_port(pf_ifname, vf_num)

2019-03-05 11:24:48.957 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py", line 381, in get_representor_port

2019-03-05 11:24:48.957 7 ERROR os_vif     raise exception.RepresentorNotFound(ifname=pf_ifname, vf_num=vf_num)

2019-03-05 11:24:48.957 7 ERROR os_vif RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7

2019-03-05 11:24:48.957 7 ERROR os_vif

2019-03-05 11:24:48.960 7 ERROR nova.virt.libvirt.driver [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Failed to start libvirt guest: InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: Failed getting representor port for PF ens2f1 with 7

2019-03-05 11:24:48.969 7 ERROR os_vif [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] Failed to unplug vif VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True): RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7

2019-03-05 11:24:48.969 7 ERROR os_vif Traceback (most recent call last):

2019-03-05 11:24:48.969 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/os_vif/__init__.py", line 110, in unplug

2019-03-05 11:24:48.969 7 ERROR os_vif     plugin.unplug(vif, instance_info)

2019-03-05 11:24:48.969 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 277, in unplug

2019-03-05 11:24:48.969 7 ERROR os_vif     self._unplug_vf_passthrough(vif, instance_info)

2019-03-05 11:24:48.969 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 250, in _unplug_vf_passthrough

2019-03-05 11:24:48.969 7 ERROR os_vif     representor = linux_net.get_representor_port(pf_ifname, vf_num)

2019-03-05 11:24:48.969 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py", line 381, in get_representor_port

2019-03-05 11:24:48.969 7 ERROR os_vif     raise exception.RepresentorNotFound(ifname=pf_ifname, vf_num=vf_num)

2019-03-05 11:24:48.969 7 ERROR os_vif RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7

2019-03-05 11:24:48.969 7 ERROR os_vif

2019-03-05 11:24:48.971 7 INFO nova.virt.libvirt.driver [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Deleting instance files /var/lib/nova/instances/1b03f2e7-0c44-46c2-87f5-d68f93654c58_del

2019-03-05 11:24:48.972 7 INFO nova.virt.libvirt.driver [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Deletion of /var/lib/nova/instances/1b03f2e7-0c44-46c2-87f5-d68f93654c58_del complete

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Instance failed to spawn: InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: Failed getting representor port for PF ens2f1 with 7

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Traceback (most recent call last):

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2368, in _build_resources

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]     yield resources

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2132, in _build_and_run_instance

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]     block_device_info=block_device_info)

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3080, in spawn

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]     destroy_disks_on_failure=True)

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5646, in _create_domain_and_network

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]     destroy_disks_on_failure)

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]     self.force_reraise()

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]     six.reraise(self.type_, self.value, self.tb)

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5605, in _create_domain_and_network

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]     self.plug_vifs(instance, network_info)

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 819, in plug_vifs

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]     self.vif_driver.plug(instance, vif)

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 807, in plug

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]     self._plug_os_vif(instance, vif_obj)

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 787, in _plug_os_vif

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]     raise exception.InternalError(msg)

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: Failed getting representor port for PF ens2f1 with 7

2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]

2019-03-05 11:24:49.085 7 INFO nova.compute.manager [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Terminating instance

2019-03-05 11:24:49.091 7 INFO nova.virt.libvirt.driver [-] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Instance destroyed successfully.

2019-03-05 11:24:49.099 7 ERROR os_vif [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] Failed to unplug vif VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True): RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7

2019-03-05 11:24:49.099 7 ERROR os_vif Traceback (most recent call last):

2019-03-05 11:24:49.099 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/os_vif/__init__.py", line 110, in unplug

2019-03-05 11:24:49.099 7 ERROR os_vif     plugin.unplug(vif, instance_info)

2019-03-05 11:24:49.099 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 277, in unplug

2019-03-05 11:24:49.099 7 ERROR os_vif     self._unplug_vf_passthrough(vif, instance_info)

2019-03-05 11:24:49.099 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 250, in _unplug_vf_passthrough

2019-03-05 11:24:49.099 7 ERROR os_vif     representor = linux_net.get_representor_port(pf_ifname, vf_num)

2019-03-05 11:24:49.099 7 ERROR os_vif   File "/usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py", line 381, in get_representor_port

2019-03-05 11:24:49.099 7 ERROR os_vif     raise exception.RepresentorNotFound(ifname=pf_ifname, vf_num=vf_num)

2019-03-05 11:24:49.099 7 ERROR os_vif RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7

2019-03-05 11:24:49.099 7 ERROR os_vif

2019-03-05 11:24:49.101 7 INFO nova.virt.libvirt.driver [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Deletion of /var/lib/nova/instances/1b03f2e7-0c44-46c2-87f5-d68f93654c58_del complete

2019-03-05 11:24:49.198 7 INFO nova.compute.manager [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Took 0.11 seconds to destroy the instance on the hypervisor.

2019-03-05 11:24:50.384 7 INFO nova.compute.manager [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Took 0.77 seconds to deallocate network for instance.

2019-03-05 11:24:50.624 7 INFO nova.scheduler.client.report [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] Deleted allocation for instance 1b03f2e7-0c44-46c2-87f5-d68f93654c58

2019-03-05 11:25:07.884 7 INFO nova.compute.resource_tracker [req-1b32a794-0bf0-4abe-a024-699920f791d2 - - - - -] Final resource view: name=zeus-54.localdomain phys_ram=524173MB used_ram=8704MB phys_disk=18630GB used_disk=80GB total_vcpus=56 used_vcpus=4 pci_stats=[PciDevicePool(count=15,numa_node=1,product_id='1016',tags={dev_type='type-VF',physical_network='physnet1'},vendor_id='15b3')]







### This is my environment/configuration



# OVS version:

docker exec -it openvswitch_vswitchd ovs-vswitchd --version

ovs-vswitchd (Open vSwitch) 2.10.1



# Mellanox firmware version:

[root at zeus-54 ~]# ibv_devinfo

hca_id: mlx5_9

        transport:                      InfiniBand (0)

        fw_ver:                         14.24.1000





# Kernel version:

[root at zeus-54 ~]# uname -r

3.10.0-862.el7.x86_64





# OS distro:

[root at zeus-54 ~]# cat /etc/*release

CentOS Linux release 7.6.1810 (Core)

NAME="CentOS Linux"

VERSION="7 (Core)"

ID="centos"

ID_LIKE="rhel fedora"

VERSION_ID="7"

PRETTY_NAME="CentOS Linux 7 (Core)"

ANSI_COLOR="0;31"

CPE_NAME="cpe:/o:centos:centos:7"

HOME_URL="https://www.centos.org/"

BUG_REPORT_URL="https://bugs.centos.org/"



CENTOS_MANTISBT_PROJECT="CentOS-7"

CENTOS_MANTISBT_PROJECT_VERSION="7"

REDHAT_SUPPORT_PRODUCT="centos"

REDHAT_SUPPORT_PRODUCT_VERSION="7"



CentOS Linux release 7.6.1810 (Core)

CentOS Linux release 7.6.1810 (Core)





# SR-IOV devices:

[root at zeus-54 ~]# lspci -D -nn | grep -i mel

0000:88:00.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015]

0000:88:00.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015]

0000:88:00.2 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

0000:88:00.3 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

0000:88:00.4 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

0000:88:00.5 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

0000:88:00.6 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

0000:88:00.7 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

0000:88:01.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

0000:88:01.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

0000:88:01.2 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

0000:88:01.3 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

0000:88:01.4 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

0000:88:01.5 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

0000:88:01.6 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

0000:88:01.7 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

0000:88:02.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

0000:88:02.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]



[root at zeus-54 ~]# ibdev2netdev -v

0000:88:00.0 mlx5_0 (MT4117 - MT1611X09936) CX4121A - ConnectX-4 LX SFP28 fw 14.24.1000 port 1 (ACTIVE) ==> ens2f0 (Up)

0000:88:00.1 mlx5_1 (MT4117 - MT1611X09936) CX4121A - ConnectX-4 LX SFP28 fw 14.24.1000 port 1 (ACTIVE) ==> ens2f1 (Up)

0000:88:01.2 mlx5_10 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f2 (Up)

0000:88:01.3 mlx5_11 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f3 (Up)

0000:88:01.4 mlx5_12 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f4 (Up)

0000:88:01.5 mlx5_13 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f5 (Up)

0000:88:01.6 mlx5_14 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f6 (Up)

0000:88:01.7 mlx5_15 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f7 (Up)

0000:88:02.0 mlx5_16 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> enp136s2 (Up)

0000:88:02.1 mlx5_17 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> enp136s2f1 (Up)

0000:88:00.2 mlx5_2 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> ens2f2 (Up)

0000:88:00.3 mlx5_3 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> ens2f3 (Up)

0000:88:00.4 mlx5_4 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> ens2f4 (Up)

0000:88:00.5 mlx5_5 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> ens2f5 (Up)

0000:88:00.6 mlx5_6 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> ens2f6 (Up)

0000:88:00.7 mlx5_7 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> ens2f7 (Up)

0000:88:01.0 mlx5_8 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1 (Up)

0000:88:01.1 mlx5_9 (MT4118 - NA)  fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f1 (Up)





# Embeded switch configuration:

[root at zeus-54 ~]# devlink dev eswitch show pci/0000:88:00.0

pci/0000:88:00.0: mode legacy inline-mode none encap enable

[root at zeus-54 ~]# devlink dev eswitch show pci/0000:88:00.1

pci/0000:88:00.1: mode legacy inline-mode none encap enable



[root at zeus-54 ~]# sudo ethtool -k ens2f0 | grep hw-tc-offload

hw-tc-offload: on

[root at zeus-54 ~]# sudo ethtool -k ens2f1 | grep hw-tc-offload

hw-tc-offload: on



[root at zeus-54 ~]# docker exec -it openvswitch_vswitchd sudo ovs-vsctl get Open_vSwitch . other_config

{hw-offload="true", max-idle="30000"}

[root at zeus-54 ~]# docker exec -it openvswitch_vswitchd sudo ovs-vsctl get Open_vSwitch . other_config

{hw-offload="true", max-idle="30000"}





# Check ovs offload

[root at zeus-54 ~]# docker exec -it openvswitch_vswitchd sudo ovs-dpctl show

2019-03-05T00:50:42Z|00001|dpif_netlink|INFO|The kernel module does not support meters.

system at ovs-system:

  lookups: hit:0 missed:0 lost:0

  flows: 0

  masks: hit:0 total:0 hit/pkt:0.00

  port 0: ovs-system (internal)

  port 1: br-ex (internal)

  port 2: br-int (internal)

  port 3: br-tun (internal)







QUESTIONS:

Is my kernel version right?

Am I supposed to see the PFs attached to a port in the ovs-dpctl show output?

What else can I check?



Thank you very much


NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190306/c7610f52/attachment-0001.html>


More information about the openstack-discuss mailing list