[nova] failing OVS offload on mellanox connectx-4 (RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7)
Manuel Sopena Ballesteros
manuel.sb at garvan.org.au
Wed Mar 6 13:59:24 UTC 2019
Hi Waleed,
Thank you for your email.
You are right the switch was in legacy mode. I changed it and now I can create vms and ports, however the vms does not have any eth interface and I can't see the pci devices.
I open a case with Mellanox Case 00578559 but nobody has replied yet.
I can copy and paste the content of the case here if it is easier for you?
Please let me know
Thank you very much
Manuel
From: Waleed Musa [mailto:waleedm at mellanox.com]
Sent: Wednesday, March 6, 2019 7:14 PM
To: Hamdy Khader; Manuel Sopena Ballesteros; openstack at lists.openstack.org; Moshe Levi
Subject: Re: [nova] failing OVS offload on mellanox connectx-4 (RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7)
Hi Manuel,
I see form the configuration below that the mode is still legacy (so for sure the representors will not be created)
# Embeded switch configuration:
[root at zeus-54 ~]# devlink dev eswitch show pci/0000:88:00.0
pci/0000:88:00.0: mode legacy inline-mode none encap enable
[root at zeus-54 ~]# devlink dev eswitch show pci/0000:88:00.1
pci/0000:88:00.1: mode legacy inline-mode none encap enable
Are you sure you configured it correctly?
https://docs.openstack.org/neutron/rocky/admin/config-ovs-offload.html#configure-open-vswitch-hardware-offloading
Please take care that you need to unbind all the vfs before changing to switchdev mode
Regards
Waleed Mousa
SW Engineer at Mellanox
________________________________
From: Hamdy Khader
Sent: 06 March 2019 09:25:39
To: Manuel Sopena Ballesteros; openstack at lists.openstack.org; Moshe Levi; Waleed Musa
Subject: Re: [nova] failing OVS offload on mellanox connectx-4 (RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7)
Hi,
@Waleed can you help configure offloading for CX4 ?
Regards,
Hamdy
________________________________
From: Manuel Sopena Ballesteros <manuel.sb at garvan.org.au>
Sent: Tuesday, March 5, 2019 7:52 AM
To: openstack at lists.openstack.org
Subject: [nova] failing OVS offload on mellanox connectx-4 (RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7)
Dear Openstack community,
first let me apologise for my ignorance in this topic, this is the first time I try to setup an environment like this and I don't know anyone who has done it before.
I am trying to setup OVS offload following this document https://docs.openstack.org/neutron/rocky/admin/config-ovs-offload.html
My Openstack is Rocky deployed through kolla-ansible.
I have a feeling my OVS offload is not setup properly as vm creation fails and the error "RepresentorNotFound: Failed getting representor port for PF" can be see in nova-compute logs.
# Nova compute logs:
2019-03-05 11:24:46.392 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Attempting claim on node zeus-54.localdomain: memory 8192 MB, disk 80 GB, vcpus 4 CPU
2019-03-05 11:24:46.393 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Total memory: 524173 MB, used: 8704.00 MB
2019-03-05 11:24:46.394 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] memory limit not specified, defaulting to unlimited
2019-03-05 11:24:46.394 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Total disk: 18630 GB, used: 80.00 GB
2019-03-05 11:24:46.395 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] disk limit not specified, defaulting to unlimited
2019-03-05 11:24:46.396 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Total vcpu: 56 VCPU, used: 4.00 VCPU
2019-03-05 11:24:46.396 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] vcpu limit not specified, defaulting to unlimited
2019-03-05 11:24:46.408 7 INFO nova.compute.claims [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Claim successful on node zeus-54.localdomain
2019-03-05 11:24:47.144 7 INFO nova.virt.libvirt.driver [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Creating image
2019-03-05 11:24:48.957 7 ERROR os_vif [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] Failed to plug vif VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True): RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7
2019-03-05 11:24:48.957 7 ERROR os_vif Traceback (most recent call last):
2019-03-05 11:24:48.957 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/os_vif/__init__.py", line 77, in plug
2019-03-05 11:24:48.957 7 ERROR os_vif plugin.plug(vif, instance_info)
2019-03-05 11:24:48.957 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 215, in plug
2019-03-05 11:24:48.957 7 ERROR os_vif self._plug_vf_passthrough(vif, instance_info)
2019-03-05 11:24:48.957 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 189, in _plug_vf_passthrough
2019-03-05 11:24:48.957 7 ERROR os_vif representor = linux_net.get_representor_port(pf_ifname, vf_num)
2019-03-05 11:24:48.957 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py", line 381, in get_representor_port
2019-03-05 11:24:48.957 7 ERROR os_vif raise exception.RepresentorNotFound(ifname=pf_ifname, vf_num=vf_num)
2019-03-05 11:24:48.957 7 ERROR os_vif RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7
2019-03-05 11:24:48.957 7 ERROR os_vif
2019-03-05 11:24:48.960 7 ERROR nova.virt.libvirt.driver [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Failed to start libvirt guest: InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: Failed getting representor port for PF ens2f1 with 7
2019-03-05 11:24:48.969 7 ERROR os_vif [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] Failed to unplug vif VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True): RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7
2019-03-05 11:24:48.969 7 ERROR os_vif Traceback (most recent call last):
2019-03-05 11:24:48.969 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/os_vif/__init__.py", line 110, in unplug
2019-03-05 11:24:48.969 7 ERROR os_vif plugin.unplug(vif, instance_info)
2019-03-05 11:24:48.969 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 277, in unplug
2019-03-05 11:24:48.969 7 ERROR os_vif self._unplug_vf_passthrough(vif, instance_info)
2019-03-05 11:24:48.969 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 250, in _unplug_vf_passthrough
2019-03-05 11:24:48.969 7 ERROR os_vif representor = linux_net.get_representor_port(pf_ifname, vf_num)
2019-03-05 11:24:48.969 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py", line 381, in get_representor_port
2019-03-05 11:24:48.969 7 ERROR os_vif raise exception.RepresentorNotFound(ifname=pf_ifname, vf_num=vf_num)
2019-03-05 11:24:48.969 7 ERROR os_vif RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7
2019-03-05 11:24:48.969 7 ERROR os_vif
2019-03-05 11:24:48.971 7 INFO nova.virt.libvirt.driver [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Deleting instance files /var/lib/nova/instances/1b03f2e7-0c44-46c2-87f5-d68f93654c58_del
2019-03-05 11:24:48.972 7 INFO nova.virt.libvirt.driver [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Deletion of /var/lib/nova/instances/1b03f2e7-0c44-46c2-87f5-d68f93654c58_del complete
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Instance failed to spawn: InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: Failed getting representor port for PF ens2f1 with 7
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Traceback (most recent call last):
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2368, in _build_resources
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] yield resources
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2132, in _build_and_run_instance
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] block_device_info=block_device_info)
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3080, in spawn
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] destroy_disks_on_failure=True)
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5646, in _create_domain_and_network
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] destroy_disks_on_failure)
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] self.force_reraise()
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] six.reraise(self.type_, self.value, self.tb)
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5605, in _create_domain_and_network
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] self.plug_vifs(instance, network_info)
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 819, in plug_vifs
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] self.vif_driver.plug(instance, vif)
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 807, in plug
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] self._plug_os_vif(instance, vif_obj)
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 787, in _plug_os_vif
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] raise exception.InternalError(msg)
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: Failed getting representor port for PF ens2f1 with 7
2019-03-05 11:24:49.083 7 ERROR nova.compute.manager [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58]
2019-03-05 11:24:49.085 7 INFO nova.compute.manager [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Terminating instance
2019-03-05 11:24:49.091 7 INFO nova.virt.libvirt.driver [-] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Instance destroyed successfully.
2019-03-05 11:24:49.099 7 ERROR os_vif [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] Failed to unplug vif VIFHostDevice(active=False,address=fa:16:3e:b8:41:ff,dev_address=0000:88:02.1,dev_type='ethernet',has_traffic_filtering=True,id=3162e3d2-08ea-4674-8b48-9ce2916faf35,network=Network(f630b54b-2b07-44a3-8196-5a0ea9cc7969),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True): RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7
2019-03-05 11:24:49.099 7 ERROR os_vif Traceback (most recent call last):
2019-03-05 11:24:49.099 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/os_vif/__init__.py", line 110, in unplug
2019-03-05 11:24:49.099 7 ERROR os_vif plugin.unplug(vif, instance_info)
2019-03-05 11:24:49.099 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 277, in unplug
2019-03-05 11:24:49.099 7 ERROR os_vif self._unplug_vf_passthrough(vif, instance_info)
2019-03-05 11:24:49.099 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/vif_plug_ovs/ovs.py", line 250, in _unplug_vf_passthrough
2019-03-05 11:24:49.099 7 ERROR os_vif representor = linux_net.get_representor_port(pf_ifname, vf_num)
2019-03-05 11:24:49.099 7 ERROR os_vif File "/usr/lib/python2.7/site-packages/vif_plug_ovs/linux_net.py", line 381, in get_representor_port
2019-03-05 11:24:49.099 7 ERROR os_vif raise exception.RepresentorNotFound(ifname=pf_ifname, vf_num=vf_num)
2019-03-05 11:24:49.099 7 ERROR os_vif RepresentorNotFound: Failed getting representor port for PF ens2f1 with 7
2019-03-05 11:24:49.099 7 ERROR os_vif
2019-03-05 11:24:49.101 7 INFO nova.virt.libvirt.driver [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Deletion of /var/lib/nova/instances/1b03f2e7-0c44-46c2-87f5-d68f93654c58_del complete
2019-03-05 11:24:49.198 7 INFO nova.compute.manager [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Took 0.11 seconds to destroy the instance on the hypervisor.
2019-03-05 11:24:50.384 7 INFO nova.compute.manager [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: 1b03f2e7-0c44-46c2-87f5-d68f93654c58] Took 0.77 seconds to deallocate network for instance.
2019-03-05 11:24:50.624 7 INFO nova.scheduler.client.report [req-8744317f-1a06-48a1-8c04-f495935867d8 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] Deleted allocation for instance 1b03f2e7-0c44-46c2-87f5-d68f93654c58
2019-03-05 11:25:07.884 7 INFO nova.compute.resource_tracker [req-1b32a794-0bf0-4abe-a024-699920f791d2 - - - - -] Final resource view: name=zeus-54.localdomain phys_ram=524173MB used_ram=8704MB phys_disk=18630GB used_disk=80GB total_vcpus=56 used_vcpus=4 pci_stats=[PciDevicePool(count=15,numa_node=1,product_id='1016',tags={dev_type='type-VF',physical_network='physnet1'},vendor_id='15b3')]
### This is my environment/configuration
# OVS version:
docker exec -it openvswitch_vswitchd ovs-vswitchd --version
ovs-vswitchd (Open vSwitch) 2.10.1
# Mellanox firmware version:
[root at zeus-54 ~]# ibv_devinfo
hca_id: mlx5_9
transport: InfiniBand (0)
fw_ver: 14.24.1000
# Kernel version:
[root at zeus-54 ~]# uname -r
3.10.0-862.el7.x86_64
# OS distro:
[root at zeus-54 ~]# cat /etc/*release
CentOS Linux release 7.6.1810 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
CentOS Linux release 7.6.1810 (Core)
CentOS Linux release 7.6.1810 (Core)
# SR-IOV devices:
[root at zeus-54 ~]# lspci -D -nn | grep -i mel
0000:88:00.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015]
0000:88:00.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015]
0000:88:00.2 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:00.3 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:00.4 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:00.5 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:00.6 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:00.7 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:01.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:01.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:01.2 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:01.3 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:01.4 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:01.5 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:01.6 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:01.7 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:02.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
0000:88:02.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
[root at zeus-54 ~]# ibdev2netdev -v
0000:88:00.0 mlx5_0 (MT4117 - MT1611X09936) CX4121A - ConnectX-4 LX SFP28 fw 14.24.1000 port 1 (ACTIVE) ==> ens2f0 (Up)
0000:88:00.1 mlx5_1 (MT4117 - MT1611X09936) CX4121A - ConnectX-4 LX SFP28 fw 14.24.1000 port 1 (ACTIVE) ==> ens2f1 (Up)
0000:88:01.2 mlx5_10 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f2 (Up)
0000:88:01.3 mlx5_11 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f3 (Up)
0000:88:01.4 mlx5_12 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f4 (Up)
0000:88:01.5 mlx5_13 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f5 (Up)
0000:88:01.6 mlx5_14 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f6 (Up)
0000:88:01.7 mlx5_15 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f7 (Up)
0000:88:02.0 mlx5_16 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> enp136s2 (Up)
0000:88:02.1 mlx5_17 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> enp136s2f1 (Up)
0000:88:00.2 mlx5_2 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> ens2f2 (Up)
0000:88:00.3 mlx5_3 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> ens2f3 (Up)
0000:88:00.4 mlx5_4 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> ens2f4 (Up)
0000:88:00.5 mlx5_5 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> ens2f5 (Up)
0000:88:00.6 mlx5_6 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> ens2f6 (Up)
0000:88:00.7 mlx5_7 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> ens2f7 (Up)
0000:88:01.0 mlx5_8 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1 (Up)
0000:88:01.1 mlx5_9 (MT4118 - NA) fw 14.24.1000 port 1 (ACTIVE) ==> enp136s1f1 (Up)
# Embeded switch configuration:
[root at zeus-54 ~]# devlink dev eswitch show pci/0000:88:00.0
pci/0000:88:00.0: mode legacy inline-mode none encap enable
[root at zeus-54 ~]# devlink dev eswitch show pci/0000:88:00.1
pci/0000:88:00.1: mode legacy inline-mode none encap enable
[root at zeus-54 ~]# sudo ethtool -k ens2f0 | grep hw-tc-offload
hw-tc-offload: on
[root at zeus-54 ~]# sudo ethtool -k ens2f1 | grep hw-tc-offload
hw-tc-offload: on
[root at zeus-54 ~]# docker exec -it openvswitch_vswitchd sudo ovs-vsctl get Open_vSwitch . other_config
{hw-offload="true", max-idle="30000"}
[root at zeus-54 ~]# docker exec -it openvswitch_vswitchd sudo ovs-vsctl get Open_vSwitch . other_config
{hw-offload="true", max-idle="30000"}
# Check ovs offload
[root at zeus-54 ~]# docker exec -it openvswitch_vswitchd sudo ovs-dpctl show
2019-03-05T00:50:42Z|00001|dpif_netlink|INFO|The kernel module does not support meters.
system at ovs-system:
lookups: hit:0 missed:0 lost:0
flows: 0
masks: hit:0 total:0 hit/pkt:0.00
port 0: ovs-system (internal)
port 1: br-ex (internal)
port 2: br-int (internal)
port 3: br-tun (internal)
QUESTIONS:
Is my kernel version right?
Am I supposed to see the PFs attached to a port in the ovs-dpctl show output?
What else can I check?
Thank you very much
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190306/c7610f52/attachment-0001.html>
More information about the openstack-discuss
mailing list