Dear Openstack community,

I am facing an issue trying to setup neutron with SR-IOV and would like to ask for some help:

openstack server show fault.

My environment is Openstack Rocky deployed with kolla-ansible.

I have edited the configuration files as suggested by the documentation but for some reason nova can’t find the PCI device for pass-through.

This is  my setup

[root@zeus-59 ~]# lspci -nn | grep -i mell

88:00.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015]

88:00.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015]

88:00.2 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

88:00.3 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

88:00.4 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

88:00.5 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

88:00.6 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

88:00.7 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

88:01.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

88:01.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

88:01.2 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

88:01.3 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

88:01.4 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

88:01.5 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

88:01.6 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

88:01.7 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

88:02.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

88:02.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]

sriov_agent.ini in compute node content:

[sriov_nic]

physical_device_mappings = sriovtenant1:ens2f0,sriovtenant1:ens2f1

exclude_devices =

[securitygroup]

firewall_driver = neutron.agent.firewall.NoopFirewallDriver

nova.conf (nova-compute):

…

[pci]

passthrough_whitelist = [{ "vendor_id": "10de", "product_id": "1db1" }, { "vendor_id": "15b3", "product_id": "1015", "physical_network": "sriovtenant1" }]

alias = { "vendor_id":"10de", "product_id":"1db1", "device_type":"type-PCI", "name":"nv_v100" }

ml2_conf.ini (neutron-server):

[ml2]

type_drivers = flat,vlan,vxlan

tenant_network_types = vxlan

mechanism_drivers = openvswitch,l2population,sriovnicswitch

extension_drivers = port_security

[ml2_type_vlan]

network_vlan_ranges = physnet1, sriovtenant1

[ml2_type_flat]

flat_networks = sriovtenant1

…

Sriov_agent.ini (compute node):

[sriov_nic]

physical_device_mappings = sriovtenant1:ens2f0,sriovtenant1:ens2f1

exclude_devices =

[securitygroup]

firewall_driver = neutron.agent.firewall.NoopFirewallDriver

Create network and subnet:

openstack network create \

    --provider-physical-network sriovtenant1 \

    --provider-network-type flat \

    sriovnet1

openstack subnet create --network sriovnet1 \

  --subnet-range=10.0.0.0/16 \

  --allocation-pool start=10.0.32.10,end=10.0.32.20 \

  sriovnet1_sub1

Create port:

openstack port create --network sriovnet1 --vnic-type direct sriovnet1-port1

Create server:

openstack server create --flavor m1.large \

  --image centos7.5-image \

  --nic port-id=373fe020-7b89-40ab-a8e4-76b82cb47490 \

  --key-name mykey \

  --availability-zone nova:zeus-59.localdomain \

  vm-sriov-neutron-1

The server does not gets created with the following error:

| fault                               | {u'message': u'PCI device not found for request ID 419c5fa0-1b7a-4a83-b691-2fcb0fba94cc.', u'code': 500, u'details': u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1940, in _do_build_and_run_instance\n    filter_properties, request_spec)\n  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2229, in _build_and_run_instance\n    instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'created': u'2019-02-26T04:21:01Z'} |

Nova logs

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager Traceback (most recent call last):

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7778, in _update_available_resource_for_node

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 705, in update_available_resource

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager     resources = self.driver.get_available_resource(nodename)

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6551, in get_available_resource

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager     self._get_pci_passthrough_devices()

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5978, in _get_pci_passthrough_devices

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager     pci_info.append(self._get_pcidev_info(name))

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5939, in _get_pcidev_info

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager     device.update(_get_device_capabilities(device, address))

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5910, in _get_device_capabilities

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager     pcinet_info = self._get_pcinet_info(address)

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5853, in _get_pcinet_info

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager     virtdev = self._host.device_lookup_by_name(devname)

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 873, in device_lookup_by_name

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager     return self.get_connection().nodeDeviceLookupByName(name)

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager     result = proxy_call(self._autowrap, f, *args, **kwargs)

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager     rv = execute(f, *args, **kwargs)

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager     six.reraise(c, e, tb)

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager     rv = meth(*args, **kwargs)

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 4305, in nodeDeviceLookupByName

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager     if ret is None:raise libvirtError('virNodeDeviceLookupByName() failed', conn=self)

2019-02-26 15:36:42.595 7 ERROR nova.compute.manager libvirtError: Node device not found: no node device with matching name 'net_enp136s1f2_36_cc_24_fb_76_e3'

Devices

[root@zeus-59 ~]# docker exec nova_libvirt sudo virsh nodedev-list

block_sda_SATA_SSD_7F2F0759012400117253

computer

drm_card0

net_enp136s1_66_0f_f8_42_3b_bb

net_enp136s1f1_82_e4_57_39_4c_bf

net_enp136s1f2_6e_52_31_b0_04_b7????

net_enp136s1f3_e6_be_60_01_f3_9f

net_enp136s1f4_be_d1_8e_9e_46_ef

net_enp136s1f5_4e_61_1c_40_98_dc

net_enp136s1f6_4a_75_ee_f7_c9_68

net_enp136s1f7_de_3e_da_36_48_02

net_enp136s2_3e_dc_23_b4_ca_c4

net_enp136s2f1_c6_12_aa_52_fa_34

net_enp1s0f0_0c_c4_7a_a4_82_ae

net_enp1s0f1_0c_c4_7a_a4_82_af

net_ens1f0_90_e2_ba_03_4c_c8

net_ens1f1_90_e2_ba_03_4c_c9

net_ens2f0_7c_fe_90_12_22_b4

net_ens2f1_7c_fe_90_12_22_b5

net_ens2f2_36_cc_24_fb_76_e3

net_ens2f3_82_42_06_38_9a_b7

net_ens2f4_7e_c0_bb_98_72_f4

net_ens2f5_be_9c_1c_25_ff_0d

net_ens2f6_2e_01_90_f8_44_b5

net_ens2f7_7e_6a_6d_4e_89_1b

pci_0000_00_00_0

pci_0000_00_01_0

pci_0000_00_02_0

pci_0000_00_02_1

pci_0000_00_02_2

pci_0000_00_03_0

pci_0000_00_03_1

pci_0000_00_03_2

...

Note: documentation says to use vlan but I am using flat networks

Why is nova looking for a device that libvirt doesn’t know about?

Thank you very much