[Openstack] Nova scheduler tries to assign an already-in-use SRIOV QAT VF to a new instance (openstack-mitaka)
Chinmaya Dwibedy
ckdwibedy at gmail.com
Fri Oct 7 11:22:55 UTC 2016
Hi ,
Can anyone please respond to this email ?
Regards,
Chinmaya
On Thu, Sep 29, 2016 at 5:26 PM, Chinmaya Dwibedy <ckdwibedy at gmail.com>
wrote:
> Hi,
>
>
> Upon trying to create VM instance (Say A) with one QAT VF, it fails with
> the following error i.e., “Requested operation is not valid: PCI device
> 0000:88:04.7 is in use by driver QEMU, domain instance-00000081”. Please
> note that, PCI device 0000:88:04.7 is already being assigned to another VM
> (Say B) . We have installed openstack-mitaka release on CentO7 system.
> It has two Intel QAT devices. There are 32 VF devices available per QAT Device/DH895xCC
> device Out of 64 VFs, only 8 VFs are allocated (to VM instances) and rest
> should be available.
>
> But the nova scheduler tries to assign an already-in-use SRIOV VF to a new
> instance and instance fails. It appears that the nova database is not
> tracking which VF's have already been taken. But if I shut down VM B
> instance, then other instance VM A boots up and vice-versa. Note that, both
> the VM instances cannot run simultaneously because of the aforesaid issue.
>
>
>
> We should always be able to create as many instances with the requested
> PCI devices as there are available VFs.
>
>
> Please feel free to let me know if additional information is needed. Can
> anyone please suggest why it tries to assign same PCI device which has been
> assigned already? Is there any way to resolve this issue? Thank you in
> advance for your support and help.
>
>
> [root at localhost ~(keystone_admin)]# lspci -d:435
>
> 83:00.0 Co-processor: Intel Corporation DH895XCC Series QAT
>
> 88:00.0 Co-processor: Intel Corporation DH895XCC Series QAT
>
> [root at localhost ~(keystone_admin)]#
>
>
>
> [root at localhost ~(keystone_admin)]# lspci -d:443 | grep "QAT Virtual
> Function" | wc -l
>
> 64
>
> [root at localhost ~(keystone_admin)]#
>
>
>
>
>
> [root at localhost ~(keystone_admin)]# mysql -u root nova -e "SELECT
> hypervisor_hostname, address, instance_uuid, status FROM pci_devices JOIN
> compute_nodes oncompute_nodes.id=compute_node_id" | grep 0000:88:04.7
>
> localhost 0000:88:04.7 e10a76f3-e58e-4071-a4dd-7a545e8000de
> allocated
>
> localhost 0000:88:04.7 c3dbac90-198d-4150-ba0f-a80b912d8021
> allocated
>
> localhost 0000:88:04.7 c7f6adad-83f0-4881-b68f-6d154d565ce3
> allocated
>
> localhost.nfv.benunets.com <http://stag48.nfv.benunets.com/> 0000:88:04.7
> 0c3c11a5-f9a4-4f0d-b120-40e4dde843d4 allocated
>
> [root at localhost ~(keystone_admin)]#
>
>
>
> [root at localhost ~(keystone_admin)]# grep -r e10a76f3-e58e-4071-a4dd-7a545e8000de
> /etc/libvirt/qemu
>
> /etc/libvirt/qemu/instance-00000081.xml: <uuid>e10a76f3-e58e-4071-a4dd-
> 7a545e8000de</uuid>
>
> /etc/libvirt/qemu/instance-00000081.xml: <entry
> name='uuid'>e10a76f3-e58e-4071-a4dd-7a545e8000de</entry>
>
> /etc/libvirt/qemu/instance-00000081.xml: <source
> file='/var/lib/nova/instances/e10a76f3-e58e-4071-a4dd-7a545e8000de/disk'/>
>
> /etc/libvirt/qemu/instance-00000081.xml: <source
> path='/var/lib/nova/instances/e10a76f3-e58e-4071-a4dd-7a545e
> 8000de/console.log'/>
>
> /etc/libvirt/qemu/instance-00000081.xml: <source
> path='/var/lib/nova/instances/e10a76f3-e58e-4071-a4dd-7a545e
> 8000de/console.log'/>
>
> [root at localhost ~(keystone_admin)]#
>
> [root at localhost ~(keystone_admin)]# grep -r 0c3c11a5-f9a4-4f0d-b120-40e4dde843d4
> /etc/libvirt/qemu
>
> /etc/libvirt/qemu/instance-000000ab.xml: <uuid>0c3c11a5-f9a4-4f0d-b120-
> 40e4dde843d4</uuid>
>
> /etc/libvirt/qemu/instance-000000ab.xml: <entry
> name='uuid'>0c3c11a5-f9a4-4f0d-b120-40e4dde843d4</entry>
>
> /etc/libvirt/qemu/instance-000000ab.xml: <source
> file='/var/lib/nova/instances/0c3c11a5-f9a4-4f0d-b120-40e4dde843d4/disk'/>
>
> /etc/libvirt/qemu/instance-000000ab.xml: <source
> path='/var/lib/nova/instances/0c3c11a5-f9a4-4f0d-b120-40e4dd
> e843d4/console.log'/>
>
> /etc/libvirt/qemu/instance-000000ab.xml: <source
> path='/var/lib/nova/instances/0c3c11a5-f9a4-4f0d-b120-40e4dd
> e843d4/console.log'/>
>
> [root at localhost ~(keystone_admin)]#
>
>
>
> On the controller, , it appears there are duplicate PCI device entries in
> the Database:
>
>
>
> MariaDB [nova]> select hypervisor_hostname,address,count(*) from
> pci_devices JOIN compute_nodes on compute_nodes.id=compute_node_id group
> by hypervisor_hostname,address having count(*) > 1;
>
> +---------------------+--------------+----------+
>
> | hypervisor_hostname | address | count(*) |
>
> +---------------------+--------------+----------+
>
> | localhost | 0000:05:00.0 | 3 |
>
> | localhost | 0000:05:00.1 | 3 |
>
> | localhost | 0000:83:01.0 | 3 |
>
> | localhost | 0000:83:01.1 | 3 |
>
> | localhost | 0000:83:01.2 | 3 |
>
> | localhost | 0000:83:01.3 | 3 |
>
> | localhost | 0000:83:01.4 | 3 |
>
> | localhost | 0000:83:01.5 | 3 |
>
> | localhost | 0000:83:01.6 | 3 |
>
> | localhost | 0000:83:01.7 | 3 |
>
> | localhost | 0000:83:02.0 | 3 |
>
> | localhost | 0000:83:02.1 | 3 |
>
> | localhost | 0000:83:02.2 | 3 |
>
> | localhost | 0000:83:02.3 | 3 |
>
> | localhost | 0000:83:02.4 | 3 |
>
> | localhost | 0000:83:02.5 | 3 |
>
> | localhost | 0000:83:02.6 | 3 |
>
> | localhost | 0000:83:02.7 | 3 |
>
> | localhost | 0000:83:03.0 | 3 |
>
> | localhost | 0000:83:03.1 | 3 |
>
> | localhost | 0000:83:03.2 | 3 |
>
> | localhost | 0000:83:03.3 | 3 |
>
> | localhost | 0000:83:03.4 | 3 |
>
> | localhost | 0000:83:03.5 | 3 |
>
> | localhost | 0000:83:03.6 | 3 |
>
> | localhost | 0000:83:03.7 | 3 |
>
> | localhost | 0000:83:04.0 | 3 |
>
> | localhost | 0000:83:04.1 | 3 |
>
> | localhost | 0000:83:04.2 | 3 |
>
> | localhost | 0000:83:04.3 | 3 |
>
> | localhost | 0000:83:04.4 | 3 |
>
> | localhost | 0000:83:04.5 | 3 |
>
> | localhost | 0000:83:04.6 | 3 |
>
> | localhost | 0000:83:04.7 | 3 |
>
> | localhost | 0000:88:01.0 | 3 |
>
> | localhost | 0000:88:01.1 | 3 |
>
> | localhost | 0000:88:01.2 | 3 |
>
> | localhost | 0000:88:01.3 | 3 |
>
> | localhost | 0000:88:01.4 | 3 |
>
> | localhost | 0000:88:01.5 | 3 |
>
> | localhost | 0000:88:01.6 | 3 |
>
> | localhost | 0000:88:01.7 | 3 |
>
> | localhost | 0000:88:02.0 | 3 |
>
> | localhost | 0000:88:02.1 | 3 |
>
> | localhost | 0000:88:02.2 | 3 |
>
> | localhost | 0000:88:02.3 | 3 |
>
> | localhost | 0000:88:02.4 | 3 |
>
> | localhost | 0000:88:02.5 | 3 |
>
> | localhost | 0000:88:02.6 | 3 |
>
> | localhost | 0000:88:02.7 | 3 |
>
> | localhost | 0000:88:03.0 | 3 |
>
> | localhost | 0000:88:03.1 | 3 |
>
> | localhost | 0000:88:03.2 | 3 |
>
> | localhost | 0000:88:03.3 | 3 |
>
> | localhost | 0000:88:03.4 | 3 |
>
> | localhost | 0000:88:03.5 | 3 |
>
> | localhost | 0000:88:03.6 | 3 |
>
> | localhost | 0000:88:03.7 | 3 |
>
> | localhost | 0000:88:04.0 | 3 |
>
> | localhost | 0000:88:04.1 | 3 |
>
> | localhost | 0000:88:04.2 | 3 |
>
> | localhost | 0000:88:04.3 | 3 |
>
> | localhost | 0000:88:04.4 | 3 |
>
> | localhost | 0000:88:04.5 | 3 |
>
> | localhost | 0000:88:04.6 | 3 |
>
> | localhost | 0000:88:04.7 | 3 |
>
> +---------------------+--------------+----------+
>
> 66 rows in set (0.00 sec)
>
>
>
> MariaDB [nova]>
>
>
>
> Regards,
>
> Chinmaya
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20161007/d8420737/attachment.html>
More information about the Openstack
mailing list