[Openstack] Issue with assignment of Intel’s QAT Card to VM (PCI-passthrough) using openstack-mitaka release on Cent OS 7.2 host
Chinmaya Dwibedy
ckdwibedy at gmail.com
Wed Jun 8 05:25:23 UTC 2016
Hi Alex,
Yes, the IOMMU is enabled. Here are the logs.
[root at localhost ~(keystone_admin)]# dmesg | grep -e DMAR -e IOMMU
[ 0.000000] ACPI: DMAR 000000007b69a000 00130 (v01 INTEL S2600WT
00000001 INTL 20091013)
[ 0.000000] Intel-IOMMU: enabled
[ 0.134919] dmar: IOMMU 0: reg_base_addr fbffc000 ver 1:0 cap
d2078c106f0466 ecap f020de
[ 0.135618] dmar: IOMMU 1: reg_base_addr c7ffc000 ver 1:0 cap
d2078c106f0466 ecap f020de
[ 0.136940] IOAPIC id 10 under DRHD base 0xfbffc000 IOMMU 0
[ 0.137195] IOAPIC id 8 under DRHD base 0xc7ffc000 IOMMU 1
[ 0.137449] IOAPIC id 9 under DRHD base 0xc7ffc000 IOMMU 1
[ 9.929543] IOMMU 0 0xfbffc000: using Queued invalidation
[ 9.929799] IOMMU 1 0xc7ffc000: using Queued invalidation
[ 9.930097] IOMMU: hardware identity mapping for device 0000:ff:08.0
[ 9.930357] IOMMU: hardware identity mapping for device 0000:ff:08.2
[ 9.930622] IOMMU: hardware identity mapping for device 0000:ff:08.3
[ 9.930880] IOMMU: hardware identity mapping for device 0000:ff:09.0
[ 9.931141] IOMMU: hardware identity mapping for device 0000:ff:09.2
[ 9.931400] IOMMU: hardware identity mapping for device 0000:ff:09.3
[ 9.931661] IOMMU: hardware identity mapping for device 0000:ff:0b.0
[ 9.931919] IOMMU: hardware identity mapping for device 0000:ff:0b.1
[ 9.932176] IOMMU: hardware identity mapping for device 0000:ff:0b.2
[ 9.932434] IOMMU: hardware identity mapping for device 0000:ff:0c.0
[ 9.932695] IOMMU: hardware identity mapping for device 0000:ff:0c.1
[ 9.932952] IOMMU: hardware identity mapping for device 0000:ff:0c.2
[ 9.933210] IOMMU: hardware identity mapping for device 0000:ff:0c.3
[ 9.933466] IOMMU: hardware identity mapping for device 0000:ff:0c.4
[ 9.933733] IOMMU: hardware identity mapping for device 0000:ff:0c.5
[ 9.933993] IOMMU: hardware identity mapping for device 0000:ff:0c.6
[ 9.934252] IOMMU: hardware identity mapping for device 0000:ff:0c.7
[ 9.934516] IOMMU: hardware identity mapping for device 0000:ff:0d.0
[ 9.934773] IOMMU: hardware identity mapping for device 0000:ff:0d.1
[ 9.935031] IOMMU: hardware identity mapping for device 0000:ff:0d.2
[ 9.935288] IOMMU: hardware identity mapping for device 0000:ff:0d.3
[ 9.935549] IOMMU: hardware identity mapping for device 0000:ff:0d.4
[ 9.935807] IOMMU: hardware identity mapping for device 0000:ff:0d.5
[ 9.936064] IOMMU: hardware identity mapping for device 0000:ff:0d.6
[ 9.936324] IOMMU: hardware identity mapping for device 0000:ff:0d.7
[ 9.936589] IOMMU: hardware identity mapping for device 0000:ff:0e.0
[ 9.936848] IOMMU: hardware identity mapping for device 0000:ff:0e.1
[ 9.937108] IOMMU: hardware identity mapping for device 0000:ff:0f.0
[ 9.937367] IOMMU: hardware identity mapping for device 0000:ff:0f.1
[ 9.937631] IOMMU: hardware identity mapping for device 0000:ff:0f.2
[ 9.937890] IOMMU: hardware identity mapping for device 0000:ff:0f.3
[ 9.938148] IOMMU: hardware identity mapping for device 0000:ff:0f.4
[ 9.938408] IOMMU: hardware identity mapping for device 0000:ff:0f.5
[ 9.938669] IOMMU: hardware identity mapping for device 0000:ff:0f.6
[ 9.938928] IOMMU: hardware identity mapping for device 0000:ff:10.0
[ 9.939184] IOMMU: hardware identity mapping for device 0000:ff:10.1
[ 9.939443] IOMMU: hardware identity mapping for device 0000:ff:10.5
[ 9.939705] IOMMU: hardware identity mapping for device 0000:ff:10.6
[ 9.939964] IOMMU: hardware identity mapping for device 0000:ff:10.7
[ 9.940220] IOMMU: hardware identity mapping for device 0000:ff:12.0
[ 9.940481] IOMMU: hardware identity mapping for device 0000:ff:12.1
[ 9.940739] IOMMU: hardware identity mapping for device 0000:ff:12.4
[ 9.940997] IOMMU: hardware identity mapping for device 0000:ff:12.5
[ 9.941252] IOMMU: hardware identity mapping for device 0000:ff:13.0
[ 9.941512] IOMMU: hardware identity mapping for device 0000:ff:13.1
[ 9.941768] IOMMU: hardware identity mapping for device 0000:ff:13.2
[ 9.942025] IOMMU: hardware identity mapping for device 0000:ff:13.3
[ 9.942281] IOMMU: hardware identity mapping for device 0000:ff:13.6
[ 9.942545] IOMMU: hardware identity mapping for device 0000:ff:13.7
[ 9.942802] IOMMU: hardware identity mapping for device 0000:ff:14.0
[ 9.943063] IOMMU: hardware identity mapping for device 0000:ff:14.1
[ 9.943321] IOMMU: hardware identity mapping for device 0000:ff:14.2
[ 9.943582] IOMMU: hardware identity mapping for device 0000:ff:14.3
[ 9.943844] IOMMU: hardware identity mapping for device 0000:ff:14.4
[ 9.944104] IOMMU: hardware identity mapping for device 0000:ff:14.5
[ 9.944364] IOMMU: hardware identity mapping for device 0000:ff:14.6
[ 9.944627] IOMMU: hardware identity mapping for device 0000:ff:14.7
[ 9.944885] IOMMU: hardware identity mapping for device 0000:ff:16.0
[ 9.945143] IOMMU: hardware identity mapping for device 0000:ff:16.1
[ 9.945403] IOMMU: hardware identity mapping for device 0000:ff:16.2
[ 9.945669] IOMMU: hardware identity mapping for device 0000:ff:16.3
[ 9.945929] IOMMU: hardware identity mapping for device 0000:ff:16.6
[ 9.946187] IOMMU: hardware identity mapping for device 0000:ff:16.7
[ 9.946445] IOMMU: hardware identity mapping for device 0000:ff:17.0
[ 9.946706] IOMMU: hardware identity mapping for device 0000:ff:17.1
[ 9.946964] IOMMU: hardware identity mapping for device 0000:ff:17.2
[ 9.947220] IOMMU: hardware identity mapping for device 0000:ff:17.3
[ 9.947480] IOMMU: hardware identity mapping for device 0000:ff:17.4
[ 9.947740] IOMMU: hardware identity mapping for device 0000:ff:17.5
[ 9.947997] IOMMU: hardware identity mapping for device 0000:ff:17.6
[ 9.948256] IOMMU: hardware identity mapping for device 0000:ff:17.7
[ 9.948520] IOMMU: hardware identity mapping for device 0000:ff:1e.0
[ 9.948777] IOMMU: hardware identity mapping for device 0000:ff:1e.1
[ 9.949037] IOMMU: hardware identity mapping for device 0000:ff:1e.2
[ 9.949296] IOMMU: hardware identity mapping for device 0000:ff:1e.3
[ 9.949556] IOMMU: hardware identity mapping for device 0000:ff:1e.4
[ 9.949815] IOMMU: hardware identity mapping for device 0000:ff:1f.0
[ 9.950073] IOMMU: hardware identity mapping for device 0000:ff:1f.2
[ 9.950338] IOMMU: hardware identity mapping for device 0000:7f:08.0
[ 9.950600] IOMMU: hardware identity mapping for device 0000:7f:08.2
[ 9.950858] IOMMU: hardware identity mapping for device 0000:7f:08.3
[ 9.951116] IOMMU: hardware identity mapping for device 0000:7f:09.0
[ 9.951375] IOMMU: hardware identity mapping for device 0000:7f:09.2
[ 9.951639] IOMMU: hardware identity mapping for device 0000:7f:09.3
[ 9.951896] IOMMU: hardware identity mapping for device 0000:7f:0b.0
[ 9.952156] IOMMU: hardware identity mapping for device 0000:7f:0b.1
[ 9.952414] IOMMU: hardware identity mapping for device 0000:7f:0b.2
[ 9.952675] IOMMU: hardware identity mapping for device 0000:7f:0c.0
[ 9.952938] IOMMU: hardware identity mapping for device 0000:7f:0c.1
[ 9.953196] IOMMU: hardware identity mapping for device 0000:7f:0c.2
[ 9.953453] IOMMU: hardware identity mapping for device 0000:7f:0c.3
[ 9.953717] IOMMU: hardware identity mapping for device 0000:7f:0c.4
[ 9.953977] IOMMU: hardware identity mapping for device 0000:7f:0c.5
[ 9.954237] IOMMU: hardware identity mapping for device 0000:7f:0c.6
[ 9.954501] IOMMU: hardware identity mapping for device 0000:7f:0c.7
[ 9.954758] IOMMU: hardware identity mapping for device 0000:7f:0d.0
[ 9.955015] IOMMU: hardware identity mapping for device 0000:7f:0d.1
[ 9.955271] IOMMU: hardware identity mapping for device 0000:7f:0d.2
[ 9.955531] IOMMU: hardware identity mapping for device 0000:7f:0d.3
[ 9.955787] IOMMU: hardware identity mapping for device 0000:7f:0d.4
[ 9.956045] IOMMU: hardware identity mapping for device 0000:7f:0d.5
[ 9.956301] IOMMU: hardware identity mapping for device 0000:7f:0d.6
[ 9.956563] IOMMU: hardware identity mapping for device 0000:7f:0d.7
[ 9.956820] IOMMU: hardware identity mapping for device 0000:7f:0e.0
[ 9.957078] IOMMU: hardware identity mapping for device 0000:7f:0e.1
[ 9.957337] IOMMU: hardware identity mapping for device 0000:7f:0f.0
[ 9.957605] IOMMU: hardware identity mapping for device 0000:7f:0f.1
[ 9.957864] IOMMU: hardware identity mapping for device 0000:7f:0f.2
[ 9.958123] IOMMU: hardware identity mapping for device 0000:7f:0f.3
[ 9.958380] IOMMU: hardware identity mapping for device 0000:7f:0f.4
[ 9.958642] IOMMU: hardware identity mapping for device 0000:7f:0f.5
[ 9.958900] IOMMU: hardware identity mapping for device 0000:7f:0f.6
[ 9.959158] IOMMU: hardware identity mapping for device 0000:7f:10.0
[ 9.959416] IOMMU: hardware identity mapping for device 0000:7f:10.1
[ 9.959675] IOMMU: hardware identity mapping for device 0000:7f:10.5
[ 9.959932] IOMMU: hardware identity mapping for device 0000:7f:10.6
[ 9.960189] IOMMU: hardware identity mapping for device 0000:7f:10.7
[ 9.960445] IOMMU: hardware identity mapping for device 0000:7f:12.0
[ 9.960709] IOMMU: hardware identity mapping for device 0000:7f:12.1
[ 9.960966] IOMMU: hardware identity mapping for device 0000:7f:12.4
[ 9.961225] IOMMU: hardware identity mapping for device 0000:7f:12.5
[ 9.961487] IOMMU: hardware identity mapping for device 0000:7f:13.0
[ 9.961744] IOMMU: hardware identity mapping for device 0000:7f:13.1
[ 9.962002] IOMMU: hardware identity mapping for device 0000:7f:13.2
[ 9.962259] IOMMU: hardware identity mapping for device 0000:7f:13.3
[ 9.962521] IOMMU: hardware identity mapping for device 0000:7f:13.6
[ 9.962780] IOMMU: hardware identity mapping for device 0000:7f:13.7
[ 9.963039] IOMMU: hardware identity mapping for device 0000:7f:14.0
[ 9.963298] IOMMU: hardware identity mapping for device 0000:7f:14.1
[ 9.963564] IOMMU: hardware identity mapping for device 0000:7f:14.2
[ 9.963825] IOMMU: hardware identity mapping for device 0000:7f:14.3
[ 9.964083] IOMMU: hardware identity mapping for device 0000:7f:14.4
[ 9.964342] IOMMU: hardware identity mapping for device 0000:7f:14.5
[ 9.964605] IOMMU: hardware identity mapping for device 0000:7f:14.6
[ 9.964862] IOMMU: hardware identity mapping for device 0000:7f:14.7
[ 9.965121] IOMMU: hardware identity mapping for device 0000:7f:16.0
[ 9.965378] IOMMU: hardware identity mapping for device 0000:7f:16.1
[ 9.965639] IOMMU: hardware identity mapping for device 0000:7f:16.2
[ 9.969888] IOMMU: hardware identity mapping for device 0000:7f:16.3
[ 9.970145] IOMMU: hardware identity mapping for device 0000:7f:16.6
[ 9.970403] IOMMU: hardware identity mapping for device 0000:7f:16.7
[ 9.970663] IOMMU: hardware identity mapping for device 0000:7f:17.0
[ 9.970919] IOMMU: hardware identity mapping for device 0000:7f:17.1
[ 9.971177] IOMMU: hardware identity mapping for device 0000:7f:17.2
[ 9.971436] IOMMU: hardware identity mapping for device 0000:7f:17.3
[ 9.971696] IOMMU: hardware identity mapping for device 0000:7f:17.4
[ 9.971955] IOMMU: hardware identity mapping for device 0000:7f:17.5
[ 9.972213] IOMMU: hardware identity mapping for device 0000:7f:17.6
[ 9.972477] IOMMU: hardware identity mapping for device 0000:7f:17.7
[ 9.972736] IOMMU: hardware identity mapping for device 0000:7f:1e.0
[ 9.972993] IOMMU: hardware identity mapping for device 0000:7f:1e.1
[ 9.973251] IOMMU: hardware identity mapping for device 0000:7f:1e.2
[ 9.973519] IOMMU: hardware identity mapping for device 0000:7f:1e.3
[ 9.973779] IOMMU: hardware identity mapping for device 0000:7f:1e.4
[ 9.974038] IOMMU: hardware identity mapping for device 0000:7f:1f.0
[ 9.974296] IOMMU: hardware identity mapping for device 0000:7f:1f.2
[ 9.974565] IOMMU: hardware identity mapping for device 0000:00:00.0
[ 9.974823] IOMMU: hardware identity mapping for device 0000:00:01.0
[ 9.975081] IOMMU: hardware identity mapping for device 0000:00:02.0
[ 9.975339] IOMMU: hardware identity mapping for device 0000:00:02.2
[ 9.975605] IOMMU: hardware identity mapping for device 0000:00:03.0
[ 9.975863] IOMMU: hardware identity mapping for device 0000:00:03.2
[ 9.976121] IOMMU: hardware identity mapping for device 0000:00:04.0
[ 9.976379] IOMMU: hardware identity mapping for device 0000:00:04.1
[ 9.976641] IOMMU: hardware identity mapping for device 0000:00:04.2
[ 9.976897] IOMMU: hardware identity mapping for device 0000:00:04.3
[ 9.977156] IOMMU: hardware identity mapping for device 0000:00:04.4
[ 9.977412] IOMMU: hardware identity mapping for device 0000:00:04.5
[ 9.977673] IOMMU: hardware identity mapping for device 0000:00:04.6
[ 9.977930] IOMMU: hardware identity mapping for device 0000:00:04.7
[ 9.978187] IOMMU: hardware identity mapping for device 0000:00:05.0
[ 9.978445] IOMMU: hardware identity mapping for device 0000:00:05.1
[ 9.978709] IOMMU: hardware identity mapping for device 0000:00:05.2
[ 9.978967] IOMMU: hardware identity mapping for device 0000:00:05.4
[ 9.979226] IOMMU: hardware identity mapping for device 0000:00:11.0
[ 9.979487] IOMMU: hardware identity mapping for device 0000:00:11.1
[ 9.979747] IOMMU: hardware identity mapping for device 0000:00:11.4
[ 9.980005] IOMMU: hardware identity mapping for device 0000:00:14.0
[ 9.980263] IOMMU: hardware identity mapping for device 0000:00:16.0
[ 9.980524] IOMMU: hardware identity mapping for device 0000:00:16.1
[ 9.980783] IOMMU: hardware identity mapping for device 0000:00:1a.0
[ 9.981041] IOMMU: hardware identity mapping for device 0000:00:1c.0
[ 9.981299] IOMMU: hardware identity mapping for device 0000:00:1c.3
[ 9.981563] IOMMU: hardware identity mapping for device 0000:00:1d.0
[ 9.981821] IOMMU: hardware identity mapping for device 0000:00:1f.0
[ 9.982079] IOMMU: hardware identity mapping for device 0000:00:1f.2
[ 9.982336] IOMMU: hardware identity mapping for device 0000:00:1f.3
[ 9.982605] IOMMU: hardware identity mapping for device 0000:03:00.0
[ 9.982862] IOMMU: hardware identity mapping for device 0000:03:00.1
[ 9.983127] IOMMU: hardware identity mapping for device 0000:05:00.0
[ 9.983385] IOMMU: hardware identity mapping for device 0000:05:00.1
[ 9.983652] IOMMU: hardware identity mapping for device 0000:07:00.0
[ 9.983910] IOMMU: hardware identity mapping for device 0000:07:00.1
[ 9.984173] IOMMU: hardware identity mapping for device 0000:0a:00.0
[ 9.984438] IOMMU: hardware identity mapping for device 0000:80:02.0
[ 9.984701] IOMMU: hardware identity mapping for device 0000:80:02.2
[ 9.984958] IOMMU: hardware identity mapping for device 0000:80:04.0
[ 9.985217] IOMMU: hardware identity mapping for device 0000:80:04.1
[ 9.985478] IOMMU: hardware identity mapping for device 0000:80:04.2
[ 9.985738] IOMMU: hardware identity mapping for device 0000:80:04.3
[ 9.985996] IOMMU: hardware identity mapping for device 0000:80:04.4
[ 9.986257] IOMMU: hardware identity mapping for device 0000:80:04.5
[ 9.986519] IOMMU: hardware identity mapping for device 0000:80:04.6
[ 9.986779] IOMMU: hardware identity mapping for device 0000:80:04.7
[ 9.987045] IOMMU: hardware identity mapping for device 0000:80:05.0
[ 9.987303] IOMMU: hardware identity mapping for device 0000:80:05.1
[ 9.987571] IOMMU: hardware identity mapping for device 0000:80:05.2
[ 9.987829] IOMMU: hardware identity mapping for device 0000:80:05.4
[ 9.988094] IOMMU: hardware identity mapping for device 0000:81:00.0
[ 9.988361] IOMMU: hardware identity mapping for device 0000:82:00.0
[ 9.988623] IOMMU: hardware identity mapping for device 0000:82:08.0
[ 9.988891] IOMMU: hardware identity mapping for device 0000:83:00.0
[ 9.989158] IOMMU: hardware identity mapping for device 0000:86:00.0
[ 9.989422] IOMMU: hardware identity mapping for device 0000:87:00.0
[ 9.989686] IOMMU: hardware identity mapping for device 0000:87:08.0
[ 9.989951] IOMMU: hardware identity mapping for device 0000:88:00.0
[ 9.990210] IOMMU: Setting RMRR:
[ 9.991349] IOMMU: Prepare 0-16MiB unity mapping for LPC
[root at localhost ~(keystone_admin)]#
[root at localhost ~(keystone_admin)]# ls /sys/kernel/iommu_groups/
0 10 12 14 16 18 2 21 23 25 27 29 30 32 34 36 38 4 41
43 45 47 49 50 52 54 56 58 6 61 63 7 9
1 11 13 15 17 19 20 22 24 26 28 3 31 33 35 37 39 40 42
44 46 48 5 51 53 55 57 59 60 62 64 8
[root at localhost ~(keystone_admin)]#
The configurations at nova.conf
pci_alias = {"name": "QuickAssist", "product_id": "0435", "vendor_id":
"8086", "device_type": "type-PCI"}
pci_passthrough_whitelist = [{"vendor_id":"8086","product_id":"0435"}]
scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,PciPassthroughFilter
I have configured the “device_type”: “type-PCI” as stated above in
nova.conf. Then why it shows “dev_type: type-PF” in MariaDB and nova.compute
log? Any idea ?
MariaDB [nova]> select count(*) from pci_devices where status="available"
and deleted=0;
+----------+
| count(*) |
+----------+
| 2 |
+----------+
1 row in set (0.00 sec)
MariaDB [nova]> select pci_stats from compute_nodes;
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
pci_stats
|
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| {"nova_object.version": "1.1", "nova_object.changes": ["objects"], "
nova_object.name": "PciDevicePoolList", "nova_object.data": {"objects":
[{"nova_object.version": "1.1", "nova_object.changes": ["count",
"numa_node", "vendor_id", "product_id", "tags"], "nova_object.name":
"PciDevicePool", "nova_object.data": {"count": 2, "numa_node": 1,
"vendor_id": "8086", "product_id": "0435", "tags": {"dev_type":
"type-PF"}}, "nova_object.namespace": "nova"}]}, "nova_object.namespace":
"nova"} |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
MariaDB [nova]> select * from pci_devices;
+---------------------+---------------------+---------------------+---------+----+-----------------+--------------+------------+-----------+----------+------------------+-----------------+-----------+------------+---------------+------------+-----------+-------------+
| created_at | updated_at | deleted_at | deleted
| id | compute_node_id | address | product_id | vendor_id | dev_type |
dev_id | label | status | extra_info | instance_uuid
| request_id | numa_node | parent_addr |
+---------------------+---------------------+---------------------+---------+----+-----------------+--------------+------------+-----------+----------+------------------+-----------------+-----------+------------+---------------+------------+-----------+-------------+
| 2016-06-08 05:07:26 | 2016-06-08 05:11:16 | NULL | 0
| 5 | 1 | 0000:83:00.0 | 0435 | 8086 | type-PF |
pci_0000_83_00_0 | label_8086_0435 | available | {} | NULL
| NULL | 1 | NULL |
| 2016-06-08 05:07:26 | 2016-06-08 05:11:16 | NULL | 0
| 6 | 1 | 0000:88:00.0 | 0435 | 8086 | type-PF |
pci_0000_88_00_0 | label_8086_0435 | available | {} | NULL
| NULL | 1 | NULL |
+---------------------+---------------------+---------------------+---------+----+-----------------+--------------+------------+-----------+----------+------------------+-----------------+-----------+------------+---------------+------------+-----------+-------------+
6 rows in set (0.00 sec)
MariaDB [nova]>
The output of nova.compute log
nova.compute.resource_tracker [req-9a355251-f9af-428e-b778-cd0bc9d92703 - -
- - -] Compute_service record updated for localhost:localhost
2016-06-08 01:13:13.344 4580 INFO nova.compute.resource_tracker
[req-9a355251-f9af-428e-b778-cd0bc9d92703 - - - - -] Auditing locally
available compute resources for node localhost
2016-06-08 01:13:13.856 4580 INFO nova.compute.resource_tracker
[req-9a355251-f9af-428e-b778-cd0bc9d92703 - - - - -] Total usable vcpus:
36, total allocated vcpus: 8
2016-06-08 01:13:13.856 4580 INFO nova.compute.resource_tracker
[req-9a355251-f9af-428e-b778-cd0bc9d92703 - - - - -] Final resource view:
name=localhost phys_ram=128721MB used_ram=16896MB phys_disk=49GB
used_disk=160GB total_vcpus=36 used_vcpus=8
pci_stats=[PciDevicePool(count=2,numa_node=1,product_id='0435',tags={dev_type='type-PF'},vendor_id='8086')]
Regards,
Chinmaya
On Wed, Jun 8, 2016 at 10:18 AM, Alex <fr487x at gmail.com> wrote:
> Hi Chinmaya,
>
> Are you sure that IOMMU is enabled? That’s how it looks like when it’s on:
> alex at homelab:~$
> alex at homelab:~$ dmesg | grep -e DMAR -e IOMMU
> [ 0.000000] ACPI: DMAR 00000000bedb3008 0000B0 (v01 INTEL DQ67SW
> 00000001 INTL 00000001)
> [ 0.000000] Intel-IOMMU: enabled
> [ 0.037894] dmar: IOMMU 0: reg_base_addr fed91000 ver 1:0 cap
> c9008020660262 ecap f0105a
> [ 0.037984] IOAPIC id 0 under DRHD base 0xfed91000 IOMMU 0
> [ 1.063541] DMAR: No ATSR found
> [ 1.063570] IOMMU 0 0xfed91000: using Queued invalidation
> [ 1.063573] IOMMU: Setting RMRR:
> [ 1.063587] IOMMU: Setting identity map for device 0000:00:1d.0
> [0xbf0ac000 - 0xbf0bafff]
> [ 1.063623] IOMMU: Setting identity map for device 0000:00:1a.0
> [0xbf0ac000 - 0xbf0bafff]
> [ 1.063639] IOMMU: Prepare 0-16MiB unity mapping for LPC
> [ 1.063649] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 -
> 0xffffff]
> [ 12.943351] vboxpci: IOMMU found
> alex at homelab:~$
> alex at homelab:~$ ls /sys/kernel/iommu_groups/
> 0 1 2 3 4 5 6 7 8 9
> alex at homelab:~$
>
> It seems like PCI-Passthrough doesn’t work without IOMMU (intel_iommu=on
> in cmdline).
>
> On Jun 7, 2016, at 7:13 PM, Moshe Levi <moshele at mellanox.com> wrote:
>
>
>
> *From:* Chinmaya Dwibedy [mailto:ckdwibedy at gmail.com <ckdwibedy at gmail.com>
> ]
> *Sent:* Tuesday, June 07, 2016 9:22 AM
> *To:* Moshe Levi <moshele at mellanox.com>
> *Cc:* openstack at lists.openstack.org
> *Subject:* Re: [Openstack] Issue with assignment of Intel’s QAT Card to
> VM (PCI-passthrough) using openstack-mitaka release on Cent OS 7.2 host
>
>
> Hi Moshe,
>
> Thank you for your response. Did not find any errors in nova-compute log.
> Here goes the output.
>
> 2016-06-07 02:10:45.181 118672 INFO nova.compute.resource_tracker
> [req-5f0325a6-2dd8-4a25-b518-9143cc0aac0c - - - - -] Auditing locally
> available compute resources for node localhost
> 2016-06-07 02:10:45.618 118672 INFO nova.compute.resource_tracker
> [req-5f0325a6-2dd8-4a25-b518-9143cc0aac0c - - - - -] Total usable vcpus:
> 36, total allocated vcpus: 8
> 2016-06-07 02:10:45.618 118672 INFO nova.compute.resource_tracker
> [req-5f0325a6-2dd8-4a25-b518-9143cc0aac0c - - - - -] Final resource view:
> name=localhost phys_ram=128657MB used_ram=16896MB phys_disk=49GB
> used_disk=160GB total_vcpus=36 used_vcpus=8
> pci_stats=[PciDevicePool(count=2,numa_node=1,product_id='0435',tags={dev_type='type-PF'},vendor_id='8086')]
> 2016-06-07 02:10:45.692 118672 INFO nova.compute.resource_tracker
> [req-5f0325a6-2dd8-4a25-b518-9143cc0aac0c - - - - -] Compute_service record
> updated for localhost:localhost
> 2016-06-07 02:11:45.182 118672 INFO nova.compute.resource_tracker
> [req-5f0325a6-2dd8-4a25-b518-9143cc0aac0c - - - - -] Auditing locally
> available compute resources for node localhost
> 2016-06-07 02:11:45.631 118672 INFO nova.compute.resource_tracker
> [req-5f0325a6-2dd8-4a25-b518-9143cc0aac0c - - - - -] Total usable vcpus:
> 36, total allocated vcpus: 8
> 2016-06-07 02:11:45.632 118672 INFO nova.compute.resource_tracker
> [req-5f0325a6-2dd8-4a25-b518-9143cc0aac0c - - - - -] Final resource view:
> name=localhost phys_ram=128657MB used_ram=16896MB phys_disk=49GB
> used_disk=160GB total_vcpus=36 used_vcpus=8
> pci_stats=[PciDevicePool(count=2,numa_node=1,product_id='0435',tags={dev_type='type-PF'},vendor_id='8086')]
> 2016-06-07 02:11:45.704 118672 INFO nova.compute.resource_tracker
> [req-5f0325a6-2dd8-4a25-b518-9143cc0aac0c - - - - -] Compute_service record
> updated for localhost:localhost
> 2016-06-07 02:12:47.181 118672 INFO nova.compute.resource_tracker
> [req-5f0325a6-2dd8-4a25-b518-9143cc0aac0c - - - - -] Auditing locally
> available compute resources for node localhost
> 2016-06-07 02:12:47.632 118672 INFO nova.compute.resource_tracker
> [req-5f0325a6-2dd8-4a25-b518-9143cc0aac0c - - - - -] Total usable vcpus:
> 36, total allocated vcpus: 8
> 2016-06-07 02:12:47.632 118672 INFO nova.compute.resource_tracker
> [req-5f0325a6-2dd8-4a25-b518-9143cc0aac0c - - - - -] Final resource view:
> name=localhost phys_ram=128657MB used_ram=16896MB phys_disk=49GB
> used_disk=160GB total_vcpus=36 used_vcpus=8
> pci_stats=[PciDevicePool(count=2,numa_node=1,product_id='0435',tags={dev_type='type-PF'},vendor_id='8086')]
> 2016-06-07 02:12:47.704 118672 INFO nova.compute.resource_tracker
> [req-5f0325a6-2dd8-4a25-b518-9143cc0aac0c - - - - -] Compute_service record
> updated for localhost:localhost
> 2016-06-07 02:13:48.180 118672 INFO nova.compute.resource_tracker
> [req-5f0325a6-2dd8-4a25-b518-9143cc0aac0c - - - - -] Auditing locally
> available compute resources for node localhost
> 2016-06-07 02:13:48.632 118672 INFO nova.compute.resource_tracker
> [req-5f0325a6-2dd8-4a25-b518-9143cc0aac0c - - - - -] Total usable vcpus:
> 36, total allocated vcpus: 8
> 2016-06-07 02:13:48.632 118672 INFO nova.compute.resource_tracker
> [req-5f0325a6-2dd8-4a25-b518-9143cc0aac0c - - - - -] Final resource view:
> name=localhost phys_ram=128657MB used_ram=16896MB phys_disk=49GB
> used_disk=160GB total_vcpus=36 used_vcpus=8
> pci_stats=[PciDevicePool(count=2,numa_node=1,product_id='0435',tags={dev_type='type-PF'},vendor_id='8086')]
> 2016-06-07 02:13:48.701 118672 INFO nova.compute.resource_tracker
> [req-5f0325a6-2dd8-4a25-b518-9143cc0aac0c - - - - -] Compute_service record
> updated for localhost:localhost
>
>
> The below output of nova-scheduler log says that, the
> PciPassthroughFilter has been passed.
> The output below say that Filter PciPassthroughFilter returned 0 hosts
> which mean it didn’t pass
>
>
> 2016-06-07 02:12:26.121 130792 INFO nova.service [-] Starting scheduler
> node (version 13.0.0-1.el7)
> 2016-06-07 02:13:12.449 130792 INFO nova.scheduler.host_manager
> [req-3500a771-cfb2-4b0a-95c4-bf05542a210a - - - - -] Successfully synced
> instances from host 'localhost'.
> 2016-06-07 02:14:09.455 130792 WARNING nova.scheduler.host_manager
> [req-9ab12b26-5e41-4365-b68c-76d177437434 266f5859848e4f39b9725203dda5c3f2
> 4bc608763cee41d9a8df26d3ef919825 - - -] Host localhost has more disk space
> than database expected (25 GB > -111 GB)
> 2016-06-07 02:14:09.457 130792 INFO nova.filters
> [req-9ab12b26-5e41-4365-b68c-76d177437434 266f5859848e4f39b9725203dda5c3f2
> 4bc608763cee41d9a8df26d3ef919825 - - -] Filter PciPassthroughFilter
> returned 0 hosts
> 2016-06-07 02:14:09.457 130792 INFO nova.filters
> [req-9ab12b26-5e41-4365-b68c-76d177437434 266f5859848e4f39b9725203dda5c3f2
> 4bc608763cee41d9a8df26d3ef919825 - - -] Filtering removed all hosts for the
> request with instance ID '7f108c1b-7df5-4336-925e-60e202f38dfc'. Filter
> results: ['RetryFilter: (start: 1, end: 1)', 'AvailabilityZoneFilter:
> (start: 1, end: 1)', 'RamFilter: (start: 1, end: 1)', 'ComputeFilter:
> (start: 1, end: 1)', 'ComputeCapabilitiesFilter: (start: 1, end: 1)',
> 'ImagePropertiesFilter: (start: 1, end: 1)', 'CoreFilter: (start: 1, end:
> 1)', 'PciPassthroughFilter: (start: 1, end: 0)']
> 2016-06-07 02:15:17.430 130792 INFO nova.scheduler.host_manager
> [req-2423445e-f1f1-4d75-9ce8-6d874ea9b27e - - - - -] Successfully synced
> instances from host 'localhost'.
> ~
>
>
>
> Regards,
> Chinmaya
>
> On Tue, Jun 7, 2016 at 11:27 AM, Moshe Levi <moshele at mellanox.com> wrote:
>
> Can you look if you have errors in the nova-computes log?
>
> Also check if you passed the PciPassthroughFilter in the nova-scheduler log
>
>
> *From:* Chinmaya Dwibedy [mailto:ckdwibedy at gmail.com]
> *Sent:* Tuesday, June 07, 2016 8:29 AM
> *To:* openstack at lists.openstack.org
> *Subject:* [Openstack] Issue with assignment of Intel’s QAT Card to VM
> (PCI-passthrough) using openstack-mitaka release on Cent OS 7.2 host
>
>
> Hi All,
>
> I want the Intel’s QAT Card to be used for PCI Passthrough device. But to
> implement PCI-passthrough, when I launch a VM using a flavor configured for
> passthrough, it gives the below errors in nova-conductor.log and instance
> goes into Error state. Note that, I have installed
> openstack-mitaka release on host (Cent OS 7.2). Can anyone please have a
> look into the below stated and let me know if I have missed anything or
> done anything wrong? Thank you in advance for your support and time.
>
>
> When I create an instance, this error is output in nova-conductor.log.
>
> 2016-06-06 05:42:42.005 4898 WARNING nova.scheduler.utils
> [req-94484e27-1998-4e3a-8aa8-06805613ae65 266f5859848e4f39b9725203dda5c3f2
> 4bc608763cee41d9a8df26d3ef919825 - - -] Failed to
> compute_task_build_instances: No valid host was found. There are not enough
> hosts available.
> Traceback (most recent call last):
>
> File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
> line 150, in inner
> return func(*args, **kwargs)
>
> File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line
> 104, in select_destinations
> dests = self.driver.select_destinations(ctxt, spec_obj)
>
> File
> "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line
> 74, in select_destinations
> raise exception.NoValidHost(reason=reason)
>
> *NoValidHost: No valid host was found. There are not enough hosts
> available.*
>
> 2016-06-06 05:42:42.006 4898 WARNING nova.scheduler.utils
> [req-94484e27-1998-4e3a-8aa8-06805613ae65 266f5859848e4f39b9725203dda5c3f2
> 4bc608763cee41d9a8df26d3ef919825 - - -] [instance:
> f1db1cce-0777-4f0e-a141-4b278c2d98b4] Setting instance to ERROR state
>
> In order to assign Intel’s QAT Card to VMs , followed below procedures
>
> Using the PCI bus ID, found out the product id
>
>
> 1) [root at localhost ~(keystone_admin)]# lspci -nn | grep QAT
> 83:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT
> [8086:0435]
> 88:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT
> [8086:0435]
>
>
> [root at localhost ~(keystone_admin)]# cat
> /sys/bus/pci/devices/0000:83:00.0/device
> 0x0435
> [root at localhost ~(keystone_admin)]# cat
> /sys/bus/pci/devices/0000:88:00.0/device
> 0x0435
> [root at localhost ~(keystone_admin)]#
>
>
> [root at localhost ~(keystone_admin)]#
>
> 2) Configured the below stated in nova.conf
>
>
> pci_alias = {"name": "QuickAssist", "product_id": "0435", "vendor_id":
> "8086", "device_type": "type-PCI"}
> pci_passthrough_whitelist = [{"vendor_id":"8086","product_id":"0435"}]
>
>
> scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,
> *PciPassthroughFilter*
> scheduler_available_filters=nova.scheduler.filters.all_filter
>
> 3) service openstack-nova-api restart
> 4) systemctl restart openstack-nova-compute
>
> 5) [root at localhost ~(keystone_admin)]# nova flavor-list
>
> +--------------------------------------+-----------------+-----------+------+-----------+------+-------+-------------+-----------+
> | ID | Name | Memory_MB |
> Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
>
> +--------------------------------------+-----------------+-----------+------+-----------+------+-------+-------------+-----------+
> | 1 | m1.tiny | 512 |
> 1 | 0 | | 1 | 1.0 | True |
> | 2 | m1.small | 2048 |
> 20 | 0 | | 1 | 1.0 | True |
> | 3 | m1.medium | 4096 |
> 40 | 0 | | 2 | 1.0 | True |
> | 4 | m1.large | 8192 |
> 80 | 0 | | 4 | 1.0 | True |
>
> +--------------------------------------+-----------------+-----------+------+-----------+------+-------+-------------+-----------+
> [root at localhost ~(keystone_admin)]#
>
> 6) nova flavor-key 4 set "pci_passthrough:alias"="QuickAssist:1"
>
> 7) [root at localhost ~(keystone_admin)]# nova flavor-show 4
> +----------------------------+--------------------------------------------+
> | Property | Value |
> +----------------------------+--------------------------------------------+
> | OS-FLV-DISABLED:disabled | False |
> | OS-FLV-EXT-DATA:ephemeral | 0 |
> | disk | 80 |
> | extra_specs | {"pci_passthrough:alias": "QuickAssist:1"} |
> | id | 4 |
> | name | m1.large |
> | os-flavor-access:is_public | True |
> | ram | 8192 |
> | rxtx_factor | 1.0 |
> | swap | |
> | vcpus | 4 |
> +----------------------------+--------------------------------------------+
> [root at localhost ~(keystone_admin)]#
>
>
>
> 8) [root at localhost ~(keystone_admin)]# nova boot --flavor 4
> --key_name oskey1 --image bc859dc5-103b-428b-814f-d36e59009454 --nic
> net-id=e2ca118d-1f25-47de-8524-bb2a2635c4be --user-data=./myfile.txt TEST
> WARNING: Option "--key_name" is deprecated; use "--key-name"; this option
> will be removed in novaclient 3.3.0.
>
> +--------------------------------------+--------------------------------------------------------------------------+
> | Property |
> Value |
>
> +--------------------------------------+--------------------------------------------------------------------------+
> | OS-DCF:diskConfig | MANUAL
> |
> | OS-EXT-AZ:availability_zone
> | |
> | OS-EXT-SRV-ATTR:host |
> - |
> | OS-EXT-SRV-ATTR:hypervisor_hostname |
> - |
> | OS-EXT-SRV-ATTR:instance_name |
> instance-00000026 |
> | OS-EXT-STS:power_state |
> 0 |
> | OS-EXT-STS:task_state |
> scheduling |
> | OS-EXT-STS:vm_state |
> building |
> | OS-SRV-USG:launched_at | -
> |
> | OS-SRV-USG:terminated_at |
> - |
> | accessIPv4 |
> |
> | accessIPv6
> | |
> | adminPass |
> 7ZKdcaQut7gu |
> | config_drive
> | |
> | created |
> 2016-06-06T09:42:41Z |
> | flavor | m1.large
> (4) |
> | hostId
> | |
> | id |
> f1db1cce-0777-4f0e-a141-4b278c2d98b4 |
> | image | Benu-vMEG-Dev-M.0.0.0-160525-1347
> (bc859dc5-103b-428b-814f-d36e59009454) |
> | key_name | oskey1
> |
> | metadata |
> {} |
> | name |
> TEST |
> | os-extended-volumes:volumes_attached |
> [] |
> | progress |
> 0 |
> | security_groups |
> default |
> | status |
> BUILD |
> | tenant_id |
> 4bc608763cee41d9a8df26d3ef919825 |
> | updated |
> 2016-06-06T09:42:41Z |
> | user_id |
> 266f5859848e4f39b9725203dda5c3f2 |
>
> +--------------------------------------+--------------------------------------------------------------------------+
> [root@ localhost ~(keystone_admin)]#
>
> .
>
>
> 9) MariaDB [nova]> select * from pci_devices;
>
> +---------------------+---------------------+------------+---------+----+-----------------+--------------+------------+-----------+----------+------------------+-----------------+-----------+------------+---------------+------------+-----------+-------------+
> | created_at | updated_at | deleted_at | deleted | id |
> compute_node_id | address | product_id | vendor_id | dev_type |
> dev_id | label | status | extra_info | instance_uuid
> | request_id | numa_node | parent_addr |
>
> +---------------------+---------------------+------------+---------+----+-----------------+--------------+------------+-----------+----------+------------------+-----------------+-----------+------------+---------------+------------+-----------+-------------+
> | 2016-06-03 12:01:45 | 2016-06-06 09:46:35 | NULL | 0 | 1
> | 1 | 0000:83:00.0 | 0435 | 8086 | type-PF |
> pci_0000_83_00_0 | label_8086_0435 | available | {} | NULL
> | NULL | 1 | NULL |
> | 2016-06-03 12:01:45 | 2016-06-06 09:46:35 | NULL | 0 | 2
> | 1 | 0000:88:00.0 | 0435 | 8086 | type-PF |
> pci_0000_88_00_0 | label_8086_0435 | available | {} | NULL
> | NULL | 1 | NULL |
>
> +---------------------+---------------------+------------+---------+----+-----------------+--------------+------------+-----------+----------+------------------+-----------------+-----------+------------+---------------+------------+-----------+-------------+
> 2 rows in set (0.00 sec)
>
> MariaDB [nova]>
>
> [root at localhost ~(keystone_admin)]# dmesg | grep -e DMAR -e IOMMU
> [ 0.000000] ACPI: DMAR 000000007b69a000 00130 (v01 INTEL S2600WT
> 00000001 INTL 20091013)
> [ 0.128779] dmar: IOMMU 0: reg_base_addr fbffc000 ver 1:0 cap
> d2078c106f0466 ecap f020de
> [ 0.128785] dmar: IOMMU 1: reg_base_addr c7ffc000 ver 1:0 cap
> d2078c106f0466 ecap f020de
> [ 0.128911] IOAPIC id 10 under DRHD base 0xfbffc000 IOMMU 0
> [ 0.128912] IOAPIC id 8 under DRHD base 0xc7ffc000 IOMMU 1
> [ 0.128913] IOAPIC id 9 under DRHD base 0xc7ffc000 IOMMU 1
> [root@ localhost ~(keystone_admin)]#
>
> [root at localhost ~(keystone_admin)]# lscpu | grep Virtualization
> Virtualization: VT-x
> [root at localhost ~(keystone_admin)]#
>
> [root at localhost ~(keystone_admin)]# nova service-list
>
> +----+------------------+--------+----------+---------+-------+----------------------------+-----------------+
> | Id | Binary | Host | Zone | Status | State |
> Updated_at | Disabled Reason |
>
> +----+------------------+--------+----------+---------+-------+----------------------------+-----------------+
> | 9 | nova-cert | localhost | internal | enabled | up |
> 2016-06-07T04:58:28.000000 | - |
> | 10 | nova-consoleauth | localhost | internal | enabled | up |
> 2016-06-07T04:58:30.000000 | - |
> | 11 | nova-scheduler | localhost | internal | enabled | up |
> 2016-06-07T04:58:30.000000 | - |
> | 12 | nova-conductor | localhost | internal | enabled | up |
> 2016-06-07T04:58:29.000000 | - |
> | 18 | nova-compute | localhost | nova | enabled | up |
> 2016-06-07T04:58:29.000000 | - |
>
> +----+------------------+--------+----------+---------+-------+----------------------------+-----------------+
> [root at localhost ~(keystone_admin)]#
>
>
> [root at localhost ~(keystone_admin)]# nova host-list
> +-----------+-------------+----------+
> | host_name | service | zone |
> +-----------+-------------+----------+
> | localhost | cert | internal |
> | localhost | consoleauth | internal |
> | localhost | scheduler | internal |
> | localhost | conductor | internal |
> | localhost | compute | nova |
> +-----------+-------------+----------+
> [root at localhost ~(keystone_admin)]#
>
> [root at localhost ~(keystone_admin)]# neutron agent-list
>
> +--------------------------------------+--------------------+--------+-------------------+-------+----------------+---------------------------+
> | id | agent_type | host |
> availability_zone | alive | admin_state_up | binary |
>
> +--------------------------------------+--------------------+--------+-------------------+-------+----------------+---------------------------+
> | 0e81d20f-b41d-490a-966a-7171880963b9 | Metadata agent | localhost
> | | :-) | True | neutron-metadata-agent |
> | 2ccb17dc-35d8-41cc-8e5d-83496a7e26b0 | Metering agent | localhost
> | | :-) | True | neutron-metering-agent |
> | 6fef2fa7-2479-4d45-889c-b38b854ac3e3 | DHCP agent | localhost |
> nova | :-) | True | neutron-dhcp-agent |
> | 87c976cc-e3cd-4818-aa4f-ee599bf812b1 | L3 agent | localhost |
> nova | :-) | True | neutron-l3-agent |
> | aeb4f399-2281-4ad3-b880-802812910ec8 | Open vSwitch agent | localhost
> | | :-) | True | neutron-openvswitch-agent |
>
> +--------------------------------------+--------------------+--------+-------------------+-------+----------------+---------------------------+
> [root at localhost ~(keystone_admin)]#
>
> [root at localhost ~(keystone_admin)]# nova hypervisor-list
> +----+---------------------+-------+---------+
> | ID | Hypervisor hostname | State | Status |
> +----+---------------------+-------+---------+
> | 1 | localhost | up | enabled |
> +----+---------------------+-------+---------+
> [root at localhost ~(keystone_admin)]#
>
> [root at localhost ~(keystone_admin)]# grep ^virt_type /etc/nova/nova.conf
> virt_type=kvm
> [root at localhost ~(keystone_admin)]#
>
> [root at localhost ~(keystone_admin)]# grep ^compute_driver
> /etc/nova/nova.conf
> compute_driver=libvirt.LibvirtDriver
> [root at localhost ~(keystone_admin)]#
> [root at localhost ~(keystone_admin)]#
>
> Regards,
> Chinmaya
>
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
> --
> Alexey
> fr487x at gmail.com
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160608/c779b843/attachment.html>
More information about the Openstack
mailing list