April 2019 - openstack-discuss - lists.openstack.org

NVMe PCI-passthrough
by Manuel Sopena Ballesteros 03 Apr '19

03 Apr '19

Dear Openstack community, I am trying to attach NVMe drives to my vms and would like to check the configuration I have because for some reason it is not working. This is my physical host details: [root@zeus-59 ~]# lspci -nn -D | grep -i ssd 0000:03:00.0 Non-Volatile memory controller [0108]: Intel Corporation PCIe Data Center SSD [8086:0953] (rev 01) 0000:04:00.0 Non-Volatile memory controller [0108]: Intel Corporation PCIe Data Center SSD [8086:0953] (rev 01) 0000:06:00.0 Non-Volatile memory controller [0108]: Intel Corporation PCIe Data Center SSD [8086:0953] (rev 01) 0000:07:00.0 Non-Volatile memory controller [0108]: Intel Corporation PCIe Data Center SSD [8086:0953] (rev 01) 0000:08:00.0 Non-Volatile memory controller [0108]: Intel Corporation PCIe Data Center SSD [8086:0953] (rev 01) 0000:09:00.0 Non-Volatile memory controller [0108]: Intel Corporation PCIe Data Center SSD [8086:0953] (rev 01) 0000:84:00.0 Non-Volatile memory controller [0108]: Intel Corporation PCIe Data Center SSD [8086:0953] (rev 01) 0000:85:00.0 Non-Volatile memory controller [0108]: Intel Corporation PCIe Data Center SSD [8086:0953] (rev 01) 0000:86:00.0 Non-Volatile memory controller [0108]: Intel Corporation PCIe Data Center SSD [8086:0953] (rev 01) 0000:87:00.0 Non-Volatile memory controller [0108]: Intel Corporation PCIe Data Center SSD [8086:0953] (rev 01) My idea is to attach either 0000:03:00.0 or 0000:04:00.0 to a vm This is how I identify the block devices for each pci device: [root@zeus-59 ~]# ls -l /sys/bus/pci/devices/0000\:03\:00.0/nvme total 0 drwxr-xr-x. 4 root root 0 Apr 3 14:02 nvme0 [root@zeus-59 ~]# ls -l /sys/bus/pci/devices/0000\:04\:00.0/nvme total 0 drwxr-xr-x. 4 root root 0 Apr 3 14:02 nvme1 I installed the OS on the physical machine leaving those 2 drives intact: [root@zeus-59 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 59.6G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 4G 0 part [SWAP] sr0 11:0 1 906M 0 rom nvme0n1 259:15 0 1.8T 0 disk nvme1n1 259:0 0 1.8T 0 disk nvme2n1 259:11 0 1.8T 0 disk └─nvme2n1p1 259:12 0 1.8T 0 part /var/lib/docker/btrfs nvme3n1 259:6 0 1.8T 0 disk └─nvme3n1p1 259:7 0 1.8T 0 part nvme4n1 259:13 0 1.8T 0 disk └─nvme4n1p1 259:14 0 1.8T 0 part nvme5n1 259:5 0 1.8T 0 disk └─nvme5n1p1 259:8 0 1.8T 0 part nvme6n1 259:17 0 1.8T 0 disk └─nvme6n1p1 259:18 0 1.8T 0 part nvme7n1 259:9 0 1.8T 0 disk └─nvme7n1p1 259:10 0 1.8T 0 part nvme8n1 259:3 0 1.8T 0 disk └─nvme8n1p1 259:4 0 1.8T 0 part nvme9n1 259:1 0 1.8T 0 disk └─nvme9n1p1 259:2 0 1.8T 0 part The next thing is to configure Openstack nova api config [pci] alias = { "vendor_id":"8086", "product_id":"0953", "device_type":"type-PCI", "name":"nvme"} nova scheduler config [filter_scheduler] enabled_filters = RetryFilter, AvailabilityZoneFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, PciPassthroughFilter available_filters = nova.scheduler.filters.all_filters nova compute config [filter_scheduler] enabled_filters = RetryFilter, AvailabilityZoneFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, PciPassthroughFilter available_filters = nova.scheduler.filters.all_filters [pci] passthrough_whitelist = [ {"address":"0000:03:00.0"}, {"address":"0000:04:00.0"} ] alias = { "vendor_id":"8086", "product_id":"0953", "device_type":"type-PCI", "name":"nvme"} Then I create my flavor openstack flavor create nvme.small --ram 64000 --disk 10 --vcpus 7 --property "pci_passthrough:alias"="nvme:1" I created the vm towards the host with the free nvme drives [root@openstack-deployment ~]# openstack server create --flavor nvme.small --image centos7.5-image --nic net-id=hpc --security-group admin --key-name mykey --availability-zone nova:zeus-59.localdomain test_nvme_small Vm creation fails and I can see this error in the nova compute logs 2019-04-03 16:50:35.512 7 ERROR nova.virt.libvirt.guest [req-8478a99f-bad0-43d5-b405-55c45e8d8cae 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] Error launching a defined domain with XML: <domain type='kvm'> <name>instance-00000090</name> ... <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/var/lib/nova/instances/a261fcd4-eca3-4982-8b2a-1df33087ab40/disk'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> ... 2019-04-03 16:50:35.513 7 ERROR nova.virt.libvirt.driver [req-8478a99f-bad0-43d5-b405-55c45e8d8cae 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: a261fcd4-eca3-4982-8b2a-1df33087ab40] Failed to start libvirt guest: libvirtError: unsupported configuration: host doesn't support passthrough of host PCI devices Any idea or advice would be very helpful Thank you very much NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.

3 3

cinder 14.0.0.0rc2 (stein)
by no-reply＠openstack.org 03 Apr '19

03 Apr '19

Hello everyone, A new release candidate for cinder for the end of the Stein cycle is available! You can find the source code tarball at: https://tarballs.openstack.org/cinder/ Unless release-critical issues are found that warrant a release candidate respin, this candidate will be formally released as the final Stein release. You are therefore strongly encouraged to test and validate this tarball! Alternatively, you can directly test the stable/stein release branch at: https://git.openstack.org/cgit/openstack/cinder/log/?h=stable/stein Release notes for cinder can be found at: https://docs.openstack.org/releasenotes/cinder/ If you find an issue that could be considered release-critical, please file it at: https://bugs.launchpad.net/cinder/+bugs and tag it *stein-rc-potential* to bring it to the cinder release crew's attention.

1 0

[trove] CI job migration to Bionic for stable/stein
by Lingxian Kong 03 Apr '19

03 Apr '19

Hi team, I've fixed the problems that blocked the job migration for the master, now according to the community requirement, we also need to finish the migration for stable/stein. I've already backported the needed patches, please review or reply directly to this email if you have any concerns. https://review.openstack.org/#/c/649744/ https://review.openstack.org/#/c/649745/ --- Cheers, Lingxian Kong Catalyst Cloud

1 0

keystone 15.0.0.0rc2 (stein)
by no-reply＠openstack.org 03 Apr '19

03 Apr '19

Hello everyone, A new release candidate for keystone for the end of the Stein cycle is available! You can find the source code tarball at: https://tarballs.openstack.org/keystone/ Unless release-critical issues are found that warrant a release candidate respin, this candidate will be formally released as the final Stein release. You are therefore strongly encouraged to test and validate this tarball! Alternatively, you can directly test the stable/stein release branch at: https://git.openstack.org/cgit/openstack/keystone/log/?h=stable/stein Release notes for keystone can be found at: https://docs.openstack.org/releasenotes/keystone/ If you find an issue that could be considered release-critical, please file it at: https://bugs.launchpad.net/keystone/+bugs and tag it *stein-rc-potential* to bring it to the keystone release crew's attention.

1 0

networking-powervm 8.0.0.0rc2 (stein)
by no-reply＠openstack.org 03 Apr '19

03 Apr '19

Hello everyone, A new release candidate for networking-powervm for the end of the Stein cycle is available! You can find the source code tarball at: https://tarballs.openstack.org/networking-powervm/ Unless release-critical issues are found that warrant a release candidate respin, this candidate will be formally released as the final Stein release. You are therefore strongly encouraged to test and validate this tarball! Alternatively, you can directly test the stable/stein release branch at: https://git.openstack.org/cgit/openstack/networking-powervm/log/?h=stable/s… Release notes for networking-powervm can be found at: https://docs.openstack.org/releasenotes/networking-powervm/ If you find an issue that could be considered release-critical, please file it at: https://bugs.launchpad.net/networking-powervm/+bugs and tag it *stein-rc-potential* to bring it to the networking-powervm release crew's attention.

1 0

ceilometer-powervm 8.0.0.0rc2 (stein)
by no-reply＠openstack.org 03 Apr '19

03 Apr '19

Hello everyone, A new release candidate for ceilometer-powervm for the end of the Stein cycle is available! You can find the source code tarball at: https://tarballs.openstack.org/ceilometer-powervm/ Unless release-critical issues are found that warrant a release candidate respin, this candidate will be formally released as the final Stein release. You are therefore strongly encouraged to test and validate this tarball! Alternatively, you can directly test the stable/stein release branch at: https://git.openstack.org/cgit/openstack/ceilometer-powervm/log/?h=stable/s… Release notes for ceilometer-powervm can be found at: https://docs.openstack.org/releasenotes/ceilometer-powervm/ If you find an issue that could be considered release-critical, please file it at: https://bugs.launchpad.net/ceilometer-powervm/+bugs and tag it *stein-rc-potential* to bring it to the ceilometer-powervm release crew's attention.

1 0

nova_powervm 8.0.0.0rc2 (stein)
by no-reply＠openstack.org 03 Apr '19

03 Apr '19

Hello everyone, A new release candidate for nova_powervm for the end of the Stein cycle is available! You can find the source code tarball at: https://tarballs.openstack.org/nova-powervm/ Unless release-critical issues are found that warrant a release candidate respin, this candidate will be formally released as the final Stein release. You are therefore strongly encouraged to test and validate this tarball! Alternatively, you can directly test the stable/stein release branch at: https://git.openstack.org/cgit/openstack/nova-powervm/log/?h=stable/stein Release notes for nova_powervm can be found at: https://docs.openstack.org/releasenotes/nova-powervm/ If you find an issue that could be considered release-critical, please file it at: https://bugs.launchpad.net/nova-powervm/+bugs and tag it *stein-rc-potential* to bring it to the nova_powervm release crew's attention.

1 0

openstack-placement 1.0.0.0rc3 (stein)
by no-reply＠openstack.org 03 Apr '19

03 Apr '19

Hello everyone, A new release candidate for openstack-placement for the end of the Stein cycle is available! You can find the source code tarball at: https://tarballs.openstack.org/placement/ Unless release-critical issues are found that warrant a release candidate respin, this candidate will be formally released as the final Stein release. You are therefore strongly encouraged to test and validate this tarball! Alternatively, you can directly test the stable/stein release branch at: https://git.openstack.org/cgit/openstack/placement/log/?h=stable/stein Release notes for openstack-placement can be found at: https://docs.openstack.org/releasenotes/placement/ If you find an issue that could be considered release-critical, please file it at: https://storyboard.openstack.org/#!/project/openstack/placement and tag it *stein-rc-potential* to bring it to the openstack-placement release crew's attention.

1 0

[release][ptl] Delays on releasing RC requests
by Sean McGinnis 03 Apr '19

03 Apr '19

Just a heads up to anyone that is proposing a new RC release by this Thursday's final RC deadline. Sphinx 2.0 was released last Thursday, removing some deprecations that were not noticed in various libs and extensions that end up being used in our docs jobs. Our immediate issue is sphinxcontib.datatemplates that has been updated to handle not using the deprecated calls. But we have been blocked on a requirements update to allow that newer release to be used. We think (fingers crossed) that we have all the stars aligned where that should be merging in about an hour or so. Once it does, we should be able to recheck any existing release requests that have failing doc jobs and get those processed. If you were planning on requesting a new RC today, please go ahead and submit the request. If you have any questions, please stop by the #openstack-release channel. Thanks! Sean

1 1

[nova] super long online_data_migrations
by Mohammed Naser 03 Apr '19

03 Apr '19

Hi there, During upgrades, I've noticed that when running online_data_migrations with "infinite-until-done" mode, it loops over all of the migrations one by one. However, one of the online data migrations (instance_obj.populate_missing_availability_zones) makes a query that takes a really long time as it seems inefficient (which eventually results in 0, cause it already ran), which means as it loops in "blocks" of 50, there's almost a 2-3 to 8 minute wait in really large environments. The question ends up in specific: SELECT count(*) AS count_1 FROM (SELECT instance_extra.created_at AS instance_extra_created_at, instance_extra.updated_at AS instance_extra_updated_at, instance_extra.deleted_at AS instance_extra_deleted_at, instance_extra.deleted AS instance_extra_deleted, instance_extra.id AS instance_extra_id, instance_extra.instance_uuid AS instance_extra_instance_uuid FROM instance_extra WHERE instance_extra.keypairs IS NULL AND instance_extra.deleted = 0) AS anon_1 The explain for the DB query in this example: +------+-------------+----------------+------+---------------+------+---------+------+--------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+----------------+------+---------------+------+---------+------+--------+-------------+ | 1 | SIMPLE | instance_extra | ALL | NULL | NULL | NULL | NULL | 382473 | Using where | +------+-------------+----------------+------+---------------+------+---------+------+--------+-------------+ It's possible that it can be ever worse, as this number is from another very-long running environments. +------+-------------+----------------+------+---------------+------+---------+------+---------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+----------------+------+---------------+------+---------+------+---------+-------------+ | 1 | SIMPLE | instance_extra | ALL | NULL | NULL | NULL | NULL | 3008741 | Using where | +------+-------------+----------------+------+---------------+------+---------+------+---------+-------------+ I'm not the SQL expert, could we not optimize this? Alternatively, could we update the online data migrations code to "pop out" any of the migrations that return 0 for the next iteration, that way it only works on those online_data_migrations that *have* to be done, and ignore those it knows are done? Thanks, Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser(a)vexxhost.com W. http://vexxhost.com

5 7