Hi colleagues, does anybody here use Linstor as backend for Openstack? I can not launch VM (synopsis below) while everything looks fine until the last step :-) Few words about Openstack's storage configuration: - controllers: cinder-api, cinder-scheduler, cinder-volume (modified with latest driver from Linstor's git), linstor-client, python-linstor - compute nodes: linstor-controller, linstor-satellite, linstor-client, python-linstor; node types - Combined - cinder.conf[1] - volume-type registered with corresponding properties[2] I'm able to manipulate volumes - create/delete[3] and every existing volume has the corresponding drbd/linstor repr[4] and LVM LV[5] on the compute nodes, so, from storage point of view, it looks as correct and ready to use. The problem is when I try to create VM using $ openstack server create --flavor [...] --network [...] --volume 15c4fa98-76a5-44d6-9701-c89a8a848b82 fearst Nova fails[6] because of invalid VM's libvirt XML definition (note "source dev='None'" below): <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='None'/> <target dev='vda' bus='virtio'/> <serial>15c4fa98-76a5-44d6-9701-c89a8a848b82</serial> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> I do not have another working DRBD/Linstor/Openstack installation and can't even imagine, where to look into the problem - whether it is in Nova, Cinder or in Linstor configuration itself. I will appreciate if anybody can give me some guidance on where to seek for the solution to the problem. In my lab I'm using Openstack Train on Ubuntu 18.04.4: nova-compute 2:20.1.1-0ubuntu1~cloud0 nova-compute-kvm 2:20.1.1-0ubuntu1~cloud0 nova-compute-libvirt 2:20.1.1-0ubuntu1~cloud0 nova-api 2:20.1.1-0ubuntu1~cloud0 nova-common 2:20.1.1-0ubuntu1~cloud0 nova-conductor 2:20.1.1-0ubuntu1~cloud0 nova-novncproxy 2:20.1.1-0ubuntu1~cloud0 nova-scheduler 2:20.1.1-0ubuntu1~cloud0 cinder-api 2:15.1.0-0ubuntu1~cloud0 cinder-common 2:15.1.0-0ubuntu1~cloud0 cinder-scheduler 2:15.1.0-0ubuntu1~cloud0 cinder-volume 2:15.1.0-0ubuntu1~cloud0 python3-cinder 2:15.1.0-0ubuntu1~cloud0 python3-cinderclient 1:5.0.0-0ubuntu2~cloud0 python3-nova 2:20.1.1-0ubuntu1~cloud0 python3-novaclient 2:15.1.0-0ubuntu2~cloud0 Thank you! ==================================== [1] cinder.conf [DEFAULT] enabled_backends = linstor default_volume_type = linstor [linstor] storage_availability_zone = nova volume_backend_name = linstor volume_driver = cinder.volume.drivers.linstordrv.LinstorDrbdDriver linstor_autoplace_count = 2 linstor_default_volume_group_name=DfltRscGrp linstor_default_uri=linstor://runner linstor_default_storage_pool_name=drbdpool linstor_default_resource_size=1 linstor_volume_downsize_factor=4096 where 'runner' is virtual ip of linstor-controller, protected by pacemaker. ==================================== [2] openstack volume type show linstor +--------------------+--------------------------------------+ | Field | Value | +--------------------+--------------------------------------+ | access_project_ids | None | | description | None | | id | d2025962-503a-4f37-93bd-b766bb346a42 | | is_public | True | | name | linstor | | properties | volume_backend_name='linstor' | | qos_specs_id | None | +--------------------+--------------------------------------+ ==================================== [3] openstack volume show fearst +--------------------------------+-----------------------------------------------------+ | Field | Value | +--------------------------------+-----------------------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | true | | consistencygroup_id | None | | created_at | 2020-05-21T08:42:11.000000 | | description | None | | encrypted | False | | id | 15c4fa98-76a5-44d6-9701-c89a8a848b82 | | migration_status | None | | multiattach | False | | name | fearst | | os-vol-host-attr:host | ctrl1@linstor#linstor | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 7acec404393344cbabde07a22bbe6b3f | | properties | | | replication_status | None | | size | 5 | | snapshot_id | None | | source_volid | None | | status | available | | type | linstor | | updated_at | 2020-05-21T08:42:18.000000 | | user_id | 1048a19e94234451a61c3d9d46907ceb | | volume_image_metadata | {'signature_verified': 'False', | | | 'image_id': '15e992e8-a601-44cd-a2f0-8f34588a4d18',| | | 'image_name': 'cirros', | | | 'checksum': '1d3062cd89af34e419f7100277f38b2b', | | | 'container_format': 'bare', | | | 'disk_format': 'qcow2', | | | 'min_disk': '0', | | | 'min_ram': '0', | | | 'size': '16338944'} | +--------------------------------+-----------------------------------------------------+ ==================================== [4] linstor r l ╭───────────────────────────────────────────────────────────────────────────────────╮ ┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊ ╞═══════════════════════════════════════════════════════════════════════════════════╡ ┊ CV_15c4fa98-76a5-44d6-9701-c89a8a848b82 ┊ cmp1 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ ┊ CV_15c4fa98-76a5-44d6-9701-c89a8a848b82 ┊ cmp2 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ ╰───────────────────────────────────────────────────────────────────────────────────╯ Note, that cmp3 is arbiter and has no 'by-disk' representation: root@cmp1:~# ls -lR /dev/drbd* brw-rw---- 1 root disk 147, 1000 May 21 11:42 /dev/drbd1000 /dev/drbd/by-disk/sds: lrwxrwxrwx 1 root root 17 May 21 11:42 CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000 -> ../../../drbd1000 /dev/drbd/by-res/CV_15c4fa98-76a5-44d6-9701-c89a8a848b82: lrwxrwxrwx 1 root root 17 May 21 11:42 0 -> ../../../drbd1000 root@cmp2:~# ls -lR /dev/drbd* brw-rw---- 1 root disk 147, 1000 May 21 11:42 /dev/drbd1000 /dev/drbd/by-disk/sds: lrwxrwxrwx 1 root root 17 May 21 11:42 CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000 -> ../../../drbd1000 /dev/drbd/by-res/CV_15c4fa98-76a5-44d6-9701-c89a8a848b82: lrwxrwxrwx 1 root root 17 May 21 11:42 0 -> ../../../drbd1000 root@cmp3:~# ls -lR /dev/drbd* brw-rw---- 1 root disk 147, 1000 May 21 11:42 /dev/drbd1000 /dev/drbd/by-res/CV_15c4fa98-76a5-44d6-9701-c89a8a848b82: lrwxrwxrwx 1 root root 17 May 21 11:42 0 -> ../../../drbd1000 ==================================== [5] root@cmp1:~# lvdisplay [ ... ] --- Logical volume --- LV Path /dev/sds/CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000 LV Name CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000 VG Name sds LV UUID jGzcIQ-yRpH-RKxP-ExsH-JSn6-kTFn-8kpKYO LV Write Access read/write LV Creation host, time cmp1, 2020-05-21 11:42:14 +0300 LV Pool name thin LV Status available # open 2 LV Size 5.00 GiB Mapped size 0.02% Current LE 1280 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:4 root@cmp2:~# lvdisplay [ ... ] --- Logical volume --- LV Path /dev/sds/CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000 LV Name CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000 VG Name sds LV UUID 8pVwMo-e3ed-Sy0w-06IZ-M7Mh-YVIL-BtKZEG LV Write Access read/write LV Creation host, time cmp2, 2020-05-21 11:42:14 +0300 LV Pool name thin LV Status available # open 2 LV Size 5.00 GiB Mapped size 0.02% Current LE 1280 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:4 ==================================== [6] Traceback of nova-compute: 3304 ERROR nova.compute.manager [] Traceback (most recent call last): 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2419, in _build_and_run_instance 3304 ERROR nova.compute.manager [] block_device_info=block_device_info) 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3474, in spawn 3304 ERROR nova.compute.manager [] power_on=power_on) 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6261, in _create_domain_and_network 3304 ERROR nova.compute.manager [] destroy_disks_on_failure) 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 3304 ERROR nova.compute.manager [] self.force_reraise() 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 3304 ERROR nova.compute.manager [] six.reraise(self.type_, self.value, self.tb) 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise 3304 ERROR nova.compute.manager [] raise value 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6230, in _create_domain_and_network 3304 ERROR nova.compute.manager [] post_xml_callback=post_xml_callback) 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6164, in _create_domain 3304 ERROR nova.compute.manager [] guest.launch(pause=pause) 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 143, in launch 3304 ERROR nova.compute.manager [] self._encoded_xml, errors='ignore') 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 3304 ERROR nova.compute.manager [] self.force_reraise() 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 3304 ERROR nova.compute.manager [] six.reraise(self.type_, self.value, self.tb) 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise 3304 ERROR nova.compute.manager [] raise value 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 138, in launch 3304 ERROR nova.compute.manager [] return self._domain.createWithFlags(flags) 3304 ERROR nova.compute.manager [] File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 190, in doit 3304 ERROR nova.compute.manager [] result = proxy_call(self._autowrap, f, *args, **kwargs) 3304 ERROR nova.compute.manager [] File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 148, in proxy_call 3304 ERROR nova.compute.manager [] rv = execute(f, *args, **kwargs) 3304 ERROR nova.compute.manager [] File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 129, in execute 3304 ERROR nova.compute.manager [] six.reraise(c, e, tb) 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise 3304 ERROR nova.compute.manager [] raise value 3304 ERROR nova.compute.manager [] File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 83, in tworker 3304 ERROR nova.compute.manager [] rv = meth(*args, **kwargs) 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/libvirt.py", line 1110, in createWithFlags 3304 ERROR nova.compute.manager [] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) 3304 ERROR nova.compute.manager [] libvirt.libvirtError: Cannot access storage file 'None': No such file or directory -- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison