[nova][cinder][linstor] VM creation failed with invalid libvirt xml <source dev='None'/>
Volodymyr Litovka
doka.ua at gmx.com
Thu May 21 11:24:27 UTC 2020
Hi colleagues,
does anybody here use Linstor as backend for Openstack? I can not launch VM (synopsis below) while everything looks
fine until the last step :-)
Few words about Openstack's storage configuration:
- controllers: cinder-api, cinder-scheduler, cinder-volume (modified with latest driver from Linstor's git), linstor-client, python-linstor
- compute nodes: linstor-controller, linstor-satellite, linstor-client, python-linstor; node types - Combined
- cinder.conf[1]
- volume-type registered with corresponding properties[2]
I'm able to manipulate volumes - create/delete[3] and every existing volume has the corresponding drbd/linstor repr[4] and LVM LV[5]
on the compute nodes, so, from storage point of view, it looks as correct and ready to use.
The problem is when I try to create VM using
$ openstack server create --flavor [...] --network [...] --volume 15c4fa98-76a5-44d6-9701-c89a8a848b82 fearst
Nova fails[6] because of invalid VM's libvirt XML definition (note "source dev='None'" below):
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source dev='None'/>
<target dev='vda' bus='virtio'/>
<serial>15c4fa98-76a5-44d6-9701-c89a8a848b82</serial>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
I do not have another working DRBD/Linstor/Openstack installation and can't even imagine, where to look into
the problem - whether it is in Nova, Cinder or in Linstor configuration itself.
I will appreciate if anybody can give me some guidance on where to seek for the solution to the problem.
In my lab I'm using Openstack Train on Ubuntu 18.04.4:
nova-compute 2:20.1.1-0ubuntu1~cloud0
nova-compute-kvm 2:20.1.1-0ubuntu1~cloud0
nova-compute-libvirt 2:20.1.1-0ubuntu1~cloud0
nova-api 2:20.1.1-0ubuntu1~cloud0
nova-common 2:20.1.1-0ubuntu1~cloud0
nova-conductor 2:20.1.1-0ubuntu1~cloud0
nova-novncproxy 2:20.1.1-0ubuntu1~cloud0
nova-scheduler 2:20.1.1-0ubuntu1~cloud0
cinder-api 2:15.1.0-0ubuntu1~cloud0
cinder-common 2:15.1.0-0ubuntu1~cloud0
cinder-scheduler 2:15.1.0-0ubuntu1~cloud0
cinder-volume 2:15.1.0-0ubuntu1~cloud0
python3-cinder 2:15.1.0-0ubuntu1~cloud0
python3-cinderclient 1:5.0.0-0ubuntu2~cloud0
python3-nova 2:20.1.1-0ubuntu1~cloud0
python3-novaclient 2:15.1.0-0ubuntu2~cloud0
Thank you!
====================================
[1] cinder.conf
[DEFAULT]
enabled_backends = linstor
default_volume_type = linstor
[linstor]
storage_availability_zone = nova
volume_backend_name = linstor
volume_driver = cinder.volume.drivers.linstordrv.LinstorDrbdDriver
linstor_autoplace_count = 2
linstor_default_volume_group_name=DfltRscGrp
linstor_default_uri=linstor://runner
linstor_default_storage_pool_name=drbdpool
linstor_default_resource_size=1
linstor_volume_downsize_factor=4096
where 'runner' is virtual ip of linstor-controller, protected by pacemaker.
====================================
[2] openstack volume type show linstor
+--------------------+--------------------------------------+
| Field | Value |
+--------------------+--------------------------------------+
| access_project_ids | None |
| description | None |
| id | d2025962-503a-4f37-93bd-b766bb346a42 |
| is_public | True |
| name | linstor |
| properties | volume_backend_name='linstor' |
| qos_specs_id | None |
+--------------------+--------------------------------------+
====================================
[3] openstack volume show fearst
+--------------------------------+-----------------------------------------------------+
| Field | Value |
+--------------------------------+-----------------------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2020-05-21T08:42:11.000000 |
| description | None |
| encrypted | False |
| id | 15c4fa98-76a5-44d6-9701-c89a8a848b82 |
| migration_status | None |
| multiattach | False |
| name | fearst |
| os-vol-host-attr:host | ctrl1 at linstor#linstor |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 7acec404393344cbabde07a22bbe6b3f |
| properties | |
| replication_status | None |
| size | 5 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| type | linstor |
| updated_at | 2020-05-21T08:42:18.000000 |
| user_id | 1048a19e94234451a61c3d9d46907ceb |
| volume_image_metadata | {'signature_verified': 'False', |
| | 'image_id': '15e992e8-a601-44cd-a2f0-8f34588a4d18',|
| | 'image_name': 'cirros', |
| | 'checksum': '1d3062cd89af34e419f7100277f38b2b', |
| | 'container_format': 'bare', |
| | 'disk_format': 'qcow2', |
| | 'min_disk': '0', |
| | 'min_ram': '0', |
| | 'size': '16338944'} |
+--------------------------------+-----------------------------------------------------+
====================================
[4] linstor r l
╭───────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊
╞═══════════════════════════════════════════════════════════════════════════════════╡
┊ CV_15c4fa98-76a5-44d6-9701-c89a8a848b82 ┊ cmp1 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊
┊ CV_15c4fa98-76a5-44d6-9701-c89a8a848b82 ┊ cmp2 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊
╰───────────────────────────────────────────────────────────────────────────────────╯
Note, that cmp3 is arbiter and has no 'by-disk' representation:
root at cmp1:~# ls -lR /dev/drbd*
brw-rw---- 1 root disk 147, 1000 May 21 11:42 /dev/drbd1000
/dev/drbd/by-disk/sds:
lrwxrwxrwx 1 root root 17 May 21 11:42 CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000 -> ../../../drbd1000
/dev/drbd/by-res/CV_15c4fa98-76a5-44d6-9701-c89a8a848b82:
lrwxrwxrwx 1 root root 17 May 21 11:42 0 -> ../../../drbd1000
root at cmp2:~# ls -lR /dev/drbd*
brw-rw---- 1 root disk 147, 1000 May 21 11:42 /dev/drbd1000
/dev/drbd/by-disk/sds:
lrwxrwxrwx 1 root root 17 May 21 11:42 CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000 -> ../../../drbd1000
/dev/drbd/by-res/CV_15c4fa98-76a5-44d6-9701-c89a8a848b82:
lrwxrwxrwx 1 root root 17 May 21 11:42 0 -> ../../../drbd1000
root at cmp3:~# ls -lR /dev/drbd*
brw-rw---- 1 root disk 147, 1000 May 21 11:42 /dev/drbd1000
/dev/drbd/by-res/CV_15c4fa98-76a5-44d6-9701-c89a8a848b82:
lrwxrwxrwx 1 root root 17 May 21 11:42 0 -> ../../../drbd1000
====================================
[5] root at cmp1:~# lvdisplay
[ ... ]
--- Logical volume ---
LV Path /dev/sds/CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000
LV Name CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000
VG Name sds
LV UUID jGzcIQ-yRpH-RKxP-ExsH-JSn6-kTFn-8kpKYO
LV Write Access read/write
LV Creation host, time cmp1, 2020-05-21 11:42:14 +0300
LV Pool name thin
LV Status available
# open 2
LV Size 5.00 GiB
Mapped size 0.02%
Current LE 1280
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:4
root at cmp2:~# lvdisplay
[ ... ]
--- Logical volume ---
LV Path /dev/sds/CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000
LV Name CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000
VG Name sds
LV UUID 8pVwMo-e3ed-Sy0w-06IZ-M7Mh-YVIL-BtKZEG
LV Write Access read/write
LV Creation host, time cmp2, 2020-05-21 11:42:14 +0300
LV Pool name thin
LV Status available
# open 2
LV Size 5.00 GiB
Mapped size 0.02%
Current LE 1280
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:4
====================================
[6] Traceback of nova-compute:
3304 ERROR nova.compute.manager [] Traceback (most recent call last):
3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2419, in _build_and_run_instance
3304 ERROR nova.compute.manager [] block_device_info=block_device_info)
3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3474, in spawn
3304 ERROR nova.compute.manager [] power_on=power_on)
3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6261, in _create_domain_and_network
3304 ERROR nova.compute.manager [] destroy_disks_on_failure)
3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
3304 ERROR nova.compute.manager [] self.force_reraise()
3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
3304 ERROR nova.compute.manager [] six.reraise(self.type_, self.value, self.tb)
3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
3304 ERROR nova.compute.manager [] raise value
3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6230, in _create_domain_and_network
3304 ERROR nova.compute.manager [] post_xml_callback=post_xml_callback)
3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6164, in _create_domain
3304 ERROR nova.compute.manager [] guest.launch(pause=pause)
3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 143, in launch
3304 ERROR nova.compute.manager [] self._encoded_xml, errors='ignore')
3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
3304 ERROR nova.compute.manager [] self.force_reraise()
3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
3304 ERROR nova.compute.manager [] six.reraise(self.type_, self.value, self.tb)
3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
3304 ERROR nova.compute.manager [] raise value
3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 138, in launch
3304 ERROR nova.compute.manager [] return self._domain.createWithFlags(flags)
3304 ERROR nova.compute.manager [] File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 190, in doit
3304 ERROR nova.compute.manager [] result = proxy_call(self._autowrap, f, *args, **kwargs)
3304 ERROR nova.compute.manager [] File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 148, in proxy_call
3304 ERROR nova.compute.manager [] rv = execute(f, *args, **kwargs)
3304 ERROR nova.compute.manager [] File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 129, in execute
3304 ERROR nova.compute.manager [] six.reraise(c, e, tb)
3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
3304 ERROR nova.compute.manager [] raise value
3304 ERROR nova.compute.manager [] File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 83, in tworker
3304 ERROR nova.compute.manager [] rv = meth(*args, **kwargs)
3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/libvirt.py", line 1110, in createWithFlags
3304 ERROR nova.compute.manager [] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
3304 ERROR nova.compute.manager [] libvirt.libvirtError: Cannot access storage file 'None': No such file or directory
--
Volodymyr Litovka
"Vision without Execution is Hallucination." -- Thomas Edison
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20200521/09418f9e/attachment-0001.html>
More information about the openstack-discuss
mailing list