Open Stack

>> Does QEMU support hardware initiators? iSER?
> No, this is only for case where you're doing pure software based
> iSCSI client connections. If we're relying on local hardware that's
> a different story.
>
>> We regularly fix issues with iSCSI attaches in the release cycles of
>> OpenStack,
>> because it's all done in python using existing linux packages.  How often
> This is a great example of the benefit that in-QEMU client gives us. The
> Linux iSCSI client tools have proved very unreliable in use by OpenStack.
> This is a reflection of the very architectural approach. We have individual
> resources needed by distinct VMs, but we're having to manage them as a host
> wide resource and that's creating us unneccessary complexity and having a
> poor effect on our reliability overall.
I've been doing more and more digging and research into doing this
and it seems that canonical removed libiscsi from qemu due to security 
problems
in the 14.04 LTS release cycle.

Trying to fire up a new vm manually with qemu attaching an iscsi disk via
the documented mechanism ends up with qemu complaining that it can't
open the disk 'unknown protocol'.

qemu-system-x86_64 -drive 
file=iscsi://10.52.1.11/iqn.2000-05.com.3pardata:20810002ac00383d/0 
-iscsi initiator-name=iqn.walt-qemu-initiator
qemu-system-x86_64: -drive 
file=iscsi://10.52.1.11/iqn.2000-05.com.3pardata:20810002ac00383d/0: 
could not open disk image 
iscsi://10.52.1.11/iqn.2000-05.com.3pardata:20810002ac00383d/0: Unknown 
protocol

There was bug filed against qemu back in 2014 and was marked as wont fix 
due to security issues.
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1271573

That looks like it has been fixed since here:
https://bugs.launchpad.net/ubuntu/+source/libiscsi/+bug/1271653
But that's only xenial (16.04) support and won't be in 14.x tree.


I have also confirmed that the 
nova.virt.libvirt.volume.net.LibvirtNetVolumeDriver
fails for iscsi for the same exact reason against nova master.

I modified the nova/virt/libvirt/driver.py and changed iscsi to point to 
the LibvirtNetVolumeDriver
and tried to attach an iSCSI volume.  It failed and the libvirtd log 
showed the unknown protocol error.

The n-cpu.log entry:
2016-06-24 08:09:21.555 8891 DEBUG nova.virt.libvirt.guest 
[req-46954106-c728-43ba-b40a-5b91a1639610 admin admin] attach device 
xml: <disk type="network" device="disk">
   <driver name="qemu" type="raw" cache="none"/>
   <source protocol="iscsi" 
name="iqn.2000-05.com.3pardata:20810002ac00383d/0">
     <host name="10.52.1.11" port="3260"/>
   </source>
   <target bus="virtio" dev="vdb"/>
   <serial>a1d0c85e-d6e6-424f-9ca7-76ecd0ce45fb</serial>
</disk>
  attach_device /opt/stack/nova/nova/virt/libvirt/guest.py:251
2016-06-24 08:09:21.574 8891 ERROR nova.virt.libvirt.driver 
[req-46954106-c728-43ba-b40a-5b91a1639610 admin admin] [instance: 
74092b75-dc20-47e5-9127-c63367d05b29] Failed to attach volume at 
mountpoint: /dev/vdb
2016-06-24 08:09:21.574 8891 ERROR nova.virt.libvirt.driver [instance: 
74092b75-dc20-47e5-9127-c63367d05b29] Traceback (most recent call last):
2016-06-24 08:09:21.574 8891 ERROR nova.virt.libvirt.driver [instance: 
74092b75-dc20-47e5-9127-c63367d05b29]   File 
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 1160, in attach_volume
2016-06-24 08:09:21.574 8891 ERROR nova.virt.libvirt.driver [instance: 
74092b75-dc20-47e5-9127-c63367d05b29] guest.attach_device(conf, 
persistent=True, live=live)
2016-06-24 08:09:21.574 8891 ERROR nova.virt.libvirt.driver [instance: 
74092b75-dc20-47e5-9127-c63367d05b29]   File 
"/opt/stack/nova/nova/virt/libvirt/guest.py", line 252, in attach_device
2016-06-24 08:09:21.574 8891 ERROR nova.virt.libvirt.driver [instance: 
74092b75-dc20-47e5-9127-c63367d05b29] 
self._domain.attachDeviceFlags(device_xml, flags=flags)
2016-06-24 08:09:21.574 8891 ERROR nova.virt.libvirt.driver [instance: 
74092b75-dc20-47e5-9127-c63367d05b29]   File 
"/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in 
doit
2016-06-24 08:09:21.574 8891 ERROR nova.virt.libvirt.driver [instance: 
74092b75-dc20-47e5-9127-c63367d05b29]     result = 
proxy_call(self._autowrap, f, *args, **kwargs)
2016-06-24 08:09:21.574 8891 ERROR nova.virt.libvirt.driver [instance: 
74092b75-dc20-47e5-9127-c63367d05b29]   File 
"/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in 
proxy_call
2016-06-24 08:09:21.574 8891 ERROR nova.virt.libvirt.driver [instance: 
74092b75-dc20-47e5-9127-c63367d05b29]     rv = execute(f, *args, **kwargs)
2016-06-24 08:09:21.574 8891 ERROR nova.virt.libvirt.driver [instance: 
74092b75-dc20-47e5-9127-c63367d05b29]   File 
"/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in 
execute
2016-06-24 08:09:21.574 8891 ERROR nova.virt.libvirt.driver [instance: 
74092b75-dc20-47e5-9127-c63367d05b29]     six.reraise(c, e, tb)
2016-06-24 08:09:21.574 8891 ERROR nova.virt.libvirt.driver [instance: 
74092b75-dc20-47e5-9127-c63367d05b29]   File 
"/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in 
tworker
2016-06-24 08:09:21.574 8891 ERROR nova.virt.libvirt.driver [instance: 
74092b75-dc20-47e5-9127-c63367d05b29]     rv = meth(*args, **kwargs)
2016-06-24 08:09:21.574 8891 ERROR nova.virt.libvirt.driver [instance: 
74092b75-dc20-47e5-9127-c63367d05b29]   File 
"/usr/local/lib/python2.7/dist-packages/libvirt.py", line 517, in 
attachDeviceFlags
2016-06-24 08:09:21.574 8891 ERROR nova.virt.libvirt.driver [instance: 
74092b75-dc20-47e5-9127-c63367d05b29]     if ret == -1: raise 
libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2016-06-24 08:09:21.574 8891 ERROR nova.virt.libvirt.driver [instance: 
74092b75-dc20-47e5-9127-c63367d05b29] libvirtError: operation failed: 
open disk image file failed


The */var/log/libvirtd.log* entry....

2016-06-24 15:09:21.572+0000: 21000: debug : qemuMonitorIOProcess:396 : 
QEMU_MONITOR_IO_PROCESS: mon=0x7fd4f000c920 buf={"return": "could not 
open disk image 
iscsi://10.52.1.11:3260/iqn.2000-05.com.3pardata%3A20810002ac00383d/0: 
Unknown protocol\r\n", "id": "libvirt-18"}^M
  len=153




So the argument that linux iSCSI client tools have proven unreliable 
also holds true for libiscsi.
This really isn't a win.



As a side note here:
   I am working on a performance report that I'm working on to test
the performance of bare metal iscsi vs. host passed through to virsh 
(like we do in Nova today)
and qemu iscsi built in support.  my preliminary results show that 
libiscsi and host attach passed
through to virsh run about the same relative io performance, but both 
are about 50% iSCSI to bare metal.

Walt

>> are QEMU
>> releases done and upgraded on customer deployments vs. python packages
>> (os-brick)?
> We're removing the entire layer of instability by removing the need to
> deal with any command line tools, and thus greatly simplifying our
> setup on compute nodes. No matter what we might do in os-brick it'll
> never give us a simple or reliable system - we're just papering over
> the flaws by doing stuff like blindly re-trying iscsi commands upon
> failure.
>
> Regards,
> Daniel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160624/a203389b/attachment.html>