<div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 15, 2016 at 5:59 PM, Preston L. Bannister <span dir="ltr"><<a href="mailto:preston@bannister.us" target="_blank">preston@bannister.us</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>QEMU has the ability to directly connect to iSCSI volumes. Running the iSCSI connections through the nova-compute host *seems* somewhat inefficient. </div></div></blockquote><div> </div><div><div class="gmail_default" style="font-family:monospace,monospace;display:inline">I know tests I've run in the past virt-io actually does a really good job here. Granted it's been a couple years since I've spent any time looking at this so really can't definitively say without looking again.</div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>There is a spec/blueprint and implementation that landed in Kilo:</div><div><br></div><div><a href="https://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/qemu-built-in-iscsi-initiator.html" target="_blank">https://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/qemu-built-in-iscsi-initiator.html</a><br></div><div><a href="https://blueprints.launchpad.net/nova/+spec/qemu-built-in-iscsi-initiator" target="_blank">https://blueprints.launchpad.net/nova/+spec/qemu-built-in-iscsi-initiator</a><br></div><div><br></div><div>From looking at the OpenStack Nova sources ... I am not entirely clear on when this behavior is invoked (just for Ceph?), and how it might change in future. </div></div></blockquote><div> </div><div><div class="gmail_default" style="font-family:monospace,monospace;display:inline">I actually hadn't seen that, glad you pointed it out :) I haven't tried configuring it but will try and do so and see what sort of differences in performance there are. One other thing to keep in mind (I could be mistaken, but...) last time I looked at this is wasn't vastly different from the model we use now. It's not actually using an iSCSI initiator on the Instance, it's still using an initiator on the compute node and passing the device in I believe. I'm sure somebody will correct me if I'm wrong here. </div></div><div><div class="gmail_default" style="font-family:monospace,monospace;display:inline"><br></div></div><div><div class="gmail_default" style="font-family:monospace,monospace;display:inline">I don't know what your reference to Ceph has to do with this here? This appears to be a Cinder iSCSI mechanism. You can see how to config in the commit message (<a href="https://review.openstack.org/#/c/135854/19">https://review.openstack.org/#/c/135854/19</a>, again I plan to try it out).</div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>Looking for a general sense where this is headed. (If anyone knows...)</div></div></blockquote><div> </div><div><div class="gmail_default" style="font-family:monospace,monospace;display:inline">Seems like you should be able to configure it and run it, assuming the work is actually done and hasn't broken while sitting.</div><div class="gmail_default" style="font-family:monospace,monospace;display:inline"></div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>If there is some problem with QEMU and directly attached iSCSI volumes, that would explain why this is not the default. Or is this simple inertia?</div></div></blockquote><div> </div><div><div class="gmail_default" style="font-family:monospace,monospace;display:inline">Virt-io is actually super flexible and lets us do all sorts of things with various connector types. I think you'd have to have some pretty compelling data to change the default here. Another thing to keep in mind, even if we just consider iSCSI and leave out FC and other protocols; one thing we absolutely wouldn't want is to give Instances direct access to the iSCSI network. This raises all sorts of security concerns for folks running public clouds. It also means more heavy weight Instances due to additional networking requirements, the iSCSI stack etc. More importantly, the last time I looked hot-plugging didn't work with this option, but again I admit it's been a long time since I've looked at it and my memory isn't always that great. </div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div><br></div><div>I have a concrete concern. I work for a company (EMC) that offers backup products, and we now have backup for instances in OpenStack. To make this efficient, we need to collect changed-block information from instances. </div></div></blockquote><div> </div><div><div class="gmail_default" style="font-family:monospace,monospace;display:inline">Ahh, ok, so you don't really have a "concrete concern" about using virt-io driver, or the way things work... or even any data that one performs better/worse than the other. What you do have apparently is a solution you'd like to integrate and sell with OpenStack. Fair enough, but we should probably be clear about the motivation until there's some data (there very well may be compelling reasons to change this).</div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>1) We could put an intercept in the Linux kernel of the nova-compute host to track writes at the block layer. This has the merit of working for containers, and potentially bare-metal instance deployments. But is not guaranteed for instances, if the iSCSI volumes are directly attached to QEMU.</div><div><br></div><div>2) We could use the QEMU support for incremental backup (first bit landed in QEMU 2.4). This has the merit of working with any storage, by only for virtual machines under QEMU.</div><div><br></div><div>As our customers are (so far) only asking about virtual machine backup. I long ago settled on (2) as most promising. </div><div><br></div><div>What I cannot clearly determine is where (1) will fail. Will all iSCSI volumes connected to QEMU instances eventually become directly connected? </div><div><br></div><div><br></div><div>Xiao's unanswered query (below) presents another question. Is this a site-choice? Could I require my customers to configure their OpenStack clouds to always route iSCSI connections through the nova-compute host? (I am not a fan of this approach, but I have to ask.)</div></div></blockquote><div><br></div><div><div class="gmail_default" style="font-family:monospace,monospace;display:inline">Certainly seems like you could. The question is would the distro in use support it? Also would it work with multi-backend configs. Honestly it sounds like there's a lot of data collection and analysis that you could do here and contribute back to the community. Perhaps Xiao or you should try it out?</div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>To answer Xiao's question, can a site configure their cloud to *always* directly connect iSCSI volumes to QEMU?</div><div><br></div><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Feb 16, 2016 at 4:54 AM, Xiao Ma (xima2) <span dir="ltr"><<a href="mailto:xima2@cisco.com" target="_blank">xima2@cisco.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div style="word-wrap:break-word">
<div>
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div>
<div>Hi, All</div>
<div><br>
</div>
<div>I want to make the qemu communicate with iscsi target using libiscsi directly, and I </div>
<div>followed <a href="https://review.openstack.org/#/c/135854/" target="_blank">https://review.openstack.org/#/c/135854/</a> to add </div>
<div><span style="background-color:rgb(255,255,255)">'</span><span style="background-color:rgb(255,255,255)"><font face="monospace" size="2"><span style="white-space:pre-wrap">volume_drivers
= iscsi=nova.virt.libvirt.volume.LibvirtNetVolumeDriver’</span></font> in </span>nova.conf</div>
<div> and then restarted nova services and cinder services, but still the volume configuration of vm is as bellow:</div>
<div><br>
</div>
<div>
<div> <disk type='block' device='disk'></div>
<div> <driver name='qemu' type='raw' cache='none'/></div>
<div> <source dev='/dev/disk/by-path/ip-10.75.195.205:3260-iscsi-iqn.2010-10.org.openstack:volume-076bb429-67fd-4c0c-9ddf-0dc7621a975a-lun-0'/></div>
<div> <target dev='vdb' bus='virtio'/></div>
<div> <serial>076bb429-67fd-4c0c-9ddf-0dc7621a975a</serial></div>
<div> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/></div>
<div> </disk></div>
<div><br>
</div>
<div><br>
</div>
<div>I use centos7 and Liberty version of OpenStack.</div>
<div>Could anybody tell me how can I achieve it?</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div>Thanks.</div></div></div></div></div></blockquote></div></div></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div>