<div dir="ltr">Yuk. So looking back in the nova libvirt volume code, it looks like nova is ignoring almost all of the device_info coming back from os-brick's connect_volume() call.<div>os-brick has the scsi id here:</div><div><a href="https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/iscsi.py#L552">https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/iscsi.py#L552</a><br></div><div><br></div><div>My guess is though, that nova uses the cinder UUID, in the domain xml so it can use it later at some point? </div><div><br></div><div>Walt</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 31, 2018 at 9:36 AM, Lucian Petrut <span dir="ltr"><<a href="mailto:lpetrut@cloudbasesolutions.com" target="_blank">lpetrut@cloudbasesolutions.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div>Actually, when using the libvirt driver, the SCSI id returned by os-brick is not exposed to the guest. The reason is that Nova explicitly sets the volume id as "serial" id in the guest disk configuration. Qemu will expose this to the guest, but with a
20 character limit.</div>
<div><br>
</div>
<div>For what is worth, Kubernetes as well as some guides rely on this behaviour.</div>
<div><br>
</div>
<div>For example:</div>
<div><br>
</div>
<div>nova volume-attach e03303e1-c20b-441c-b94a-<wbr>724cb2469487 10780b60-ad70-479f-a612-<wbr>14d03b1cc64d</div>
<div>virsh dumpxml `nova show cirros | grep instance_name | cut -d "|" -f 3`</div>
<div><br>
</div>
<div><domain type='qemu' id='10'></div>
<div> <name>instance-0000013d</name></div>
<div> <uuid>e03303e1-c20b-441c-b94a-<wbr>724cb2469487</uuid></div>
<div></div>
<div>....</div>
<div><disk type='block' device='disk'></div>
<div> <driver name='qemu' type='raw' cache='none' io='native'/></div>
<div> <source dev='/dev/sdb'/></div>
<div> <backingStore/></div>
<div> <target dev='vdb' bus='virtio'/></div>
<div><b> <serial>10780b60-ad70-<wbr>479f-a612-14d03b1cc64d</<wbr>serial></b></div>
<div> <alias name='virtio-disk1'/></div>
<div> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/></div>
<div> </disk></div>
<div><br>
</div>
<div>nova log:</div>
<div>
<div>Jan 31 15:39:54 ubuntu nova-compute[46142]: DEBUG os_brick.initiator.connectors.<wbr>iscsi [None req-d0c62440-133c-4e89-8798-<wbr>20278ca50f00 admin admin] <== connect_volume: return (2578ms) {'path': u'/dev/sdb', 'scsi_wwn': u'<wbr>360000000000000000e00000000010<wbr>001',
'type': u'block'} {{(pid=46142) trace_logging_wrapper /usr/local/lib/python2.7/dist-<wbr>packages/os_brick/utils.py:<wbr>170}}</div>
<div>Jan 31 15:39:54 ubuntu nova-compute[46142]: DEBUG nova.virt.libvirt.volume.iscsi [None req-d0c62440-133c-4e89-8798-<wbr>20278ca50f00 admin admin] Attached iSCSI volume {'path': u'/dev/sdb', 'scsi_wwn': '<wbr>360000000000000000e00000000010<wbr>001', 'type': 'block'} {{(pid=46142)
connect_volume /opt/stack/nova/nova/virt/<wbr>libvirt/volume/iscsi.py:65}}</div>
<div>Jan 31 15:39:54 ubuntu nova-compute[46142]: DEBUG nova.virt.libvirt.guest [None req-d0c62440-133c-4e89-8798-<wbr>20278ca50f00 admin admin] attach device xml: <disk type="block" device="disk"></div>
<div>Jan 31 15:39:54 ubuntu nova-compute[46142]: <driver name="qemu" type="raw" cache="none" io="native"/></div>
<div>Jan 31 15:39:54 ubuntu nova-compute[46142]: <source dev="/dev/sdb"/></div>
<div>Jan 31 15:39:54 ubuntu nova-compute[46142]: <target bus="virtio" dev="vdb"/></div>
<div>Jan 31 15:39:54 ubuntu nova-compute[46142]: <<wbr>serial>10780b60-ad70-479f-<wbr>a612-14d03b1cc64d</serial></div>
<div>Jan 31 15:39:54 ubuntu nova-compute[46142]: </disk></div>
<div>Jan 31 15:39:54 ubuntu nova-compute[46142]: {{(pid=<wbr>46142) attach_device /opt/stack/nova/nova/virt/<wbr>libvirt/guest.py:302}}</div>
</div>
<div><br>
</div>
<div>Regards,</div>
<div>Lucian Petrut</div><div><div class="h5">
<div><br>
</div>
<div>On Wed, 2018-01-31 at 07:59 -0500, Walter Boring wrote:</div>
<blockquote type="cite" style="margin:0 0 0 .8ex;border-left:2px #729fcf solid;padding-left:1ex">
<div dir="ltr">First off, the id's you are showing there are Cinder uuid's to identify the volumes in the cinder DB and are used for cinder based actions. The Ids that are seen and used by the system for discovery and passing to qemu are the disk SCSI ids,
which are embedded in the volume's themselves. os-brick returns the SCSI id to nova for use in attaching and it's not limited to the 20 characters.
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Jan 16, 2018 at 4:19 AM, Yikun Jiang <span dir="ltr">
<<a href="mailto:yikunkero@gmail.com" target="_blank">yikunkero@gmail.com</a>></span> wrote:<br>
<blockquote type="cite" style="margin:0 0 0 .8ex;border-left:2px #729fcf solid;padding-left:1ex">
<div dir="ltr">Some detail steps as below:
<div>1. First, We have 2 volumes with same part-uuid prefix.</div>
<div><img src="cid:1517409417.4421.76.camel@cloudbasesolutions.com" alt="内嵌图片 1" width="562" height="71"></div>
<div><br>
</div>
<div>volume(yikun2) is attached to server(test)</div>
<div><br>
</div>
<div>2. In GuestOS(Cent OS 7), take a look at by path and by id:</div>
<div><img src="cid:1517409417.4421.75.camel@cloudbasesolutions.com" alt="内嵌图片 2" width="562" height="134"></div>
<div>we found both by-path and by-id vdb links was generated successfully.</div>
<div><br>
</div>
<div>3. attach volume(yikun2_1) to server(test)</div>
<div><img src="cid:1517409417.4421.74.camel@cloudbasesolutions.com" alt="内嵌图片 4" style="margin-right:0px"><br>
</div>
<div><br>
</div>
<div>4. In GuestOS(Cent OS 7), take a look at by path and by id:</div>
<div><br>
</div>
<div><img src="cid:1517409417.4421.73.camel@cloudbasesolutions.com" alt="内嵌图片 6" width="562" height="126"><br>
</div>
<div><br>
</div>
<div>by-path soft link was generated successfully, but <span style="font-size:14px">by-id link was failed to generate.</span></div>
<div><span style="font-size:14px"><b>That is, in this case, if a user find the device by by-id, it would be failed to find it or find a wrong device.</b></span></div>
<div><span style="font-size:14px"><br>
</span></div>
<div><span style="font-size:14px">one of the user cases was happened on k8s device finding, more info you can see the ref as below:</span><br>
</div>
<div><a href="https://github.com/kubernetes/kubernetes/blob/53a8ac753bf468eaf6bcb5a07e34a0a67480df43/pkg/cloudprovider/providers/openstack/openstack_volumes.go#L463" target="_blank">https://github.com/kubernetes/<wbr>kubernetes/blob/53a8ac753bf468<wbr>eaf6bcb5a07e34a0a67480df43/<wbr>pkg/cloudprovider/providers/<wbr>openstack/openstack_volumes.<wbr>go#L463</a><br>
</div>
<div><br>
</div>
<div>So, I think by-id is NOT a good way to find the device, but what the best practice is? let's see other idea.</div>
<div><br>
</div>
<div>
<div class="gmail_extra">
<div>
<div class="m_-8000387952010212589m_7923193885501554668gmail_signature">
<div>Regards,</div>
<div>Yikun</div>
<div><br>
</div>
<div><font color="#666666">------------------------------<wbr>----------</font></div>
<div><font color="#666666">Jiang Yikun（</font><span style="color:rgb(102,102,102)">Kero）</span></div>
<div><font color="#666666">Mail: <a href="mailto:yikunkero@gmail.com" target="_blank">
yikunkero@gmail.com</a></font></div>
</div>
</div>
<br>
<div class="gmail_quote">
<div>
<div class="m_-8000387952010212589h5">2018-01-16 14:36 GMT+08:00 Zhenyu Zheng <span dir="ltr"><<a href="mailto:zhengzhenyulixi@gmail.com" target="_blank">zhengzhenyulixi@gmail.com</a>></span>:<br>
</div>
</div>
<blockquote type="cite" style="margin:0 0 0 .8ex;border-left:2px #729fcf solid;padding-left:1ex">
<div>
<div class="m_-8000387952010212589h5">
<div dir="ltr">Ops, forgot references:
<div>[1] <a href="https://github.com/torvalds/linux/blob/1cc15701cd89b0ce695bbc5cff3a2bf3e2efd25f/include/uapi/linux/virtio_blk.h#L54" style="margin:0px;padding:0px;white-space:pre-wrap;font-family:"Helvetica Neue",Arial,sans-serif;font-size:12px" target="_blank">https://github.com/torvald<wbr>s/linux/blob/1cc15701cd89b0ce6<wbr>95bbc5cff3a2bf3e2efd25f/includ<wbr>e/uapi/linux/virtio_blk.h#L54</a></div>
<div>[2] <a href="https://github.com/torvalds/linux/blob/1cc15701cd89b0ce695bbc5cff3a2bf3e2efd25f/drivers/block/virtio_blk.c#L363" style="margin:0px;padding:0px;white-space:pre-wrap;font-family:"Helvetica Neue",Arial,sans-serif;font-size:12px" target="_blank">https://github.com/torvald<wbr>s/linux/blob/1cc15701cd89b0ce6<wbr>95bbc5cff3a2bf3e2efd25f/driver<wbr>s/block/virtio_blk.c#L363</a></div>
</div>
<div class="m_-8000387952010212589m_7923193885501554668gmail-HOEnZb">
<div class="m_-8000387952010212589m_7923193885501554668gmail-h5">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Jan 16, 2018 at 2:35 PM, Zhenyu Zheng <span dir="ltr">
<<a href="mailto:zhengzhenyulixi@gmail.com" target="_blank">zhengzhenyulixi@gmail.com</a>></span> wrote:<br>
<blockquote type="cite" style="margin:0 0 0 .8ex;border-left:2px #729fcf solid;padding-left:1ex">
<div dir="ltr">Hi,
<div><br>
</div>
<div>I meet a problem like this recently:</div>
<div><br>
</div>
<div>When attaching a volume to an instance, in the xml, the disk is described as:</div>
<div><br>
</div>
<div><img src="cid:1517409417.4421.72.camel@cloudbasesolutions.com" alt="Inline image 1" width="544" height="133"><br>
</div>
<div>where the serial number here is the volume uuid in Cinder. While inside the vm:</div>
<div>in /dev/disk/by-id, there is a link for /vdb with the name of "virtio"+truncated serial number:</div>
<div><br>
</div>
<div><img src="cid:1517409417.4421.71.camel@cloudbasesolutions.com" alt="Inline image 2" width="544" height="49"><br>
</div>
<div><br>
</div>
<div>and according to <a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/2/html/Getting_Started_Guide/ch16s03.html" target="_blank">https://access.redhat.com/d<wbr>ocumentation/en-US/Red_Hat_Ent<wbr>erprise_Linux_OpenStack_Platfo<wbr>rm/2/html/Getting_Started_Guid<wbr>e/ch16s03.html</a></div>
<div><br>
</div>
<div>it seems that we will use this mount the volume.</div>
<div><br>
</div>
<div>The truncate seems to be happen in here [1][2] which is 20 digits.</div>
<div><br>
</div>
<div><b>My question here is: </b>if two volume have the identical first 20 digits in their uuids, it seems that the latter attached one will overwrite the first one's link:</div>
<div><img src="cid:1517409417.4421.70.camel@cloudbasesolutions.com" alt="Inline image 3" width="544" height="343"><br>
</div>
<div>(the above graph is snapshot for an volume backed instance, the virtio-15exxxxx was point to vda before, the by-path seems correct though)</div>
<div><br>
</div>
<div>It is rare to have the identical first 20 digits of two uuids, but possible, so what was the consideration of truncate only 20 digits of the volume uuid instead of use full 32?</div>
<div><br>
</div>
<div>BR,</div>
<div><br>
</div>
<div>Kevin Zheng</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
</div>
</div>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">
OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">
OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<pre>______________________________<wbr>______________________________<wbr>______________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org</a>?subject:<wbr>unsubscribe
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a>
</pre>
</blockquote>
</div></div></div>
<br>______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
<br></blockquote></div><br></div>