<div dir="ltr"><div class="gmail_default" style="font-family:'courier new',monospace"><br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Feb 25, 2014 at 10:28 PM, yunling <span dir="ltr"><<a href="mailto:yunlingzeng@hotmail.com" target="_blank">yunlingzeng@hotmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><div dir="ltr"><p align="left" style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)"><span lang="EN-US" style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black">According to some further analysis, I tend to consider what I found out as a bug now...</span></p>
<p align="left" style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)"><span lang="EN-US" style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black"> </span></p>
<p align="left" style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)"><span lang="EN-US" style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black">The execution flow of attaching a volume to an instance can be decoupled as the following steps:</span></p>
<p align="left" style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)"><span lang="EN-US" style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black"> </span></p>
<p align="left" style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)"><span lang="EN-US" style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black">Step 1. Nova to Cinder: Volume reservation Volume status: attaching</span></p>
<p align="left" style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)"><span lang="EN-US" style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black">Step 2. Nova to Cinder: Connection initialization Volume status: attaching</span></p>
<p align="left" style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)"><span lang="EN-US" style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black">Step 3. Nova: Volume protocol checking Volume status: attaching</span></p>
<p align="left" style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)"><span lang="EN-US" style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black">Step 4. Nova connects to cinder-volume volume Volume status: attaching</span></p>
<p align="left" style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)"><span lang="EN-US" style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black">Step 5. Nova adds a volume to a VM via libvirt. The VM can access this volume from now on. </span></p>
<p align="left" style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)"><span lang="EN-US" style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black"> Volume status: attaching (<b>but we can do "force delete" now!!</b>)</span></p>
<p align="left" style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)"><span lang="EN-US" style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black">Step 6. Nova to Cinder: Volume attaching and volume status update Volume status: attached</span></p>
<p align="left" style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)"><span lang="EN-US" style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black"> </span></p>
<p align="left" style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)"><span lang="EN-US" style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black">To verify that, I tested Nova and Cinder by force deleting a volume being attached to an instance and in "attaching" status.</span></p>
<p align="left" style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)"><span lang="EN-US" style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black">After the volume is deleted, Disk IO errors just happened (I used "fdisk -l" to test it). A screen snapshot is attached below.</span></p>
<div style="display:inline-block"><span> </span><div style="display:inline-block"><img src="cid:inlineImage0" height="242" width="553"></div><span> </span></div><br><p align="left" style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)">
<span lang="EN-US" style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black"> </span></p><p align="left" style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)">
<span lang="EN-US" style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black">Moreover, disk info was still left as below:</span></p><div style="display:inline-block"><span> </span><div style="display:inline-block">
<img src="cid:inlineImage1" height="54" width="841"></div><span> </span></div><br><p align="left" style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)">
<span lang="EN-US" style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black">Therefore, I think a "force delete" operation to a volume in "attaching" status is really risky. We need to get it fixed.</span></p>
<p align="left" style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)"><span lang="EN-US" style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black"><br>
</span></p><p align="left" style="margin-bottom:1.35em"><font face="宋体"><span style="line-height:18.933332443237305px">Thanks!</span></font></p><div><hr>From: <a href="mailto:zhangyu11@huawei.com" target="_blank">zhangyu11@huawei.com</a><br>
To: <a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a><br>Date: Wed, 26 Feb 2014 00:56:17 +0000<br>Subject: Re: [openstack-dev] [Cinder] [Nova]Do you think volume force delete operation should not apply to the volume being used?<div>
<div class="h5"><br><br>
<div>
<p><span lang="EN-US" style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">If I understand your question correctly, the case you describe should be like the following:</span></p>
<p><span lang="EN-US" style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
<p><span lang="EN-US" style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Assume we have created both an instance and a volume, then we try to attach that volume to the instance.</span></p>
<p><span lang="EN-US" style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Before that operation is completed (the status of the volume is “attaching” now), for whatever reasons we decide to apply a “force delete” operation
on that volume.</span></p>
<p><span lang="EN-US" style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Then, after we applied that force delete, we come to see that, from the Cinder side, the volume has been successfully deleted and the status is
surely “deleted”.</span></p>
<p><span lang="EN-US" style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">However, from the Nova side, we see that the status of the deleted volume remains to be “attaching”.</span></p>
<p><span lang="EN-US" style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
<p><span lang="EN-US" style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">If this is truly your case, I think it is a bug. The reason might lie in that, Cinder forgets to refresh the attach_status attribute of a volume
in DB when applying a “force delete” operation.</span></p>
<p><span lang="EN-US" style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Is there any other suggestions?</span></p>
<p><span lang="EN-US" style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
<p><span lang="EN-US" style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Thanks!</span></p>
<p><span lang="EN-US" style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
<p><span lang="EN-US" style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
<p><span lang="EN-US" style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
<div>
<div style="border-style:solid none none;border-top-color:rgb(181,196,223);border-top-width:1pt;padding:3pt 0cm 0cm">
<p><b><span lang="EN-US" style="font-size:10pt;font-family:Tahoma,sans-serif">From:</span></b><span lang="EN-US" style="font-size:10pt;font-family:Tahoma,sans-serif"> yunling [mailto:<a href="mailto:yunlingzeng@hotmail.com" target="_blank">yunlingzeng@hotmail.com</a>]
<br>
<b>Sent:</b> Monday, February 17, 2014 9:14 PM<br>
<b>To:</b> <a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a><br>
<b>Subject:</b> [openstack-dev] [Cinder]Do you think volume force delete operation should not apply to the volume being used?</span></p>
</div>
</div>
<p><span lang="EN-US"> </span></p>
<div>
<div>
<p style="vertical-align:baseline"><span><span lang="EN-US" style="font-size:14pt;font-family:'Segoe UI',sans-serif">Hi stackers:</span></span><span><span lang="EN-US" style="font-size:14pt;font-family:'Segoe UI',sans-serif"> </span></span><span lang="EN-US" style="font-size:6pt"></span></p>
</div>
<div>
<p style="vertical-align:baseline;word-wrap:break-word"><span><span lang="EN-US" style="font-size:11pt"> </span></span><span lang="EN-US" style="font-size:11pt"><br>
<span> </span></span><span lang="EN-US" style="font-size:6pt"></span></p>
</div>
<div>
<p style="vertical-align:baseline;word-wrap:break-word"><span><span lang="EN-US" style="font-size:14pt;font-family:'Segoe UI',sans-serif">I found that volume status become inconsistent (nova volume status is attaching,
verus cinder volume status is deleted) between nova and cinder when doing volume force delete operation on an attaching volume.</span></span><span><span lang="EN-US" style="font-size:14pt;font-family:'Segoe UI',sans-serif"> </span></span><span lang="EN-US" style="font-size:6pt"></span></p>
</div>
<div>
<p style="vertical-align:baseline;word-wrap:break-word"><span><span lang="EN-US" style="font-size:14pt;font-family:'Segoe UI',sans-serif">I think volume force delete operation should not apply to the volume being used,
which included the attached status of attaching, attached and detached.</span></span><span><span lang="EN-US" style="font-size:14pt;font-family:'Segoe UI',sans-serif"> </span></span><span lang="EN-US" style="font-size:6pt"></span></p>
</div>
<div>
<p style="vertical-align:baseline;word-wrap:break-word"><span><span lang="EN-US" style="font-size:11pt"> </span></span><span lang="EN-US" style="font-size:11pt"><br>
<span> </span></span><span lang="EN-US" style="font-size:6pt"></span></p>
</div>
<div>
<p style="vertical-align:baseline;word-wrap:break-word"><span><span lang="EN-US" style="font-size:14pt;font-family:'Segoe UI',sans-serif">How do you think?</span></span><span><span lang="EN-US" style="font-size:14pt;font-family:'Segoe UI',sans-serif"> </span></span><span lang="EN-US" style="font-size:6pt"></span></p>
</div>
<div>
<p style="vertical-align:baseline;word-wrap:break-word"><span lang="EN-US" style="font-size:6pt"> </span></p>
<p style="vertical-align:baseline;word-wrap:break-word"><span lang="EN-US" style="font-size:6pt"> </span></p>
</div>
<div>
<p style="vertical-align:baseline;word-wrap:break-word"><span><span lang="EN-US" style="font-size:14pt;font-family:'Segoe UI',sans-serif">thanks</span></span><span lang="EN-US" style="font-size:6pt"></span></p>
</div>
</div>
</div>
<br></div></div>_______________________________________________
OpenStack-dev mailing list
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></div> </div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><div class="gmail_default" style="font-family:'courier new',monospace">The reason the attaching and detaching states are not on the black-list for force-delete is because those are states that historically were problematic. Things can go wrong on the nova attach/detach side when dealing with the bdm. The reality is that detach/attach on the cinder side is just a DB status update (and now in Icehouse a create/remove export). If you search the ML archives or the Cinder bug list for "stuck in attaching/detaching" you'll find why we implemented it this way to begin with. Here's just one example [1].</div>
<div class="gmail_default" style="font-family:'courier new',monospace"><br></div><div class="gmail_default" style="font-family:'courier new',monospace">I'd prefer we didn't remove the capability to force-delete attaching or detaching. If you take this away the only option left if something goes wrong external to Cinder is to use the change-state command (not terrible, but I'd rather call the force delete to make sure things like exports are removed).</div>
<div class="gmail_default" style="font-family:'courier new',monospace"><br></div><div class="gmail_default" style="font-family:'courier new',monospace">[1] <a href="https://lists.launchpad.net/openstack/msg13401.html">https://lists.launchpad.net/openstack/msg13401.html</a></div>
<div class="gmail_default" style="font-family:'courier new',monospace"><br></div><div class="gmail_default" style="font-family:'courier new',monospace">Thanks,</div><div class="gmail_default" style="font-family:'courier new',monospace">
John</div><br></div></div>