<div dir="ltr"><div>The bp:<a href="https://blueprints.launchpad.net/nova/+spec/add-delete-on-termination-option">https://blueprints.launchpad.net/nova/+spec/add-delete-on-termination-option</a><br><br></div>is what's your qustion?<br>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">2014-02-26 14:01 GMT+08:00 John Griffith <span dir="ltr"><<a href="mailto:john.griffith@solidfire.com" target="_blank">john.griffith@solidfire.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:'courier new',monospace"><br></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Tue, Feb 25, 2014 at 10:28 PM, yunling <span dir="ltr"><<a href="mailto:yunlingzeng@hotmail.com" target="_blank">yunlingzeng@hotmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><div dir="ltr"><p style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)" align="left"><span style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black" lang="EN-US">According to some further analysis, I tend to consider what I found out as a bug now...</span></p>
<p style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)" align="left"><span style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black" lang="EN-US"> </span></p>
<p style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)" align="left"><span style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black" lang="EN-US">The execution flow of attaching a volume to an instance can be decoupled as the following steps:</span></p>
<p style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)" align="left"><span style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black" lang="EN-US"> </span></p>
<p style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)" align="left"><span style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black" lang="EN-US">Step 1. Nova to Cinder: Volume reservation Volume status: attaching</span></p>
<p style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)" align="left"><span style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black" lang="EN-US">Step 2. Nova to Cinder: Connection initialization Volume status: attaching</span></p>
<p style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)" align="left"><span style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black" lang="EN-US">Step 3. Nova: Volume protocol checking Volume status: attaching</span></p>
<p style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)" align="left"><span style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black" lang="EN-US">Step 4. Nova connects to cinder-volume volume Volume status: attaching</span></p>
<p style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)" align="left"><span style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black" lang="EN-US">Step 5. Nova adds a volume to a VM via libvirt. The VM can access this volume from now on. </span></p>
<p style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)" align="left"><span style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black" lang="EN-US"> Volume status: attaching (<b>but we can do "force delete" now!!</b>)</span></p>
<p style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)" align="left"><span style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black" lang="EN-US">Step 6. Nova to Cinder: Volume attaching and volume status update Volume status: attached</span></p>
<p style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)" align="left"><span style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black" lang="EN-US"> </span></p>
<p style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)" align="left"><span style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black" lang="EN-US">To verify that, I tested Nova and Cinder by force deleting a volume being attached to an instance and in "attaching" status.</span></p>
<p style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)" align="left"><span style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black" lang="EN-US">After the volume is deleted, Disk IO errors just happened (I used "fdisk -l" to test it). A screen snapshot is attached below.</span></p>
<div style="display:inline-block"><span> </span><div style="display:inline-block"><img src="cid:inlineImage0" height="242" width="553"></div><span> </span></div><br><p style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)" align="left">
<span style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black" lang="EN-US"> </span></p><p style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)" align="left">
<span style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black" lang="EN-US">Moreover, disk info was still left as below:</span></p><div style="display:inline-block"><span> </span><div style="display:inline-block">
<img src="cid:inlineImage1" height="54" width="841"></div><span> </span></div><br><p style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)" align="left">
<span style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black" lang="EN-US">Therefore, I think a "force delete" operation to a volume in "attaching" status is really risky. We need to get it fixed.</span></p>
<p style="line-height:19.880001068115234px;margin-bottom:1.35em;font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(68,68,68)" align="left"><span style="line-height:18.933332443237305px;font-size:10pt;font-family:宋体;color:black" lang="EN-US"><br>
</span></p><p style="margin-bottom:1.35em" align="left"><font face="宋体"><span style="line-height:18.933332443237305px">Thanks!</span></font></p><div><hr>From: <a href="mailto:zhangyu11@huawei.com" target="_blank">zhangyu11@huawei.com</a><br>
To: <a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a><br>Date: Wed, 26 Feb 2014 00:56:17 +0000<br>Subject: Re: [openstack-dev] [Cinder] [Nova]Do you think volume force delete operation should not apply to the volume being used?<div>
<div><br><br>
<div>
<p><span style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)" lang="EN-US">If I understand your question correctly, the case you describe should be like the following:</span></p>
<p><span style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)" lang="EN-US"> </span></p>
<p><span style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)" lang="EN-US">Assume we have created both an instance and a volume, then we try to attach that volume to the instance.</span></p>
<p><span style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)" lang="EN-US">Before that operation is completed (the status of the volume is “attaching” now), for whatever reasons we decide to apply a “force delete” operation
on that volume.</span></p>
<p><span style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)" lang="EN-US">Then, after we applied that force delete, we come to see that, from the Cinder side, the volume has been successfully deleted and the status is
surely “deleted”.</span></p>
<p><span style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)" lang="EN-US">However, from the Nova side, we see that the status of the deleted volume remains to be “attaching”.</span></p>
<p><span style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)" lang="EN-US"> </span></p>
<p><span style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)" lang="EN-US">If this is truly your case, I think it is a bug. The reason might lie in that, Cinder forgets to refresh the attach_status attribute of a volume
in DB when applying a “force delete” operation.</span></p>
<p><span style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)" lang="EN-US">Is there any other suggestions?</span></p>
<p><span style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)" lang="EN-US"> </span></p>
<p><span style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)" lang="EN-US">Thanks!</span></p>
<p><span style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)" lang="EN-US"> </span></p>
<p><span style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)" lang="EN-US"> </span></p>
<p><span style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)" lang="EN-US"> </span></p>
<div>
<div style="border-style:solid none none;border-top-color:rgb(181,196,223);border-top-width:1pt;padding:3pt 0cm 0cm">
<p><b><span style="font-size:10pt;font-family:Tahoma,sans-serif" lang="EN-US">From:</span></b><span style="font-size:10pt;font-family:Tahoma,sans-serif" lang="EN-US"> yunling [mailto:<a href="mailto:yunlingzeng@hotmail.com" target="_blank">yunlingzeng@hotmail.com</a>]
<br>
<b>Sent:</b> Monday, February 17, 2014 9:14 PM<br>
<b>To:</b> <a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a><br>
<b>Subject:</b> [openstack-dev] [Cinder]Do you think volume force delete operation should not apply to the volume being used?</span></p>
</div>
</div>
<p><span lang="EN-US"> </span></p>
<div>
<div>
<p style="vertical-align:baseline"><span><span style="font-size:14pt;font-family:'Segoe UI',sans-serif" lang="EN-US">Hi stackers:</span></span><span><span style="font-size:14pt;font-family:'Segoe UI',sans-serif" lang="EN-US"> </span></span><span style="font-size:6pt" lang="EN-US"></span></p>
</div>
<div>
<p style="vertical-align:baseline;word-wrap:break-word"><span><span style="font-size:11pt" lang="EN-US"> </span></span><span style="font-size:11pt" lang="EN-US"><br>
<span> </span></span><span style="font-size:6pt" lang="EN-US"></span></p>
</div>
<div>
<p style="vertical-align:baseline;word-wrap:break-word"><span><span style="font-size:14pt;font-family:'Segoe UI',sans-serif" lang="EN-US">I found that volume status become inconsistent (nova volume status is attaching,
verus cinder volume status is deleted) between nova and cinder when doing volume force delete operation on an attaching volume.</span></span><span><span style="font-size:14pt;font-family:'Segoe UI',sans-serif" lang="EN-US"> </span></span><span style="font-size:6pt" lang="EN-US"></span></p>
</div>
<div>
<p style="vertical-align:baseline;word-wrap:break-word"><span><span style="font-size:14pt;font-family:'Segoe UI',sans-serif" lang="EN-US">I think volume force delete operation should not apply to the volume being used,
which included the attached status of attaching, attached and detached.</span></span><span><span style="font-size:14pt;font-family:'Segoe UI',sans-serif" lang="EN-US"> </span></span><span style="font-size:6pt" lang="EN-US"></span></p>
</div>
<div>
<p style="vertical-align:baseline;word-wrap:break-word"><span><span style="font-size:11pt" lang="EN-US"> </span></span><span style="font-size:11pt" lang="EN-US"><br>
<span> </span></span><span style="font-size:6pt" lang="EN-US"></span></p>
</div>
<div>
<p style="vertical-align:baseline;word-wrap:break-word"><span><span style="font-size:14pt;font-family:'Segoe UI',sans-serif" lang="EN-US">How do you think?</span></span><span><span style="font-size:14pt;font-family:'Segoe UI',sans-serif" lang="EN-US"> </span></span><span style="font-size:6pt" lang="EN-US"></span></p>
</div>
<div>
<p style="vertical-align:baseline;word-wrap:break-word"><span style="font-size:6pt" lang="EN-US"> </span></p>
<p style="vertical-align:baseline;word-wrap:break-word"><span style="font-size:6pt" lang="EN-US"> </span></p>
</div>
<div>
<p style="vertical-align:baseline;word-wrap:break-word"><span><span style="font-size:14pt;font-family:'Segoe UI',sans-serif" lang="EN-US">thanks</span></span><span style="font-size:6pt" lang="EN-US"></span></p>
</div>
</div>
</div>
<br></div></div>_______________________________________________
OpenStack-dev mailing list
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></div> </div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><div class="gmail_default" style="font-family:'courier new',monospace">The reason the attaching and detaching states are not on the black-list for force-delete is because those are states that historically were problematic. Things can go wrong on the nova attach/detach side when dealing with the bdm. The reality is that detach/attach on the cinder side is just a DB status update (and now in Icehouse a create/remove export). If you search the ML archives or the Cinder bug list for "stuck in attaching/detaching" you'll find why we implemented it this way to begin with. Here's just one example [1].</div>
<div class="gmail_default" style="font-family:'courier new',monospace"><br></div><div class="gmail_default" style="font-family:'courier new',monospace">I'd prefer we didn't remove the capability to force-delete attaching or detaching. If you take this away the only option left if something goes wrong external to Cinder is to use the change-state command (not terrible, but I'd rather call the force delete to make sure things like exports are removed).</div>
<div class="gmail_default" style="font-family:'courier new',monospace"><br></div><div class="gmail_default" style="font-family:'courier new',monospace">[1] <a href="https://lists.launchpad.net/openstack/msg13401.html" target="_blank">https://lists.launchpad.net/openstack/msg13401.html</a></div>
<div class="gmail_default" style="font-family:'courier new',monospace"><br></div><div class="gmail_default" style="font-family:'courier new',monospace">Thanks,</div><div class="gmail_default" style="font-family:'courier new',monospace">
John</div><br></div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>