<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body dir="auto">
<div>Except I think the CAP theorem would say that u can't accurately give back there quota under thing like network partitions.</div>
<div><br>
</div>
<div>If nova-compute and the message queue have a network partition then u can release there quota but can't actually delete there vms. I would actually prefer to not release there quota, but then this should be a deployer decision and not a one size fits all
decision (IMHO).<br>
<br>
Sent from my really tiny device...</div>
<div><br>
On Oct 28, 2013, at 7:18 AM, "Day, Phil" <<a href="mailto:philip.day@hp.com">philip.day@hp.com</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
tt
{mso-style-priority:99;
font-family:"Courier New";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I’d disagree that that – from a user perspective they should always be able to delete an Instance regardless of its state, and the delete should always work
(or at least always appear to work to the user so that it no longer counts against their quota, and they are no longer charged for it)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Abhishek Lahiri [<a href="mailto:aviostack@gmail.com">mailto:aviostack@gmail.com</a>]
<br>
<b>Sent:</b> 26 October 2013 17:10<br>
<b>To:</b> OpenStack Development Mailing List<br>
<b>Cc:</b> OpenStack Development Mailing List<br>
<b>Subject:</b> Re: [openstack-dev] [nova] Thoughs please on how to address a problem with mutliple deletes leading to a nova-compute thread pool problem<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Deletes should only be allowed when the vm is in a power off state. This will allow consistent state transition.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Al<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
On Oct 26, 2013, at 8:55 AM, Joshua Harlow <<a href="mailto:harlowja@yahoo-inc.com">harlowja@yahoo-inc.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">I think I will try to have a unconference at the HK summit about ideas the cinder developers (and the taskflow developers, since it's not a concept that is unique /applicable to just cinder) are having about said state machine (and it's
potential usage).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">So look out for that, be interesting to have some nova folks involved there also :-)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
Sent from my really tiny device...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
On Oct 26, 2013, at 3:14 AM, "Alex Glikson" <<a href="mailto:GLIKSON@il.ibm.com">GLIKSON@il.ibm.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">+1</span>
<br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Regards,</span> <br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Alex</span> <br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""><br>
</span><br>
<tt><span style="font-size:10.0pt">Joshua Harlow <<a href="mailto:harlowja@yahoo-inc.com">harlowja@yahoo-inc.com</a>> wrote on 26/10/2013 09:29:03 AM:</span></tt><span style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>> </tt><br>
<tt>> An idea that others and I are having for a similar use case in </tt><br>
<tt>> cinder (or it appears to be similar).</tt><br>
<tt>> </tt><br>
<tt>> If there was a well defined state machine/s in nova with well </tt><br>
<tt>> defined and managed transitions between states then it seems like </tt><br>
<tt>> this state machine could resume on failure as well as be interrupted</tt><br>
<tt>> when a "dueling" or preemptable operation arrives (a delete while </tt><br>
<tt>> being created for example). This way not only would it be very clear</tt><br>
<tt>> the set of states and transitions but it would also be clear how </tt><br>
<tt>> preemption occurs (and under what cases). </tt><br>
<tt>> </tt><br>
<tt>> Right now in nova there is a distributed and ad-hoc state machine </tt><br>
<tt>> which if it was more formalized it could inherit some if the </tt><br>
<tt>> described useful capabilities. It would also be much more resilient </tt><br>
<tt>> to these types of locking problems that u described. </tt><br>
<tt>> </tt><br>
<tt>> IMHO that's the only way these types of problems will be fully be </tt><br>
<tt>> fixed, not by more queues or more periodic tasks, but by solidifying</tt><br>
<tt>> & formalizing the state machines that compose the work nova does.</tt><br>
<tt>> </tt><br>
<tt>> Sent from my really tiny device...</tt><br>
<tt>> </tt><br>
<tt>> > On Oct 25, 2013, at 3:52 AM, "Day, Phil" <<a href="mailto:philip.day@hp.com">philip.day@hp.com</a>> wrote:</tt><br>
<tt>> > </tt><br>
<tt>> > Hi Folks,</tt><br>
<tt>> > </tt><br>
<tt>> > We're very occasionally seeing problems where a thread processing </tt><br>
<tt>> a create hangs (and we've seen when taking to Cinder and Glance). </tt><br>
<tt>> Whilst those issues need to be hunted down in their own rights, they</tt><br>
<tt>> do show up what seems to me to be a weakness in the processing of </tt><br>
<tt>> delete requests that I'd like to get some feedback on.</tt><br>
<tt>> > </tt><br>
<tt>> > Delete is the one operation that is allowed regardless of the </tt><br>
<tt>> Instance state (since it's a one-way operation, and users should </tt><br>
<tt>> always be able to free up their quota). However when we get a </tt><br>
<tt>> create thread hung in one of these states, the delete requests when </tt><br>
<tt>> they hit the manager will also block as they are synchronized on the</tt><br>
<tt>> uuid. Because the user making the delete request doesn't see </tt><br>
<tt>> anything happen they tend to submit more delete requests. The </tt><br>
<tt>> Service is still up, so these go to the computer manager as well, </tt><br>
<tt>> and eventually all of the threads will be waiting for the lock, and </tt><br>
<tt>> the compute manager will stop consuming new messages.</tt><br>
<tt>> > </tt><br>
<tt>> > The problem isn't limited to deletes - although in most cases the </tt><br>
<tt>> change of state in the API means that you have to keep making </tt><br>
<tt>> different calls to get past the state checker logic to do it with an</tt><br>
<tt>> instance stuck in another state. Users also seem to be more </tt><br>
<tt>> impatient with deletes, as they are trying to free up quota for other things.
</tt><br>
<tt>> > </tt><br>
<tt>> > So while I know that we should never get a thread into a hung </tt><br>
<tt>> state into the first place, I was wondering about one of the </tt><br>
<tt>> following approaches to address just the delete case:</tt><br>
<tt>> > </tt><br>
<tt>> > i) Change the delete call on the manager so it doesn't wait for </tt><br>
<tt>> the uuid lock. Deletes should be coded so that they work regardless</tt><br>
<tt>> of the state of the VM, and other actions should be able to cope </tt><br>
<tt>> with a delete being performed from under them. There is of course </tt><br>
<tt>> no guarantee that the delete itself won't block as well. </tt><br>
<tt>> > </tt><br>
<tt>> > ii) Record in the API server that a delete has been started (maybe</tt><br>
<tt>> enough to use the task state being set to DELETEING in the API if </tt><br>
<tt>> we're sure this doesn't get cleared), and add a periodic task in the</tt><br>
<tt>> compute manager to check for and delete instances that are in a </tt><br>
<tt>> "DELETING" state for more than some timeout. Then the API, knowing </tt><br>
<tt>> that the delete will be processes eventually can just no-op any </tt><br>
<tt>> further delete requests.</tt><br>
<tt>> > </tt><br>
<tt>> > iii) Add some hook into the ServiceGroup API so that the timer </tt><br>
<tt>> could depend on getting a free thread from the compute manager pool </tt><br>
<tt>> (ie run some no-op task) - so that of there are no free threads then</tt><br>
<tt>> the service becomes down. That would (eventually) stop the scheduler</tt><br>
<tt>> from sending new requests to it, and make deleted be processed in </tt><br>
<tt>> the API server but won't of course help with commands for other </tt><br>
<tt>> instances on the same host.</tt><br>
<tt>> > </tt><br>
<tt>> > iv) Move away from having a general topic and thread pool for all </tt><br>
<tt>> requests, and start a listener on an instance specific topic for </tt><br>
<tt>> each running instance on a host (leaving the general topic and pool </tt><br>
<tt>> just for creates and other non-instance calls like the hypervisor </tt><br>
<tt>> API). Then a blocked task would only affect request for a specificinstance.</tt><br>
<tt>> > </tt><br>
<tt>> > I'm tending towards ii) as a simple and pragmatic solution in the </tt><br>
<tt>> near term, although I like both iii) and iv) as being both generally</tt><br>
<tt>> good enhancments - but iv) in particular feels like a pretty seismic change.</tt><br>
<tt>> > </tt><br>
<tt>> > Thoughts please,</tt><br>
<tt>> > </tt><br>
<tt>> > Phil </tt><br>
<tt>> > </tt><br>
<tt>> > _______________________________________________</tt><br>
<tt>> > OpenStack-dev mailing list</tt><br>
<tt>> > <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a></tt><br>
<tt>> > </tt></span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"><tt><span style="font-size:10.0pt">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</span></tt></a><span style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>> </tt><br>
<tt>> _______________________________________________</tt><br>
<tt>> OpenStack-dev mailing list</tt><br>
<tt>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a></tt><br>
<tt>> </tt></span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"><tt><span style="font-size:10.0pt">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</span></tt></a><span style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>> </tt></span><o:p></o:p></p>
</div>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</div>
</blockquote>
</div>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div><span>_______________________________________________</span><br>
<span>OpenStack-dev mailing list</span><br>
<span><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a></span><br>
<span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></span><br>
</div>
</blockquote>
</body>
</html>