<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>
<div>I agree that we should embrace eventual consistency (under certain cases), but it begs the question of what are u eventually consistent on (maybe u shouldn't be eventually consistent on resource knowledge). U don't have to be eventually consistent on all
the things. </div>
<div><br>
</div>
<div>So lets assume we are always consistent about static-like resources and then you could offer a 'consistent' scheduler that has no races; u could offer a less-consistent scheduler if there was a pluggable way to do this. But then the question becomes how
does the 'consistent' scheduler reserve resources on a compute-node, before actually asking that compute-node to do the work required to fulfill the resource request, this is where I think the reservation process would be useful (of course it then also brings
along the question of what do u do about reservation timeouts and cleaning up inactive/unfulfilled reservations). Think of this as planning how to carve up a cake before u carve it up. Nova has enough knowledge (or should) to know what the cake currently looks
like (with-in reason, aka minus the dynamic eventually consistent resources) and therefore it should be able to know how to plan the cake carving, before actually doing the cake carving.</div>
<div><br>
</div>
<div>This is similar/the same issue (?) that cinder is dealing with with its work on having a defined state-machine (see: <a href="https://etherpad.openstack.org/p/CinderTaskFlowFSM">https://etherpad.openstack.org/p/CinderTaskFlowFSM</a>) and integrating with
taskflow to gain reliable workflows. Personally I prefer a slower (optimize it later) and reliable consistent scheduler & workflow that keeps my operations people sane over a eventually consistent one that has a higher chance of making them insane ;)</div>
<div><br>
</div>
<div>Anyway, that’s my current brain dump (and cake analogy, ha).</div>
<div><br>
</div>
<div>-Josh</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Joe Gordon <<a href="mailto:joe.gordon0@gmail.com">joe.gordon0@gmail.com</a>><br>
<span style="font-weight:bold">Date: </span>Monday, November 18, 2013 5:33 PM<br>
<span style="font-weight:bold">To: </span>Joshua Harlow <<a href="mailto:harlowja@yahoo-inc.com">harlowja@yahoo-inc.com</a>><br>
<span style="font-weight:bold">Cc: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [Nova] Does Nova really need an SQL database?<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Mon, Nov 18, 2013 at 4:47 PM, Joshua Harlow <span dir="ltr">
<<a href="mailto:harlowja@yahoo-inc.com" target="_blank">harlowja@yahoo-inc.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="font-size:14px;font-family:Calibri,sans-serif;word-wrap:break-word">
<div>An idea related to this, what would need to be done to make the DB have the exact state that a compute node is going through (and therefore the scheduler would not make unreliable/racey decisions, even when there are multiple schedulers). It's not like
we are dealing with a system which can not know the exact state (as long as the compute nodes are connected to the network, and a network partition does not occur).</div>
<div><br>
</div>
</div>
</blockquote>
<div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">Good question, I don't have a clear idea of the amount of work required to do this.</div>
<div class="im" style="font-family:arial,sans-serif;font-size:13px"></div>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="font-size:14px;font-family:Calibri,sans-serif;word-wrap:break-word">
<div></div>
<div>So maybe if we think about ways to correctly reserve resources, and keep up to date information about reserved resources we could then eliminate the race and eliminate the retries entirely?</div>
</div>
</blockquote>
<div><br>
</div>
<div>
<div style="font-family:arial,sans-serif;font-size:13px">What is the trade off here? What benefits do we get at what cost? I have a vague idea but just want to be explicit here. Also for 'cloudy' things we embrace the eventually consistent model, and I don't
think we should drop that.<span style="font-family:arial;font-size:small"> </span></div>
</div>
<div style="font-family:arial,sans-serif;font-size:13px"><span style="font-family:arial;font-size:small"><br>
</span></div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="font-size:14px;font-family:Calibri,sans-serif;word-wrap:break-word">
<div><br>
</div>
<span>
<div style="border-width:1pt medium medium;border-style:solid none none;padding:3pt 0in 0in;text-align:left;font-size:11pt;font-family:Calibri;border-top-color:rgb(181,196,223)">
<span style="font-weight:bold">From: </span>Joe Gordon <<a href="mailto:joe.gordon0@gmail.com" target="_blank">joe.gordon0@gmail.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Monday, November 18, 2013 3:32 PM
<div class="im"><br>
<span style="font-weight:bold">To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
</div>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [Nova] Does Nova really need an SQL database?<br>
</div>
<div>
<div class="h5">
<div><br>
</div>
<div>
<div>
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Mon, Nov 18, 2013 at 4:08 PM, yunhong jiang <span dir="ltr">
<<a href="mailto:yunhong.jiang@linux.intel.com" target="_blank">yunhong.jiang@linux.intel.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div>On Mon, 2013-11-18 at 14:09 -0800, Joe Gordon wrote:<br>
><br>
> Phil Day discussed this at the summit and I have finally gotten around<br>
> to posting a POC of this.<br>
><br>
> <a href="https://review.openstack.org/#/c/57053/" target="_blank">https://review.openstack.org/#/c/57053/</a><br>
<br>
</div>
Hi, Joe, why you think the DB is not exact state in your followed commit<br>
message? I think the DB is updated to date by resource tracker, am I<br>
right (the resource tracker get the underlying resource information<br>
periodically but I think that information is mostly static). And I think<br>
the scheduler retry mainly comes from the race condition of multiple<br>
scheduler instance.<br>
</blockquote>
<div><br>
</div>
<div><br>
</div>
<div>You answered the question yourself, the compute nodes (indirectly) update the DB periodically, so the further you are from the last periodic update the less up to date the DB is.</div>
<div><br>
</div>
<div>Its there for both reasons. But yes it was originally put there because of the multi scheduler race condition.</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
"We already have the concept that the DB isn't the exact state of the<br>
world, right now it's updated every 10 seconds. And we use the scheduler<br>
retry mechanism to handle cases where the scheduler was wrong. "<br>
<div>
<div><br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</span></div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</span>
</body>
</html>