<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:618070177;
mso-list-template-ids:-1969183418;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:#1F497D">On Wed, 17 February 2016, Sylvain Bauza wrote</span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">Le 17/02/2016 12:59, Chris Dent a écrit :<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:.5in">On Wed, 17 Feb 2016, Cheng, Yingxin wrote:
<br>
<br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:.5in">To better illustrate the differences between shared-state, resource-
<br>
provider and legacy scheduler, I've drew 3 simplified pictures [1] in <br>
emphasizing the location of resource view, the location of claim and <br>
resource consumption, and the resource update/refresh pattern in three <br>
kinds of schedulers. Hoping I'm correct in the "resource-provider <br>
scheduler" part. <o:p></o:p></p>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<br>
That's a useful visual aid, thank you. It aligns pretty well with my <br>
understanding of each idea. <br>
<br>
A thing that may be missing, which may help in exploring the usefulness <br>
of each idea, is a representation of resources which are separate <br>
from compute nodes and shared by them, such as shared disk or pools <br>
of network addresses. In addition some would argue that we need to <br>
see bare-metal nodes for a complete picture. <br>
<br>
One of the driving motivations of the resource-provider work is to <br>
make it possible to adequately and accurately track and consume the <br>
shared resources. The legacy scheduler currently fails to do that <br>
well. As you correctly points out it does this by having "strict <br>
centralized consistency" as a design goal. <o:p></o:p></p>
</blockquote>
<p class="MsoNormal" style="margin-left:.5in"><br>
So, to be clear, I'm really happy to see the resource-providers series for many reasons :<br>
- it will help us getting a nice Facade for getting the resources and attributing them<br>
- it will help a shared-storage deployment by making sure that we don't have some resource problems when the resource is shared<br>
- it will create a possibility for external resource providers to provide some resource types to Nova so the Nova scheduler could use them (like Neutron related resources)<br>
<br>
That, I really want to have it implemented in Mitaka and Newton and I'm totally on-board and supporting it.<br>
<br>
TBC, the only problem I see with the series is [2], not the whole, please.<br>
<br>
<span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">@cdent:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">As far as I know, some resources are defined “shared”, simply because they are not the resources of compute node service. In other words, the compute node resource
tracker does not have the authority of those “shared” resources. For example, the “shared” storage resources are actually managed by the storage service, and the “shared” network resource “IP pool” is actually owned by network service. If all the resources
are labeled “shared” only because they are not owned by compute node services, the shared-resource-tracking/consumption problem can be solved by implementing resource trackers in all the authorized services. Those resource trackers are constantly providing
incremental updates to schedulers, and have the responsibilities to reserve and consume resources independently/distributedly, no matter where they are from, compute service, storage service or network service etc.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:.5in">As can be seen in the illustrations [1], the main compatibility issue
<br>
between shared-state and resource-provider scheduler is caused by the <br>
different location of claim/consumption and the assumed consistent <br>
resource view. IMO unless the claims are allowed to happen in both <br>
places(resource tracker and resource-provider db), it seems difficult <br>
to make shared-state and resource-provider scheduler work together. <o:p></o:p></p>
</blockquote>
<p class="MsoNormal" style="margin-left:.5in"><br>
Yes, but doing claims twice feels intuitively redundant. <br>
<br>
As I've explored this space I've often wondered why we feel it is <br>
necessary to persist the resource data at all. Your shared-state <br>
model is appealing because it lets the concrete resource(-provider) <br>
be the authority about its own resources. That is information which <br>
it can broadcast as it changes or on intervals (or both) to other <br>
things which need that information. That feels like the correct <br>
architecture in a massively distributed system, especially one where <br>
resources are not scarce. <o:p></o:p></p>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<br>
So, IMHO, we should only have the compute nodes being the authority for allocating resources. They are many reasons for that I provided in the spec review, but I can reply again :<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:1.0in;text-indent:-.25in;mso-list:l0 level1 lfo1">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>#1 If we consider that an external system, as a resource provider, will provide a single resource class usage (like network segment availability), it will still require the instance to be spawned *for* consuming that resource
class, even if the scheduler accounts for it. That would mean that the scheduler would have to manage a list of allocations with TTL, and periodically verify that the allocation succeeded by asking the external system (or getting feedback from the external
system). See, that's racy.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:1.0in;text-indent:-.25in;mso-list:l0 level1 lfo1">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>#2 the scheduler is just a decision maker, by any case it doesn't account for the real instance creation (it doesn't hold the ownership of the instance). Having it being accountable for the instances usage is heavily difficult.
Take for example a request for CPU pinning or NUMA affinity. The user can't really express which pin of the pCPU he will get, that's the compute node which will do that for him. Of course, the scheduler will help picking an host that can fit the request, but
the real pinning will happen in the compute node.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">IMHO, the authority to allocate resources is not limited by compute nodes, but also include network
service, storage service or all other services which have the authority to manage their own resources. Those “shared” resources are coming from external services(i.e. system) which are not compute service. They all have responsibilities to push their own resource
updates to schedulers, make resource reservation and consumption. The resource provider series provides a flexible representation of all kinds of resources, so that scheduler can handle them without having the specific knowledge of all the resources.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><br>
Also, I'm very interested in keeping an optimistic scheduler which wouldn't lock the entire view of the world anytime a request comes in. There are many papers showing different architectures and benchmarks against different possibilities and TBH, I'm very
concerned by the scaling effect.<br>
Also, we should keep in mind our new paradigm called Cells V2, which implies a global distributed scheduler for handling all requests. Having it following the same design tenets of OpenStack [3] by having a "eventually consistent shared-state" makes my guts
saying that I'd love to see that.<br>
<br>
<br>
<br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<br>
The advantage of a centralized datastore for that information is <br>
that it provides administrative control (e.g. reserving resources for <br>
other needs) and visibility. That level of command and control seems <br>
to be something people really want (unfortunately). <br>
<br>
<o:p></o:p></p>
</blockquote>
<p class="MsoNormal" style="margin-left:.5in"><br>
My point is that while I truly understand the need of getting an API resource like "scheduler, get me how much my cloud is free", that shouldn't necessarly need to be accurate but rather eventually consistent.<br>
If operators want to do capacity planning, they need trends and thresholds, not exactly knowing the precise amounts that can change everytime a request comes in.<span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The disadvantage of a centralized datastore is that administrators/monitors should refresh their resource view from db and make comparisons see changes. The distributed
resource trackers with shared-state scheduler provides a natural way to present the constantly changing resource view with the lowest overhead. It is worth noting that the shared-state scheduler is a natural resource monitor if it does no scheduling.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><br>
<br>
-Sylvain<br>
<br>
<br>
<br>
<br>
[2] <a href="https://review.openstack.org/#/c/271823/">https://review.openstack.org/#/c/271823/</a>
<br>
[3] <a href="https://wiki.openstack.org/wiki/BasicDesignTenets">https://wiki.openstack.org/wiki/BasicDesignTenets</a><br>
<br>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-Yingxin<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>