<div dir="ltr"><div>As I know placement and nova scheduler dedicate to <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">filter<span> and <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">weight.
<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span> </span>Placement and nova scheduler </span></span></span></span></span>is responsible for avoiding race.</div><br><div>Nested provider + traits should cover most scenarios.</div><div><br></div><div>Any special case please let the nova developer and cyborg developer know, let work together to get a solution.</div><div><br></div><div><br></div><div><br></div><div>I re-paste our design (for a POC) I have send it before as follow, hopeful it can helpful.</div><div>We do not let cyborg do any scheduler function( include filter and weight). It just responsible to do binding for FPGA device and vm instance ( <span style="font-size:14px">or call it FPGA devices assignment</span>)</div><div><br></div><div>===============================================<br></div><div>
<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">hi all</span><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">IMHO, we can consider the upstream of image management and resource provider management, even scheduler weight. </div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">1. <span> </span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">image<span> <span> </span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">management</span></span></span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">For <span> </span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">image management, I miss one things in the meeting.</span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">We have discussed it before.</span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">And Li Liu suggested to add a<span> </span><span class="gmail-il">cyborg</span><span> </span>wrapper to upload the FPGA image. </span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">This is a good ideas. </div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">For example:</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div class="gmail-m_-6266989323224467919gmail-m_46890594963785521gmail-col-md-2" style="box-sizing:border-box;min-height:1px;padding-right:15px;padding-left:15px;float:left;width:114.839px;color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div class="gmail-m_-6266989323224467919gmail-m_46890594963785521gmail-operation" style="box-sizing:border-box;margin-right:1.25em;white-space:nowrap"><span class="gmail-m_-6266989323224467919gmail-m_46890594963785521gmail-label gmail-m_-6266989323224467919gmail-m_46890594963785521gmail-label-PUT" style="box-sizing:border-box;display:inline;padding:0.2em 0.6em 0.3em;font-size:10.5px;font-weight:700;line-height:1;color:rgb(255,255,255);text-align:center;white-space:nowrap;vertical-align:baseline;border-radius:0.25em;background-color:rgb(240,173,78)">PUT</span></div></div><div class="gmail-m_-6266989323224467919gmail-m_46890594963785521gmail-col-md-9" style="box-sizing:border-box;min-height:1px;padding-right:15px;padding-left:15px;float:left;width:516.809px;color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div class="gmail-m_-6266989323224467919gmail-m_46890594963785521gmail-row" style="box-sizing:border-box"><div class="gmail-m_-6266989323224467919gmail-m_46890594963785521endpoint-container" style="box-sizing:border-box;padding-left:15px"><div class="gmail-m_-6266989323224467919gmail-m_46890594963785521gmail-row gmail-m_-6266989323224467919gmail-m_46890594963785521gmail-col-md-12" style="box-sizing:border-box;min-height:1px;padding-right:15px;padding-left:15px;float:left;width:501.809px">/<span class="gmail-il">cyborg</span>/v1/images/<span class="gmail-m_-6266989323224467919gmail-m_46890594963785521gmail-path_parameter" style="box-sizing:border-box;font-family:monospace;padding:2px 4px;font-size:12.6px;color:rgb(199,37,78);white-space:nowrap;background-color:rgb(249,242,244);border-radius:4px">{image_id}</span>/f<wbr>ile</div></div></div></div></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="text-align:start;text-indent:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div><span style="font-size:14px">It will call glance upload API to upload the image. </span><br></div><div><span style="font-size:14px">This is helpful for us to normalize the tags of image and properties. </span></div><div><span style="font-size:14px"><br></span></div><div><span style="font-size:14px">To Dutch, Li Liu, Dolpher, Sunder and other FPGA experts:</span></div><div><span style="font-size:14px"> How about get agreement on the standardization of glance image metadata, especially, tags and property. </span></div><div><span style="font-size:14px"><br></span></div><div><span style="font-size:14px">For the tags: </span></div><div><span style="font-size:14px"> IMHO, the "FPGA" is necessary, for there maybe many images managed by glance, not only fpga image but also VM image. This tag can be a filter help us to get only fpga images.</span></div><div><span style="font-size:14px"> The vendor name is necessary as a tag? Such as "INTEL" or "XILINX"</span></div><div><span style="font-size:14px"> The product model is necessary as a tag? Such as "STRATIX10"</span></div><div><span style="font-size:14px"> Any others should be in the image tags?</span></div><div><span style="font-size:14px">For the properties : </span></div><div><span style="font-size:14px"> It should include the function name(this means the accelerator type). Should it also include stream id and vendor name?</span></div><div><span style="font-size:14px"> such as: --property vendor=xilinx --property type=crypto,transcoding</span></div><div><span style="font-size:14px"> Any others should be in the image properties?</span></div><div><span style="font-size:14px"><br></span></div><div><span style="font-size:14px">Li Liu is working on the spec.</span></div><div><span style="font-size:14px"><br></span></div><div><span style="font-size:14px"><br></span></div><div><span style="font-size:14px">2. provider management. </span></div><div><span style="font-size:14px"> resource class, maybe the nested provider supported. </span></div><div><span style="font-size:14px"> we can define them as fellow:</span></div><div><span style="font-size:14px"> level 1 provider resource class is CUSTOM_FPGA_<type>, and level 2 is CUSTOM_FPGA_<vendor>_<type>, level 3 is CUSTOM_FPGA_<vendor>_<model>_<type></span></div><div><span style="font-size:14px"> { "CUSTOM_FPGA_VF": </span></div><div><span style="font-size:14px"> { "num": 3</span></div><div><span style="font-size:14px"> "CUSTOM_FPGA_ XILINX _VF": { "num": 1 }</span></div><div><span style="font-size:14px"> "CUSTOM_FPGA_INTEL_VF": </span></div><div><span style="font-size:14px"> { "CUSTOM_FPGA_INTEL_STRATIX10_VF": "num": 1 }</span></div><div><span style="font-size:14px"> { "CUSTOM_FPGA_INTEL_STRATIX11_VF": "num": 1 } </span></div><div><span style="font-size:14px"> }</span></div><div><span style="font-size:14px"> }</span></div><div><span style="font-size:14px"> Not sure I understand correctly. </span></div><div><span style="font-size:14px"><br></span></div><div><span style="font-size:14px"> And traits should include: CUSTOM_<domain>_FUNCTION_<function></span></div><div><span style="font-size:14px"> domain means which project to consume these traits. CYBORG or ACCELERATOR which is better? Here it means cyborg care these traits. Nova, neutron, cinder can ignore them.</span></div><div><span style="font-size:14px"> function, can be CRYPTO, TRANSCODING.</span></div><div><span style="font-size:14px"><br></span></div><div><span style="font-size:14px">To Jay Pipes, Dutch, Li Liu, Dolpher, Sunder and other FPGA/placement experts: </span></div><div><span style="font-size:14px"> Any suggestion on it? </span></div><div><span style="font-size:14px"><br></span></div><div><span style="font-size:14px">3. scheduler weight.</span></div><div><span style="font-size:14px">I think this is not the high priority at present for cyborg.</span></div><div><span style="font-size:14px">Zhipeng, Li Liu, Zhuli, Dopher and I have discussed them before for the deployable model implementation. </span></div><div><span style="font-size:14px">We need to add steaming or image information for deployable. </span></div><div><span style="font-size:14px">Li Liu and Zhuli's design, they do have add extra info for deployable. So it can be used for steaming or image information.</span></div><div><span style="font-size:14px"><br></span></div><div><span style="font-size:14px">And cyborg API had better support filters for scheduler weighting. </span></div><div><span style="font-size:14px">Such as:</span></div><div><span style="font-size:14px">GET /cyborg/v1/accelerators?hosts=cyborg-1, cyborg-2, cyborg-3&function=crypto,transcoding</span></div><div><span style="font-size:14px">It query all the hosts cyborg-1, cyborg-2, cyborg-3 to get all accelerators support crypto and transcoding function.</span></div><div><span style="font-size:14px">Cyborg API call conductor to get the accelerators information from by these filters</span></div><div><span style="font-size:14px">scheduler can leverage the the accelerators information for weighting.</span></div><div><span style="font-size:14px">Maybe Cyborg API can also help to do the weighting. But I think this is not a good idea.</span></div><div><span style="font-size:14px"><br></span></div><div><span style="font-size:14px">To Sunder:</span></div><div><span style="font-size:14px">I know you are interested in scheduler weight and you have some other weighting solutions. </span></div><div><span style="font-size:14px">Hopeful this can useful for you.</span></div></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">REF:<span> </span><a href="https://etherpad.openstack.org/p/cyborg-nova-poc" target="_blank" style="color:rgb(17,85,204)">https://etherpad.openstack.org<wbr>/p/<span class="gmail-il">cyborg</span>-nova-poc</a><br></div><br class="gmail-Apple-interchange-newline">
<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2018-03-23 12:27 GMT+08:00 Nadathur, Sundar <span dir="ltr"><<a href="mailto:sundar.nadathur@intel.com" target="_blank">sundar.nadathur@intel.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
Hi all,<br>
There seems to be a possibility of a race condition in the
Cyborg/Nova flow. Apologies for missing this earlier. (You can refer
to the <a href="https://review.openstack.org/#/c/554717/1/doc/specs/rocky/cyborg-nova-sched.rst" target="_blank">proposed
Cyborg/Nova spec</a> for details.)<br>
<br>
Consider the scenario where the flavor specifies a resource class
for a device type, and also specifies a function (e.g. encrypt) in
the extra specs. The Nova scheduler would only track the device type
as a resource, and Cyborg needs to track the availability of
functions. Further, to keep it simple, say all the functions exist
all the time (no reprogramming involved).<br>
<br>
To recap, here is the scheduler flow for this case:<br>
<ul>
<li>A request spec with a flavor comes to Nova
conductor/scheduler. The flavor has a device type as a resource
class, and a function in the extra specs.</li>
<li>Placement API returns the list of RPs (compute nodes) which
contain the requested device types (but not necessarily the
function).</li>
<li>Cyborg will provide a custom filter which queries Cyborg DB.
This needs to check which hosts contain the needed function, and
filter out the rest.</li>
<li>The scheduler selects one node from the filtered list, and the
request goes to the compute node.<br>
</li>
</ul>
For the filter to work, the Cyborg DB needs to maintain a table with
triples of (host, function type, #free units). The filter checks if
a given host has one or more free units of the requested function
type. But, to keep the # free units up to date, Cyborg on the
selected compute node needs to notify the Cyborg API to decrement
the #free units when an instance is spawned, and to increment them
when resources are released. <br>
<br>
Therein lies the catch: this loop from the compute node to
controller is susceptible to race conditions. For example, if two
simultaneous requests each ask for function A, and there is only one
unit of that available, the Cyborg filter will approve both, both
may land on the same host, and one will fail. This is because Cyborg
on the controller does not decrement resource usage due to one
request before processing the next request.<br>
<br>
This is similar to <a href="https://specs.openstack.org/openstack/nova-specs/specs/pike/implemented/placement-claims.html" target="_blank">this
previous Nova scheduling issue</a>. That was solved by having the
scheduler claim a resource in Placement for the selected node. I
don't see an analog for Cyborg, since it would not know which node
is selected. <br>
<br>
Thanks in advance for suggestions and solutions.<br>
<br>
Regards,<br>
Sundar <br>
<br>
<br>
<br>
<br>
<br>
<br>
</div>
<br>______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
<br></blockquote></div><br></div>