<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></a></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><a name="_____replyseparator"></a><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Matthew Booth [mailto:mbooth@redhat.com]
<br>
<b>Sent:</b> Saturday, March 3, 2018 4:15 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org><br>
<b>Subject:</b> Re: [openstack-dev] [Nova] [Cyborg] Tracking multiple functions<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">On 2 March 2018 at 14:31, Jay Pipes <<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">On 03/02/2018 02:00 PM, Nadathur, Sundar wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">Hello Nova team,<br>
<br>
During the Cyborg discussion at Rocky PTG, we proposed a flow for FPGAs wherein the request spec asks for a device type as a resource class, and optionally a function (such as encryption) in the extra specs. This does not seem to work well for the usage
model that I’ll describe below.<br>
<br>
An FPGA device may implement more than one function. For example, it may implement both compression and encryption. Say a cluster has 10 devices of device type X, and each of them is programmed to offer 2 instances of function A and 4 instances of function
B. More specifically, the device may implement 6 PCI functions, with 2 of them tied to function A, and the other 4 tied to function B. So, we could have 6 separate instances accessing functions on the same device.<o:p></o:p></p>
</blockquote>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Does this imply that Cyborg can't reprogram the FPGA at all?<span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[Mooney, Sean K] cyborg is intended to support fixed function acclerators also so it will not always be able to program the accelerator. In this case where
an fpga is preprogramed with a multi function bitstream that is statically provisioned cyborge will not be able to reprogram the slot if any of the fuctions from that slot are already allocated to an instance. In this case it will have to treat it like a fixed
function device and simply allocate a unused vf of the corret type if available.
<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><br>
<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal"><br>
In the current flow, the device type X is modeled as a resource class, so Placement will count how many of them are in use. A flavor for ‘RC device-type-X + function A’ will consume one instance of the RC device-type-X. But this is not right because this precludes
other functions on the same device instance from getting used.<br>
<br>
One way to solve this is to declare functions A and B as resource classes themselves and have the flavor request the function RC. Placement will then correctly count the function instances. However, there is still a problem: if the requested function A is not
available, Placement will return an empty list of RPs, but we need some way to reprogram some device to create an instance of function A.<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><br>
Clearly, nova is not going to be reprogramming devices with an instance of a particular function.<br>
<br>
Cyborg might need to have a separate agent that listens to the nova notifications queue and upon seeing an event that indicates a failed build due to lack of resources, then Cyborg can try and reprogram a device and then try rebuilding the original request.<o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">It was my understanding from that discussion that we intend to insert Cyborg into the spawn workflow for device configuration in the same way that we currently insert resources provided by Cinder and Neutron.
So while Nova won't be reprogramming a device, it will be calling out to Cyborg to reprogram a device, and waiting while that happens.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">My understanding is (and I concede some areas are a little hazy):<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">* The flavors says device type X with function Y<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">* Placement tells us everywhere with device type X<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">* A weigher orders these by devices which already have an available function Y (where is this metadata stored?)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">* Nova schedules to host Z<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">* Nova host Z asks cyborg for a local function Y and blocks<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> * Cyborg hopefully returns function Y which is already available<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"> * If not, Cyborg reprograms a function Y, then returns it<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Can anybody correct me/fill in the gaps?<o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[Mooney, Sean K] that correlates closely to my recollection also. As for the metadata I think the weigher may need to call
to cyborg to retrieve this as it will not be available in the host state object.</span></i></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Matt<o:p></o:p></p>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">Matthew Booth</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Red Hat OpenStack Engineer, Compute DFG<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Phone: +442070094448 (UK)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>