<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi Sylvain,<br>
Glad to know we are on the same page. I haven't updated the spec
with this proposal yet, in case I got more comments :). I will do so
by today. <br>
<br>
Thanks,<br>
Sundar<br>
<br>
<div class="moz-cite-prefix">On 5/30/2018 12:34 AM, Sylvain Bauza
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CALOCmuks-_c5XCy=y6bntgb7-A1bvrCKuHQ80UsHNx6LV7284A@mail.gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, May 30, 2018 at 1:33 AM,
Nadathur, Sundar <span dir="ltr"><<a
href="mailto:sundar.nadathur@intel.com" target="_blank"
moz-do-not-send="true">sundar.nadathur@intel.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> Hi all,<br>
The Cyborg/Nova scheduling spec [1] details what
traits will be applied to the resource providers that
represent devices like GPUs. Some of the traits referred
to vendor names. I got feedback that traits must not
refer to products or specific models of devices. I
agree. However, we need some reference to device types
to enable matching the VM driver with the device.<br>
<br>
TL;DR We need some reference to device types, but we
don't need product names. I will update the spec [1] to
clarify that. Rest of this email clarifies why we need
device types in traits, and what traits we propose to
include.<br>
<br>
In general, an accelerator device is operated by two
pieces of software: a driver in the kernel (which may
discover and handle the PF for SR-IOV devices), and a
driver/library in the guest (which may handle the
assigned VF). <br>
<br>
The device assigned to the VM must match the
driver/library packaged in the VM. For this, the request
must explicitly state what category of devices it needs.
For example, if the VM needs a GPU, it needs to say
whether it needs an AMD GPU or an Nvidia GPU, since it
may have the driver/libraries for that vendor alone. It
may also need to state what version of Cuda is needed,
if it is a Nvidia GPU. These aspects are necessarily
vendor-specific.<br>
<br>
</div>
</blockquote>
<div><br>
</div>
<div>FWIW, the vGPU implementation for Nova also has the
same concern. We want to provide traits for explicitly say
"use this vGPU type" but given it's related to a specific
vendor, we can't just say "ask for this frame buffer size,
or just for the display heads", but rather "we need a vGPU
accepting Quadro vDWS license".<br>
<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> Further, one
driver/library version may handle multiple devices.
Since a new driver version may be backwards compatible,
multiple driver versions may manage the same device. The
development/release of the driver/library inside the VM
should be independent of the kernel driver for that
device.<br>
<br>
</div>
</blockquote>
<div><br>
</div>
<div>I agree.<br>
<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> For FPGAs, there is
an additional twist as the VM may need specific
bitstream(s), and they match only specific device/region
types. The bitstream for a device from a vendor will not
fit any other device from the same vendor, let alone
other vendors. IOW, the region type is specific not just
to a vendor but to a device type within the vendor. So,
it is essential to identify the device type.<br>
<br>
So, the proposed set of RCs and traits are as below. As
we learn more about actual usages by operators, we may
need to evolve this set.<br>
<ul>
<li>There is a resource class per device category e.g.
CUSTOM_ACCELERATOR_GPU, CUSTOM_ACCELERATOR_FPGA.</li>
<li>The resource provider that represents a device has
the following traits:</li>
<ul>
<li>Vendor/Category trait: e.g. CUSTOM_GPU_AMD,
CUSTOM_FPGA_XILINX.</li>
<li>Device type trait which is a refinement of
vendor/category trait e.g.
CUSTOM_FPGA_XILINX_VU9P.</li>
</ul>
</ul>
<blockquote>
<blockquote>NOTE: This is not a product or model, at
least for FPGAs. Multiple products may use the same
FPGA chip.<br>
NOTE: The reason for having both the vendor/category
and this one is that a flavor may ask for either,
depending on the granularity desired. IOW, if one
driver can handle all devices from a vendor (*eye
roll*), the flavor can ask for the vendor/category
trait alone. If there are separate drivers for
different device families from the same vendor, the
flavor must specify the trait for the device family.<br>
NOTE: The equivalent trait for GPUs may be like
CUSTOM_GPU_NVIDIA_P90, but I'll let others decide if
that is a product or not.<br>
</blockquote>
</blockquote>
</div>
</blockquote>
<div><br>
</div>
<div>I was about to propose the same for vGPUs in Nova, ie.
using custom traits. The only concern is that we need
operators to set the traits directly using osc-placement
instead of having Nova magically provide those traits. But
anyway, given operators need to set the vGPU types they
want, I think it's acceptable.<br>
<br>
<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<blockquote>
<blockquote> </blockquote>
</blockquote>
<ul>
<ul>
<li>For FPGAs, we have additional traits:</li>
<ul>
<li>Functionality trait: e.g. CUSTOM_FPGA_COMPUTE,
CUSTOM_FPGA_NETWORK, CUSTOM_FPGA_STORAGE</li>
<li>Region type ID. e.g.
CUSTOM_FPGA_INTEL_REGION_<<wbr>uuid>.</li>
<li>Optionally, a function ID, indicating what
function is currently programmed in the region
RP. e.g. CUSTOM_FPGA_INTEL_FUNCTION_<<wbr>uuid>.
Not all implementations may provide it. The
function trait may change on reprogramming, but
it is not expected to be frequent.</li>
<li>Possibly, CUSTOM_PROGRAMMABLE as a separate
trait.<br>
</li>
</ul>
</ul>
</ul>
[1] <a
class="m_5186080365214722662moz-txt-link-freetext"
href="https://review.openstack.org/#/c/554717/"
target="_blank" moz-do-not-send="true">https://review.openstack.org/#<wbr>/c/554717/</a></div>
</blockquote>
<div><br>
</div>
<div><br>
I'll try to review the spec as soon as I can.<br>
<br>
</div>
<div>-Sylvain <br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><br>
<br>
Thanks.<br>
<br>
Regards,<br>
Sundar<br>
</div>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage
questions)<br>
Unsubscribe: <a
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank" moz-do-not-send="true">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
<!--'"--><br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>