<div dir="ltr">@jay I'm also against a weigher in nova/placement. This should be an optional step depends on vendor implementation, not a default one. <div><br><div>@Alex I think we should explore the idea of preferred trait.</div><div><br></div><div>@Mathew: Like Sean said, Cyborg wants to support both reprogrammable FPGA and pre-programed ones. </div><div>Therefore it is correct that in your description, the programming operation should be a call from Nova to Cyborg, and cyborg will complete the operation while nova waits. The only problem is that the weigher step should be an optional one. <br><div><br></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 7, 2018 at 9:21 PM, Jay Pipes <span dir="ltr"><<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 03/06/2018 09:36 PM, Alex Xu wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
2018-03-07 10:21 GMT+08:00 Alex Xu <<a href="mailto:soulxu@gmail.com" target="_blank">soulxu@gmail.com</a> <mailto:<a href="mailto:soulxu@gmail.com" target="_blank">soulxu@gmail.com</a>>>:<span class=""><br>
<br>
<br>
<br>
2018-03-06 22:45 GMT+08:00 Mooney, Sean K <<a href="mailto:sean.k.mooney@intel.com" target="_blank">sean.k.mooney@intel.com</a><br></span>
<mailto:<a href="mailto:sean.k.mooney@intel.com" target="_blank">sean.k.mooney@intel.co<wbr>m</a>>>:<br>
<br>
__ __<br>
<br>
__ __<br>
<br>
*From:*Matthew Booth [mailto:<a href="mailto:mbooth@redhat.com" target="_blank">mbooth@redhat.com</a><br>
<mailto:<a href="mailto:mbooth@redhat.com" target="_blank">mbooth@redhat.com</a>>]<br>
*Sent:* Saturday, March 3, 2018 4:15 PM<br>
*To:* OpenStack Development Mailing List (not for usage<br>
questions) <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack<wbr>.org</a><br>
<mailto:<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.op<wbr>enstack.org</a>>><br>
*Subject:* Re: [openstack-dev] [Nova] [Cyborg] Tracking multiple<br>
functions____<br>
<br>
__ __<span class=""><br>
<br>
On 2 March 2018 at 14:31, Jay Pipes <<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a><br></span>
<mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>>> wrote:____<br>
<br>
On 03/02/2018 02:00 PM, Nadathur, Sundar wrote:____<span class=""><br>
<br>
Hello Nova team,<br>
<br>
During the Cyborg discussion at Rocky PTG, we<br>
proposed a flow for FPGAs wherein the request spec asks<br>
for a device type as a resource class, and optionally a<br>
function (such as encryption) in the extra specs. This<br>
does not seem to work well for the usage model that I’ll<br>
describe below.<br>
<br>
An FPGA device may implement more than one function. For<br>
example, it may implement both compression and<br>
encryption. Say a cluster has 10 devices of device type<br>
X, and each of them is programmed to offer 2 instances<br>
of function A and 4 instances of function B. More<br>
specifically, the device may implement 6 PCI functions,<br>
with 2 of them tied to function A, and the other 4 tied<br>
to function B. So, we could have 6 separate instances<br></span>
accessing functions on the same device.____<br>
<br>
__ __<br>
<br>
Does this imply that Cyborg can't reprogram the FPGA at all?____<br>
<br>
*/[Mooney, Sean K] cyborg is intended to support fixed function<span class=""><br>
acclerators also so it will not always be able to program the<br>
accelerator. In this case where an fpga is preprogramed with a<br>
multi function bitstream that is statically provisioned cyborge<br>
will not be able to reprogram the slot if any of the fuctions<br>
from that slot are already allocated to an instance. In this<br>
case it will have to treat it like a fixed function device and<br>
simply allocate a unused vf of the corret type if available.<br></span>
____/*<br>
<br>
<br>
____<span class=""><br>
<br>
<br>
In the current flow, the device type X is modeled as a<br>
resource class, so Placement will count how many of them<br>
are in use. A flavor for ‘RC device-type-X + function A’<br>
will consume one instance of the RC device-type-X. But<br>
this is not right because this precludes other functions<br>
on the same device instance from getting used.<br>
<br>
One way to solve this is to declare functions A and B as<br>
resource classes themselves and have the flavor request<br>
the function RC. Placement will then correctly count the<br>
function instances. However, there is still a problem:<br>
if the requested function A is not available, Placement<br>
will return an empty list of RPs, but we need some way<br>
to reprogram some device to create an instance of<br></span>
function A.____<span class=""><br>
<br>
<br>
Clearly, nova is not going to be reprogramming devices with<br>
an instance of a particular function.<br>
<br>
Cyborg might need to have a separate agent that listens to<br>
the nova notifications queue and upon seeing an event that<br>
indicates a failed build due to lack of resources, then<br>
Cyborg can try and reprogram a device and then try<br></span>
rebuilding the original request.____<br>
<br>
__ __<span class=""><br>
<br>
It was my understanding from that discussion that we intend to<br>
insert Cyborg into the spawn workflow for device configuration<br>
in the same way that we currently insert resources provided by<br>
Cinder and Neutron. So while Nova won't be reprogramming a<br>
device, it will be calling out to Cyborg to reprogram a device,<br></span>
and waiting while that happens.____<span class=""><br>
<br>
My understanding is (and I concede some areas are a little<br></span>
hazy):____<br>
<br>
* The flavors says device type X with function Y____<br>
<br>
* Placement tells us everywhere with device type X____<span class=""><br>
<br>
* A weigher orders these by devices which already have an<br></span>
available function Y (where is this metadata stored?)____<br>
<br>
* Nova schedules to host Z____<br>
<br>
* Nova host Z asks cyborg for a local function Y and blocks____<span class=""><br>
<br>
* Cyborg hopefully returns function Y which is already<br></span>
available____<br>
<br>
* If not, Cyborg reprograms a function Y, then returns it____<br>
<br>
Can anybody correct me/fill in the gaps?____<br>
<br>
*/[Mooney, Sean K] that correlates closely to my recollection<span class=""><br>
also. As for the metadata I think the weigher may need to call<br>
to cyborg to retrieve this as it will not be available in the<br></span>
host state object./*<span class=""><br>
<br>
Is it the nova scheduler weigher or we want to support weigh on<br>
placement? Function is traits as I think, so can we have<br>
preferred_traits? I remember we talk about that parameter in the<br>
past, but we don't have good use-case at that time. This is good<br>
use-case.<br>
<br>
<br>
If we call the Cyborg from the nova scheduler weigher, that will slow down the scheduling a lot also.<br>
</span></blockquote>
<br>
Right, which is why I don't want to do any weighing in Placement at all. If folks want to sort by things that require long-running code/callbacks or silly temporal things like metrics, they can do that in a custom weigher in the nova-scheduler and take the performance hit there.<br>
<br>
Best,<br>
-jay<div class="HOEnZb"><div class="h5"><br>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Zhipeng (Howard) Huang</div><div dir="ltr"><br></div><div dir="ltr">Standard Engineer</div><div>IT Standard & Patent/IT Product Line</div><div dir="ltr">Huawei Technologies Co,. Ltd</div><div dir="ltr">Email: <a href="mailto:huangzhipeng@huawei.com" target="_blank">huangzhipeng@huawei.com</a></div><div dir="ltr">Office: Huawei Industrial Base, Longgang, Shenzhen</div><div dir="ltr"><br></div><div dir="ltr">(Previous)<br><div>Research Assistant</div><div>Mobile Ad-Hoc Network Lab, Calit2</div><div>University of California, Irvine</div><div>Email: <a href="mailto:zhipengh@uci.edu" target="_blank">zhipengh@uci.edu</a></div><div>Office: Calit2 Building Room 2402</div><div><br></div><div>OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado</div></div></div></div></div></div></div></div></div>
</div>