<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Sorry for the delayed response. I broadly agree with previous
replies. <br>
</p>
For the concerns about the impact of Cyborg weigher on scheduling
performance , there are some options (apart from filtering
candidates as much as possible in Placement):<br>
* Handle hosts in bulk by extending <a moz-do-not-send="true"
href="https://github.com/openstack/nova/blob/master/nova/weights.py#L67">BaseWeigher</a>
and overriding <a moz-do-not-send="true"
href="https://github.com/openstack/nova/blob/master/nova/weights.py#L92">weigh_objects</a>(),
instead of handling one host at a time.<br>
* If we have to handle one host at a time for whatever reason, since
the weigher is maintained by Cyborg, it could directly query Cyborg
DB rather than go through Cyborg REST API. This will be not unlike
other weighers. <br>
<br>
Given these and other possible optimizations, it may be too soon to
worry about the performance impact.<br>
<br>
I am working on a spec that will capture the flow discussed in the
PTG. I will try to address these aspects as well.<br>
<br>
Thanks & Regards,<br>
Sundar<br>
<br>
<div class="moz-cite-prefix">On 3/8/2018 4:53 AM, Zhipeng Huang
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAHZqm+WS3-xNuXUwc+wamjsczr-_eHJkp2mD=URhBwzzbWkSZQ@mail.gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div dir="ltr">@jay I'm also against a weigher in nova/placement.
This should be an optional step depends on vendor
implementation, not a default one.
<div><br>
<div>@Alex I think we should explore the idea of preferred
trait.</div>
<div><br>
</div>
<div>@Mathew: Like Sean said, Cyborg wants to support both
reprogrammable FPGA and pre-programed ones. </div>
<div>Therefore it is correct that in your description, the
programming operation should be a call from Nova to Cyborg,
and cyborg will complete the operation while nova waits. The
only problem is that the weigher step should be an optional
one. <br>
<div><br>
</div>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Mar 7, 2018 at 9:21 PM, Jay
Pipes <span dir="ltr"><<a href="mailto:jaypipes@gmail.com"
target="_blank" moz-do-not-send="true">jaypipes@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">On
03/06/2018 09:36 PM, Alex Xu wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
2018-03-07 10:21 GMT+08:00 Alex Xu <<a
href="mailto:soulxu@gmail.com" target="_blank"
moz-do-not-send="true">soulxu@gmail.com</a> <mailto:<a
href="mailto:soulxu@gmail.com" target="_blank"
moz-do-not-send="true">soulxu@gmail.com</a>>>:<span
class=""><br>
<br>
<br>
<br>
2018-03-06 22:45 GMT+08:00 Mooney, Sean K <<a
href="mailto:sean.k.mooney@intel.com" target="_blank"
moz-do-not-send="true">sean.k.mooney@intel.com</a><br>
</span>
<mailto:<a href="mailto:sean.k.mooney@intel.com"
target="_blank" moz-do-not-send="true">sean.k.mooney@intel.co<wbr>m</a>>>:<br>
<br>
__ __<br>
<br>
__ __<br>
<br>
*From:*Matthew Booth [mailto:<a
href="mailto:mbooth@redhat.com" target="_blank"
moz-do-not-send="true">mbooth@redhat.com</a><br>
<mailto:<a href="mailto:mbooth@redhat.com"
target="_blank" moz-do-not-send="true">mbooth@redhat.com</a>>]<br>
*Sent:* Saturday, March 3, 2018 4:15 PM<br>
*To:* OpenStack Development Mailing List (not for
usage<br>
questions) <<a
href="mailto:openstack-dev@lists.openstack.org"
target="_blank" moz-do-not-send="true">openstack-dev@lists.openstack<wbr>.org</a><br>
<mailto:<a
href="mailto:openstack-dev@lists.openstack.org"
target="_blank" moz-do-not-send="true">openstack-dev@lists.op<wbr>enstack.org</a>>><br>
*Subject:* Re: [openstack-dev] [Nova] [Cyborg]
Tracking multiple<br>
functions____<br>
<br>
__ __<span class=""><br>
<br>
On 2 March 2018 at 14:31, Jay Pipes <<a
href="mailto:jaypipes@gmail.com" target="_blank"
moz-do-not-send="true">jaypipes@gmail.com</a><br>
</span>
<mailto:<a href="mailto:jaypipes@gmail.com"
target="_blank" moz-do-not-send="true">jaypipes@gmail.com</a>>>
wrote:____<br>
<br>
On 03/02/2018 02:00 PM, Nadathur, Sundar
wrote:____<span class=""><br>
<br>
Hello Nova team,<br>
<br>
During the Cyborg discussion at
Rocky PTG, we<br>
proposed a flow for FPGAs wherein the
request spec asks<br>
for a device type as a resource class,
and optionally a<br>
function (such as encryption) in the
extra specs. This<br>
does not seem to work well for the usage
model that I’ll<br>
describe below.<br>
<br>
An FPGA device may implement more than
one function. For<br>
example, it may implement both
compression and<br>
encryption. Say a cluster has 10 devices
of device type<br>
X, and each of them is programmed to
offer 2 instances<br>
of function A and 4 instances of
function B. More<br>
specifically, the device may implement 6
PCI functions,<br>
with 2 of them tied to function A, and
the other 4 tied<br>
to function B. So, we could have 6
separate instances<br>
</span>
accessing functions on the same
device.____<br>
<br>
__ __<br>
<br>
Does this imply that Cyborg can't reprogram the
FPGA at all?____<br>
<br>
*/[Mooney, Sean K] cyborg is intended to support
fixed function<span class=""><br>
acclerators also so it will not always be able
to program the<br>
accelerator. In this case where an fpga is
preprogramed with a<br>
multi function bitstream that is statically
provisioned cyborge<br>
will not be able to reprogram the slot if any of
the fuctions<br>
from that slot are already allocated to an
instance. In this<br>
case it will have to treat it like a fixed
function device and<br>
simply allocate a unused vf of the corret type
if available.<br>
</span>
____/*<br>
<br>
<br>
____<span class=""><br>
<br>
<br>
In the current flow, the device type X
is modeled as a<br>
resource class, so Placement will count
how many of them<br>
are in use. A flavor for ‘RC
device-type-X + function A’<br>
will consume one instance of the RC
device-type-X. But<br>
this is not right because this precludes
other functions<br>
on the same device instance from getting
used.<br>
<br>
One way to solve this is to declare
functions A and B as<br>
resource classes themselves and have the
flavor request<br>
the function RC. Placement will then
correctly count the<br>
function instances. However, there is
still a problem:<br>
if the requested function A is not
available, Placement<br>
will return an empty list of RPs, but we
need some way<br>
to reprogram some device to create an
instance of<br>
</span>
function A.____<span class=""><br>
<br>
<br>
Clearly, nova is not going to be
reprogramming devices with<br>
an instance of a particular function.<br>
<br>
Cyborg might need to have a separate agent
that listens to<br>
the nova notifications queue and upon seeing
an event that<br>
indicates a failed build due to lack of
resources, then<br>
Cyborg can try and reprogram a device and
then try<br>
</span>
rebuilding the original request.____<br>
<br>
__ __<span class=""><br>
<br>
It was my understanding from that discussion
that we intend to<br>
insert Cyborg into the spawn workflow for device
configuration<br>
in the same way that we currently insert
resources provided by<br>
Cinder and Neutron. So while Nova won't be
reprogramming a<br>
device, it will be calling out to Cyborg to
reprogram a device,<br>
</span>
and waiting while that happens.____<span class=""><br>
<br>
My understanding is (and I concede some areas
are a little<br>
</span>
hazy):____<br>
<br>
* The flavors says device type X with function
Y____<br>
<br>
* Placement tells us everywhere with device type
X____<span class=""><br>
<br>
* A weigher orders these by devices which
already have an<br>
</span>
available function Y (where is this metadata
stored?)____<br>
<br>
* Nova schedules to host Z____<br>
<br>
* Nova host Z asks cyborg for a local function Y
and blocks____<span class=""><br>
<br>
* Cyborg hopefully returns function Y which
is already<br>
</span>
available____<br>
<br>
* If not, Cyborg reprograms a function Y, then
returns it____<br>
<br>
Can anybody correct me/fill in the gaps?____<br>
<br>
*/[Mooney, Sean K] that correlates closely to my
recollection<span class=""><br>
also. As for the metadata I think the weigher
may need to call<br>
to cyborg to retrieve this as it will not be
available in the<br>
</span>
host state object./*<span class=""><br>
<br>
Is it the nova scheduler weigher or we want to
support weigh on<br>
placement? Function is traits as I think, so can we
have<br>
preferred_traits? I remember we talk about that
parameter in the<br>
past, but we don't have good use-case at that time.
This is good<br>
use-case.<br>
<br>
<br>
If we call the Cyborg from the nova scheduler weigher,
that will slow down the scheduling a lot also.<br>
</span></blockquote>
<br>
Right, which is why I don't want to do any weighing in
Placement at all. If folks want to sort by things that
require long-running code/callbacks or silly temporal things
like metrics, they can do that in a custom weigher in the
nova-scheduler and take the performance hit there.<br>
<br>
Best,<br>
-jay
<div class="HOEnZb">
<div class="h5"><br>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage
questions)<br>
Unsubscribe: <a
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank"
moz-do-not-send="true">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">Zhipeng (Howard) Huang</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Standard Engineer</div>
<div>IT Standard & Patent/IT Product Line</div>
<div dir="ltr">Huawei Technologies Co,. Ltd</div>
<div dir="ltr">Email: <a
href="mailto:huangzhipeng@huawei.com"
target="_blank" moz-do-not-send="true">huangzhipeng@huawei.com</a></div>
<div dir="ltr">Office: Huawei Industrial Base,
Longgang, Shenzhen</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">(Previous)<br>
<div>Research Assistant</div>
<div>Mobile Ad-Hoc Network Lab, Calit2</div>
<div>University of California, Irvine</div>
<div>Email: <a href="mailto:zhipengh@uci.edu"
target="_blank" moz-do-not-send="true">zhipengh@uci.edu</a></div>
<div>Office: Calit2 Building Room 2402</div>
<div><br>
</div>
<div>OpenStack, OPNFV, OpenDaylight, OpenCompute
Aficionado</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>