<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <p>Sorry for the delayed response. I broadly agree with previous
      replies.  <br>
    </p>
    For the concerns about the impact of Cyborg weigher on scheduling
    performance , there are some options (apart from filtering
    candidates as much as possible in Placement):<br>
    * Handle hosts in bulk by extending <a moz-do-not-send="true"
      href="https://github.com/openstack/nova/blob/master/nova/weights.py#L67">BaseWeigher</a>
    and overriding <a moz-do-not-send="true"
      href="https://github.com/openstack/nova/blob/master/nova/weights.py#L92">weigh_objects</a>(),
    instead of handling one host at a time.<br>
    * If we have to handle one host at a time for whatever reason, since
    the weigher is maintained by Cyborg, it could directly query Cyborg
    DB rather than go through Cyborg REST API. This will be not unlike
    other weighers. <br>
    <br>
    Given these and other possible optimizations, it may be too soon to
    worry about the performance impact.<br>
    <br>
    I am working on a spec that will capture the flow discussed in the
    PTG. I will try to address these aspects as well.<br>
    <br>
    Thanks & Regards,<br>
    Sundar<br>
    <br>
    <div class="moz-cite-prefix">On 3/8/2018 4:53 AM, Zhipeng Huang
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAHZqm+WS3-xNuXUwc+wamjsczr-_eHJkp2mD=URhBwzzbWkSZQ@mail.gmail.com">
      <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
      <div dir="ltr">@jay I'm also against a weigher in nova/placement.
        This should be an optional step depends on vendor
        implementation, not a default one. 
        <div><br>
          <div>@Alex I think we should explore the idea of preferred
            trait.</div>
          <div><br>
          </div>
          <div>@Mathew: Like Sean said, Cyborg wants to support both
            reprogrammable FPGA and pre-programed ones. </div>
          <div>Therefore it is correct that in your description, the
            programming operation should be a call from Nova to Cyborg,
            and cyborg will complete the operation while nova waits. The
            only problem is that the weigher step should be an optional
            one. <br>
            <div><br>
            </div>
          </div>
        </div>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">On Wed, Mar 7, 2018 at 9:21 PM, Jay
          Pipes <span dir="ltr"><<a href="mailto:jaypipes@gmail.com"
              target="_blank" moz-do-not-send="true">jaypipes@gmail.com</a>></span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">On
            03/06/2018 09:36 PM, Alex Xu wrote:<br>
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">
              2018-03-07 10:21 GMT+08:00 Alex Xu <<a
                href="mailto:soulxu@gmail.com" target="_blank"
                moz-do-not-send="true">soulxu@gmail.com</a> <mailto:<a
                href="mailto:soulxu@gmail.com" target="_blank"
                moz-do-not-send="true">soulxu@gmail.com</a>>>:<span
                class=""><br>
                <br>
                <br>
                <br>
                    2018-03-06 22:45 GMT+08:00 Mooney, Sean K <<a
                  href="mailto:sean.k.mooney@intel.com" target="_blank"
                  moz-do-not-send="true">sean.k.mooney@intel.com</a><br>
              </span>
                  <mailto:<a href="mailto:sean.k.mooney@intel.com"
                target="_blank" moz-do-not-send="true">sean.k.mooney@intel.co<wbr>m</a>>>:<br>
              <br>
                      __ __<br>
              <br>
                      __ __<br>
              <br>
                      *From:*Matthew Booth [mailto:<a
                href="mailto:mbooth@redhat.com" target="_blank"
                moz-do-not-send="true">mbooth@redhat.com</a><br>
                      <mailto:<a href="mailto:mbooth@redhat.com"
                target="_blank" moz-do-not-send="true">mbooth@redhat.com</a>>]<br>
                      *Sent:* Saturday, March 3, 2018 4:15 PM<br>
                      *To:* OpenStack Development Mailing List (not for
              usage<br>
                      questions) <<a
                href="mailto:openstack-dev@lists.openstack.org"
                target="_blank" moz-do-not-send="true">openstack-dev@lists.openstack<wbr>.org</a><br>
                      <mailto:<a
                href="mailto:openstack-dev@lists.openstack.org"
                target="_blank" moz-do-not-send="true">openstack-dev@lists.op<wbr>enstack.org</a>>><br>
                      *Subject:* Re: [openstack-dev] [Nova] [Cyborg]
              Tracking multiple<br>
                      functions____<br>
              <br>
                      __ __<span class=""><br>
                <br>
                        On 2 March 2018 at 14:31, Jay Pipes <<a
                  href="mailto:jaypipes@gmail.com" target="_blank"
                  moz-do-not-send="true">jaypipes@gmail.com</a><br>
              </span>
                      <mailto:<a href="mailto:jaypipes@gmail.com"
                target="_blank" moz-do-not-send="true">jaypipes@gmail.com</a>>>
              wrote:____<br>
              <br>
                          On 03/02/2018 02:00 PM, Nadathur, Sundar
              wrote:____<span class=""><br>
                <br>
                                Hello Nova team,<br>
                <br>
                                      During the Cyborg discussion at
                Rocky PTG, we<br>
                                proposed a flow for FPGAs wherein the
                request spec asks<br>
                                for a device type as a resource class,
                and optionally a<br>
                                function (such as encryption) in the
                extra specs. This<br>
                                does not seem to work well for the usage
                model that I’ll<br>
                                describe below.<br>
                <br>
                                An FPGA device may implement more than
                one function. For<br>
                                example, it may implement both
                compression and<br>
                                encryption. Say a cluster has 10 devices
                of device type<br>
                                X, and each of them is programmed to
                offer 2 instances<br>
                                of function A and 4 instances of
                function B. More<br>
                                specifically, the device may implement 6
                PCI functions,<br>
                                with 2 of them tied to function A, and
                the other 4 tied<br>
                                to function B. So, we could have 6
                separate instances<br>
              </span>
                              accessing functions on the same
              device.____<br>
              <br>
                      __ __<br>
              <br>
                      Does this imply that Cyborg can't reprogram the
              FPGA at all?____<br>
              <br>
                      */[Mooney, Sean K] cyborg is intended to support
              fixed function<span class=""><br>
                        acclerators also so it will not always be able
                to program the<br>
                        accelerator. In this case where an fpga is
                preprogramed with a<br>
                        multi function bitstream that is statically
                provisioned cyborge<br>
                        will not be able to reprogram the slot if any of
                the fuctions<br>
                        from that slot are already allocated to an
                instance. In this<br>
                        case it will have to treat it like a fixed
                function device and<br>
                        simply allocate a unused  vf  of the corret type
                if available.<br>
              </span>
                      ____/*<br>
              <br>
              <br>
                      ____<span class=""><br>
                <br>
                <br>
                                In the current flow, the device type X
                is modeled as a<br>
                                resource class, so Placement will count
                how many of them<br>
                                are in use. A flavor for ‘RC
                device-type-X + function A’<br>
                                will consume one instance of the RC
                device-type-X.  But<br>
                                this is not right because this precludes
                other functions<br>
                                on the same device instance from getting
                used.<br>
                <br>
                                One way to solve this is to declare
                functions A and B as<br>
                                resource classes themselves and have the
                flavor request<br>
                                the function RC. Placement will then
                correctly count the<br>
                                function instances. However, there is
                still a problem:<br>
                                if the requested function A is not
                available, Placement<br>
                                will return an empty list of RPs, but we
                need some way<br>
                                to reprogram some device to create an
                instance of<br>
              </span>
                              function A.____<span class=""><br>
                <br>
                <br>
                            Clearly, nova is not going to be
                reprogramming devices with<br>
                            an instance of a particular function.<br>
                <br>
                            Cyborg might need to have a separate agent
                that listens to<br>
                            the nova notifications queue and upon seeing
                an event that<br>
                            indicates a failed build due to lack of
                resources, then<br>
                            Cyborg can try and reprogram a device and
                then try<br>
              </span>
                          rebuilding the original request.____<br>
              <br>
                      __ __<span class=""><br>
                <br>
                        It was my understanding from that discussion
                that we intend to<br>
                        insert Cyborg into the spawn workflow for device
                configuration<br>
                        in the same way that we currently insert
                resources provided by<br>
                        Cinder and Neutron. So while Nova won't be
                reprogramming a<br>
                        device, it will be calling out to Cyborg to
                reprogram a device,<br>
              </span>
                      and waiting while that happens.____<span class=""><br>
                <br>
                        My understanding is (and I concede some areas
                are a little<br>
              </span>
                      hazy):____<br>
              <br>
                      * The flavors says device type X with function
              Y____<br>
              <br>
                      * Placement tells us everywhere with device type
              X____<span class=""><br>
                <br>
                        * A weigher orders these by devices which
                already have an<br>
              </span>
                      available function Y (where is this metadata
              stored?)____<br>
              <br>
                      * Nova schedules to host Z____<br>
              <br>
                      * Nova host Z asks cyborg for a local function Y
              and blocks____<span class=""><br>
                <br>
                           * Cyborg hopefully returns function Y which
                is already<br>
              </span>
                      available____<br>
              <br>
                         * If not, Cyborg reprograms a function Y, then
              returns it____<br>
              <br>
                      Can anybody correct me/fill in the gaps?____<br>
              <br>
                      */[Mooney, Sean K] that correlates closely to my
              recollection<span class=""><br>
                        also. As for the metadata I think the weigher
                may need to call<br>
                        to cyborg to retrieve this as it will not be
                available in the<br>
              </span>
                      host state object./*<span class=""><br>
                <br>
                    Is it the nova scheduler weigher or we want to
                support weigh on<br>
                    placement? Function is traits as I think, so can we
                have<br>
                    preferred_traits? I remember we talk about that
                parameter in the<br>
                    past, but we don't have good use-case at that time.
                This is good<br>
                    use-case.<br>
                <br>
                <br>
                If we call the Cyborg from the nova scheduler weigher,
                that will slow down the scheduling a lot also.<br>
              </span></blockquote>
            <br>
            Right, which is why I don't want to do any weighing in
            Placement at all. If folks want to sort by things that
            require long-running code/callbacks or silly temporal things
            like metrics, they can do that in a custom weigher in the
            nova-scheduler and take the performance hit there.<br>
            <br>
            Best,<br>
            -jay
            <div class="HOEnZb">
              <div class="h5"><br>
                <br>
                ______________________________<wbr>______________________________<wbr>______________<br>
                OpenStack Development Mailing List (not for usage
                questions)<br>
                Unsubscribe: <a
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
                  rel="noreferrer" target="_blank"
                  moz-do-not-send="true">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
                <a
                  href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
                  rel="noreferrer" target="_blank"
                  moz-do-not-send="true">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
        <br clear="all">
        <div><br>
        </div>
        -- <br>
        <div class="gmail_signature" data-smartmail="gmail_signature">
          <div dir="ltr">
            <div>
              <div dir="ltr">
                <div>
                  <div dir="ltr">
                    <div>
                      <div dir="ltr">Zhipeng (Howard) Huang</div>
                      <div dir="ltr"><br>
                      </div>
                      <div dir="ltr">Standard Engineer</div>
                      <div>IT Standard & Patent/IT Product Line</div>
                      <div dir="ltr">Huawei Technologies Co,. Ltd</div>
                      <div dir="ltr">Email: <a
                          href="mailto:huangzhipeng@huawei.com"
                          target="_blank" moz-do-not-send="true">huangzhipeng@huawei.com</a></div>
                      <div dir="ltr">Office: Huawei Industrial Base,
                        Longgang, Shenzhen</div>
                      <div dir="ltr"><br>
                      </div>
                      <div dir="ltr">(Previous)<br>
                        <div>Research Assistant</div>
                        <div>Mobile Ad-Hoc Network Lab, Calit2</div>
                        <div>University of California, Irvine</div>
                        <div>Email: <a href="mailto:zhipengh@uci.edu"
                            target="_blank" moz-do-not-send="true">zhipengh@uci.edu</a></div>
                        <div>Office: Calit2 Building Room 2402</div>
                        <div><br>
                        </div>
                        <div>OpenStack, OPNFV, OpenDaylight, OpenCompute
                          Aficionado</div>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </div>
        </div>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
    </blockquote>
    <br>
  </body>
</html>