<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Sep 17, 2018 at 6:43 AM, Jay Pipes <span dir="ltr"><<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 09/16/2018 09:28 PM, Naichuan Sun wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi, Sylvain,<br>
<br>
In truth I’m worrying about the old root rp which include the vgpu inventory. There is no field in the inventory which can display which GPU/GPUG it belong to, right? Anyway,  will discuss it after you come back.<br>
</blockquote>
<br></span>
As Sylvain mentions below, you will need to have some mechanism in the XenAPI virt driver which creates child resource providers under the existing root provider (which is the compute node resource provider). You will need to have the virt driver persist the mapping between your internal physical GPU group name and the UUID of the resource provider record that the virt driver creates for that PGPU group.<br></blockquote><div><br></div><div>AFAICT, we even don't need to persist the mapping. As we only support one GPU type (or group for Xen) in Rocky, you just have to know what was the original type for Stein and then just look at the related resource provider;</div><div>That's why i wrote an upgrade impact section in my multiple-types spec (see below) saying that in Stein, you need to make sure you only accept one type until the reshape is fully done.</div><div><br></div><div>-Sylvain</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
So, for example, let's say you have two PGPU groups on the host. They are named PGPU_A and PGPU_B. The XenAPI virt driver will need to ask the ProviderTree object it receives in the update_provider_tree() virt driver method whether there is a resource provider named "PGPU_A" in the tree. If not, the virt driver needs to create a new child resource provider with the name "PGPU_A" with a parent provider pointing to the root compute node provider. The ProviderTree.new_child() method is used to create new child providers:<br>
<br>
<a href="https://github.com/openstack/nova/blob/82270cc261f6c1d9d2cc386f1fb445dd66023f75/nova/compute/provider_tree.py#L411" rel="noreferrer" target="_blank">https://github.com/openstack/n<wbr>ova/blob/82270cc261f6c1d9d2cc3<wbr>86f1fb445dd66023f75/nova/<wbr>compute/provider_tree.py#L411</a><br>
<br>
Hope that makes sense,<br>
-jay<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
Thank very much.<br>
<br>
BR.<br>
<br>
Naichuan Sun<br>
<br></span>
*From:*Sylvain Bauza [mailto:<a href="mailto:sbauza@redhat.com" target="_blank">sbauza@redhat.com</a>]<br>
*Sent:* Friday, September 14, 2018 9:34 PM<br>
*To:* OpenStack Development Mailing List (not for usage questions) <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack<wbr>.org</a>><br>
*Subject:* Re: [openstack-dev] About microversion setting to enable nested resource provider<span class=""><br>
<br>
Le jeu. 13 sept. 2018 à 19:29, Naichuan Sun <<a href="mailto:naichuan.sun@citrix.com" target="_blank">naichuan.sun@citrix.com</a> <mailto:<a href="mailto:naichuan.sun@citrix.com" target="_blank">naichuan.sun@citrix.co<wbr>m</a>>> a écrit :<br>
<br>
    Hi, Sylvain,<br>
<br>
    Thank you very much for the information. It is pity that I can’t<br>
    attend the meeting.<br>
<br>
    I have a concern about reshaper in multi-type vgpu support.<br>
<br>
    In the old vgpu support, we only have one vgpu inventory in root<br>
    resource provider, which means we only support one vgpu type. When<br>
    do reshape, placement will send allocations(which include just one<br>
    vgpu resource allocation information) to the driver, if the host<br>
    have more than one pgpu/pgpug(which support different vgpu type),<br>
    how do we know which pgpu/pgpug own the allocation information? Do<br>
    we need to communicate with hypervisor the confirm that?<br>
<br>
The reshape will actually move the existing allocations for a VGPU resource class to the inventory for this class that is on the child resource provider now with the reshape.<br>
<br>
Since we agreed on keeping consistent naming, there is no need to guess which is which. That said, you raise a point that was discussed during the PTG and we all agreed there was an upgrade impact as multiple vGPUs shouldn't be allowed until the reshape is done.<br>
<br>
Accordingly, see my spec I reproposed for Stein which describes the upgrade impact <a href="https://review.openstack.org/#/c/602474/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/602474/</a><br>
<br>
Since I'm at the PTG, we have huge time difference between you and me, but we can discuss on that point next week when I'm back (my mornings match then your afternoons)<br>
<br>
-Sylvain<br>
<br>
    Thank you very much.<br>
<br>
    BR.<br>
<br>
    Naichuan Sun<br>
<br></span>
    *From:*Sylvain Bauza [mailto:<a href="mailto:sbauza@redhat.com" target="_blank">sbauza@redhat.com</a><br>
    <mailto:<a href="mailto:sbauza@redhat.com" target="_blank">sbauza@redhat.com</a>>]<br>
    *Sent:* Thursday, September 13, 2018 11:47 PM<br>
    *To:* OpenStack Development Mailing List (not for usage questions)<br>
    <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack<wbr>.org</a><br>
    <mailto:<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.op<wbr>enstack.org</a>>><br>
    *Subject:* Re: [openstack-dev] About microversion setting to enable<span class=""><br>
    nested resource provider<br>
<br>
    Hey Naichuan,<br>
<br>
    FWIW, we discussed on the missing pieces for nested resource<br>
    providers. See the (currently-in-use) etherpad<br>
    <a href="https://etherpad.openstack.org/p/nova-ptg-stein" rel="noreferrer" target="_blank">https://etherpad.openstack.org<wbr>/p/nova-ptg-stein</a> and lookup for<br>
    "closing the gap on nested resource providers" (L144 while I speak)<br>
<br>
    The fact that we are not able to schedule yet is a critical piece<br>
    that we said we're going to work on it as soon as we can.<br>
<br>
    -Sylvain<br>
<br>
    On Thu, Sep 13, 2018 at 9:14 AM, Eric Fried <openstack@fried.cc<br></span><span class="">
    <mailto:<a href="mailto:openstack@fried.cc" target="_blank">openstack@fried.cc</a>>> wrote:<br>
<br>
        There's a patch series in progress for this:<br>
<br>
        <a href="https://review.openstack.org/#/q/topic:use-nested-allocation-candidates" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/q/topic:use-nested-allocation<wbr>-candidates</a><br>
<br>
        It needs some TLC. I'm sure gibi and tetsuro would welcome some<br>
        help...<br>
<br>
        efried<br>
<br>
<br>
        On 09/13/2018 08:31 AM, Naichuan Sun wrote:<br>
         > Thank you very much, Jay.<br>
         > Is there somewhere I could set microversion(some configure<br>
        file?), Or just modify the source code to set it?<br>
         ><br>
         > BR.<br>
         > Naichuan Sun<br>
         ><br>
         > -----Original Message-----<br>
         > From: Jay Pipes [mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a><br>
        <mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>>]<br>
         > Sent: Thursday, September 13, 2018 9:19 PM<br>
         > To: Naichuan Sun <<a href="mailto:naichuan.sun@citrix.com" target="_blank">naichuan.sun@citrix.com</a><br></span><span class="">
        <mailto:<a href="mailto:naichuan.sun@citrix.com" target="_blank">naichuan.sun@citrix.co<wbr>m</a>>>; OpenStack Development Mailing<br>
        List (not for usage questions)<br>
        <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack<wbr>.org</a><br></span><span class="">
        <mailto:<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.op<wbr>enstack.org</a>>><br>
         > Cc: melanie witt <<a href="mailto:melwittt@gmail.com" target="_blank">melwittt@gmail.com</a><br></span>
        <mailto:<a href="mailto:melwittt@gmail.com" target="_blank">melwittt@gmail.com</a>>>; <a href="mailto:efried@us.ibm.com" target="_blank">efried@us.ibm.com</a><br>
        <mailto:<a href="mailto:efried@us.ibm.com" target="_blank">efried@us.ibm.com</a>>; Sylvain Bauza <<a href="mailto:sbauza@redhat.com" target="_blank">sbauza@redhat.com</a><br>
        <mailto:<a href="mailto:sbauza@redhat.com" target="_blank">sbauza@redhat.com</a>>><div><div class="h5"><br>
         > Subject: Re: About microversion setting to enable nested<br>
        resource provider<br>
         ><br>
         > On 09/13/2018 06:39 AM, Naichuan Sun wrote:<br>
         >> Hi, guys,<br>
         >><br>
         >> Looks n-rp is disabled by default because microversion<br>
        matches 1.29 :<br>
         >><br>
        <a href="https://github.com/openstack/nova/blob/master/nova/api/openstack/place" rel="noreferrer" target="_blank">https://github.com/openstack/n<wbr>ova/blob/master/nova/api/opens<wbr>tack/place</a><br>
         >> ment/handlers/allocation_candi<wbr>date.py#L252<br>
         >><br>
         >> Anyone know how to set the microversion to enable n-rp in<br>
        placement?<br>
         ><br>
         > It is the client which must send the 1.29+ placement API<br>
        microversion header to indicate to the placement API server that<br>
        the client wants to receive nested provider information in the<br>
        allocation candidates response.<br>
         ><br>
         > Currently, nova-scheduler calls the scheduler reportclient's<br>
         > get_allocation_candidates() method:<br>
         ><br>
         ><br>
        <a href="https://github.com/openstack/nova/blob/0ba34a818414823eda5e693dc2127a534410b5df/nova/scheduler/manager.py#L138" rel="noreferrer" target="_blank">https://github.com/openstack/n<wbr>ova/blob/0ba34a818414823eda5e6<wbr>93dc2127a534410b5df/nova/<wbr>scheduler/manager.py#L138</a><br>
         ><br>
         > The scheduler reportclient's get_allocation_candidates()<br>
        method currently passes the 1.25 placement API microversion header:<br>
         ><br>
         ><br>
        <a href="https://github.com/openstack/nova/blob/0ba34a818414823eda5e693dc2127a534410b5df/nova/scheduler/client/report.py#L353" rel="noreferrer" target="_blank">https://github.com/openstack/n<wbr>ova/blob/0ba34a818414823eda5e6<wbr>93dc2127a534410b5df/nova/<wbr>scheduler/client/report.py#<wbr>L353</a><br>
         ><br>
         ><br>
        <a href="https://github.com/openstack/nova/blob/0ba34a818414823eda5e693dc2127a534410b5df/nova/scheduler/client/report.py#L53" rel="noreferrer" target="_blank">https://github.com/openstack/n<wbr>ova/blob/0ba34a818414823eda5e6<wbr>93dc2127a534410b5df/nova/<wbr>scheduler/client/report.py#L53</a><br>
         ><br>
         > In order to get the nested information returned in the<br>
        allocation candidates response, that would need to be upped to 1.29.<br>
         ><br>
         > Best,<br>
         > -jay<br>
<br>
         ><br>
        ______________________________<wbr>______________________________<wbr>______________<br>
         > OpenStack Development Mailing List (not for usage questions)<br>
         > Unsubscribe:<br>
        <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br></div></div>
        <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@<wbr>lists.openstack.org?subject:un<wbr>subscribe</a>><span class=""><br>
         > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
         ><br>
<br>
        ______________________________<wbr>______________________________<wbr>______________<br>
        OpenStack Development Mailing List (not for usage questions)<br>
        Unsubscribe:<br>
        <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br></span>
        <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@<wbr>lists.openstack.org?subject:un<wbr>subscribe</a>><span class=""><br>
        <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
<br>
    ______________________________<wbr>______________________________<wbr>______________<br>
    OpenStack Development Mailing List (not for usage questions)<br>
    Unsubscribe:<br>
    <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br></span>
    <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@<wbr>lists.openstack.org?subject:un<wbr>subscribe</a>><span class=""><br>
    <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
<br>
<br>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
<br>
</span></blockquote><div class="HOEnZb"><div class="h5">
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
</div></div></blockquote></div><br></div></div>