[openstack-dev] [nova] [cyborg] Race condition in the Cyborg/Nova flow

Nadathur, Sundar sundar.nadathur at intel.com
Wed Mar 28 18:06:44 UTC 2018


Hi Shaohe,
   I have responded in the Etherpad. The Cyborg/Nova scheduling spec 
details the 4 types of user requests 
<https://git.openstack.org/cgit/openstack/cyborg/tree/doc/specs/rocky/cyborg-nova-sched.rst?h=refs/changes/17/554717/1#n136>. 


I believe you are looking for more details on what the RC names, traits 
and flavors will look like. I will add that to the spec itself.

Thanks,
Sundar

On 3/28/2018 2:10 AM, 少合冯 wrote:
> I have summarize some scenarios for fpga devices request.
> https://etherpad.openstack.org/p/cyborg-fpga-request-scenarios
>
> Please add more  more scenarios to find out the exceptions that 
> placement can not satisfy the filter and weight.
>
> IMOH, I refer placementto do filter and weight. If we have to let 
> cyborg do filter and weight.  Nova scheduler just need call cyborg 
> once for all host weight though we do the weigh one by one.
>
>
> 2018-03-23 12:27 GMT+08:00 Nadathur, Sundar <sundar.nadathur at intel.com 
> <mailto:sundar.nadathur at intel.com>>:
>
>     Hi all,
>         There seems to be a possibility of a race condition in the
>     Cyborg/Nova flow. Apologies for missing this earlier. (You can
>     refer to the proposed Cyborg/Nova spec
>     <https://review.openstack.org/#/c/554717/1/doc/specs/rocky/cyborg-nova-sched.rst>
>     for details.)
>
>     Consider the scenario where the flavor specifies a resource class
>     for a device type, and also specifies a function (e.g. encrypt) in
>     the extra specs. The Nova scheduler would only track the device
>     type as a resource, and Cyborg needs to track the availability of
>     functions. Further, to keep it simple, say all the functions exist
>     all the time (no reprogramming involved).
>
>     To recap, here is the scheduler flow for this case:
>
>       * A request spec with a flavor comes to Nova
>         conductor/scheduler. The flavor has a device type as a
>         resource class, and a function in the extra specs.
>       * Placement API returns the list of RPs (compute nodes) which
>         contain the requested device types (but not necessarily the
>         function).
>       * Cyborg will provide a custom filter which queries Cyborg DB.
>         This needs to check which hosts contain the needed function,
>         and filter out the rest.
>       * The scheduler selects one node from the filtered list, and the
>         request goes to the compute node.
>
>     For the filter to work, the Cyborg DB needs to maintain a table
>     with triples of (host, function type, #free units). The filter
>     checks if a given host has one or more free units of the requested
>     function type. But, to keep the # free units up to date, Cyborg on
>     the selected compute node needs to notify the Cyborg API to
>     decrement the #free units when an instance is spawned, and to
>     increment them when resources are released.
>
>     Therein lies the catch: this loop from the compute node to
>     controller is susceptible to race conditions. For example, if two
>     simultaneous requests each ask for function A, and there is only
>     one unit of that available, the Cyborg filter will approve both,
>     both may land on the same host, and one will fail. This is because
>     Cyborg on the controller does not decrement resource usage due to
>     one request before processing the next request.
>
>     This is similar to this previous Nova scheduling issue
>     <https://specs.openstack.org/openstack/nova-specs/specs/pike/implemented/placement-claims.html>.
>     That was solved by having the scheduler claim a resource in
>     Placement for the selected node. I don't see an analog for Cyborg,
>     since it would not know which node is selected.
>
>     Thanks in advance for suggestions and solutions.
>
>     Regards,
>     Sundar
>
>
>
>
>
>
>
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>     <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180328/dc2f173a/attachment.html>


More information about the OpenStack-dev mailing list