[openstack-dev] [nova] [cyborg] Race condition in the Cyborg/Nova flow
Nadathur, Sundar
sundar.nadathur at intel.com
Wed Mar 28 18:06:44 UTC 2018
Hi Shaohe,
I have responded in the Etherpad. The Cyborg/Nova scheduling spec
details the 4 types of user requests
<https://git.openstack.org/cgit/openstack/cyborg/tree/doc/specs/rocky/cyborg-nova-sched.rst?h=refs/changes/17/554717/1#n136>.
I believe you are looking for more details on what the RC names, traits
and flavors will look like. I will add that to the spec itself.
Thanks,
Sundar
On 3/28/2018 2:10 AM, 少合冯 wrote:
> I have summarize some scenarios for fpga devices request.
> https://etherpad.openstack.org/p/cyborg-fpga-request-scenarios
>
> Please add more more scenarios to find out the exceptions that
> placement can not satisfy the filter and weight.
>
> IMOH, I refer placementto do filter and weight. If we have to let
> cyborg do filter and weight. Nova scheduler just need call cyborg
> once for all host weight though we do the weigh one by one.
>
>
> 2018-03-23 12:27 GMT+08:00 Nadathur, Sundar <sundar.nadathur at intel.com
> <mailto:sundar.nadathur at intel.com>>:
>
> Hi all,
> There seems to be a possibility of a race condition in the
> Cyborg/Nova flow. Apologies for missing this earlier. (You can
> refer to the proposed Cyborg/Nova spec
> <https://review.openstack.org/#/c/554717/1/doc/specs/rocky/cyborg-nova-sched.rst>
> for details.)
>
> Consider the scenario where the flavor specifies a resource class
> for a device type, and also specifies a function (e.g. encrypt) in
> the extra specs. The Nova scheduler would only track the device
> type as a resource, and Cyborg needs to track the availability of
> functions. Further, to keep it simple, say all the functions exist
> all the time (no reprogramming involved).
>
> To recap, here is the scheduler flow for this case:
>
> * A request spec with a flavor comes to Nova
> conductor/scheduler. The flavor has a device type as a
> resource class, and a function in the extra specs.
> * Placement API returns the list of RPs (compute nodes) which
> contain the requested device types (but not necessarily the
> function).
> * Cyborg will provide a custom filter which queries Cyborg DB.
> This needs to check which hosts contain the needed function,
> and filter out the rest.
> * The scheduler selects one node from the filtered list, and the
> request goes to the compute node.
>
> For the filter to work, the Cyborg DB needs to maintain a table
> with triples of (host, function type, #free units). The filter
> checks if a given host has one or more free units of the requested
> function type. But, to keep the # free units up to date, Cyborg on
> the selected compute node needs to notify the Cyborg API to
> decrement the #free units when an instance is spawned, and to
> increment them when resources are released.
>
> Therein lies the catch: this loop from the compute node to
> controller is susceptible to race conditions. For example, if two
> simultaneous requests each ask for function A, and there is only
> one unit of that available, the Cyborg filter will approve both,
> both may land on the same host, and one will fail. This is because
> Cyborg on the controller does not decrement resource usage due to
> one request before processing the next request.
>
> This is similar to this previous Nova scheduling issue
> <https://specs.openstack.org/openstack/nova-specs/specs/pike/implemented/placement-claims.html>.
> That was solved by having the scheduler claim a resource in
> Placement for the selected node. I don't see an analog for Cyborg,
> since it would not know which node is selected.
>
> Thanks in advance for suggestions and solutions.
>
> Regards,
> Sundar
>
>
>
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180328/dc2f173a/attachment.html>
More information about the OpenStack-dev
mailing list