Open Stack

From: Alex Xu <soulxu at gmail.com>
Sent: Monday, April 15, 2019 4:50 PM
> The cyborg only needs to return un-numbered request group, then Nova will
> base on all the 'hw:xxx' extra specs and 'accel:device_profile.[numa node
> id]' to generate a placement request like above.

I am not quite following the idea(s) proposed here. Cyborg returns only the device-related request groups. The un-numbered request group in the flavor is not touched by Cyborg.

Secondly, if you use the ‘accel:’ stuff in the flavor to decide NUMA affinity, how will you pass that to Placement? This thread is about the syntax of the GET /a-c call.

> For example, if it is PCI device under first numa node, the extra spec will
> be 'accel:device_profile.0=<profile_name>' the cyborg can return a simple
> request 'resources=CYBORG_PCI_XX_DEVICE:1', then we merge this into the
> request group 'resources1=VCPU:2,MEMORY_MB:128,CYBORG_PCI_XX_DEVICE:1'. If
> the pci device has a special trait, then cyborg should return request group
> as 'resources1=CYBORG_PCI_XX_DEVICE:1&required=SOME_TRAIT', then nova merge
> this into placement request as 'resources1.1'.

Sorry, I don’t follow this either. The request groups have entries like ‘resources:CUSTOM_FOO=1’, not 'resources=CYBORG_PCI_XX_DEVICE:1'. So, I don’t see where to stick the NUMA node #.

Anyways, for Cyborg, it seems to me that there is a fairly straightforward scheme to address NUMA affinity: annotate the device’s nested RP with a trait indicating which NUMA node it belongs to (e.g. CUSTOM_NUMA_NODE_0), and use that to guide scheduling. This should be a valid use of traits because it expresses a property of the resource provider and is used for scheduling (only).

As for how the annotation is done, it could be automated. The operator’s tool that configures a device to affinitize with a NUMA node (by setting MSI-X vectors, etc.) also invokes a Cyborg API (yet to be written) with the NUMA node # -- that would identify the device RP and update Placement with that trait. The tool needs to ensure that the device has been discovered by Cyborg and updated in Placement before invoking the API.

Regards,
Sundar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190425/65a3ff07/attachment-0001.html>