From: Alex Xu <soulxu@gmail.com> Sent: Monday, April 15, 2019 4:50 PM
The cyborg only needs to return un-numbered request group, then Nova will base on all the 'hw:xxx' extra specs and 'accel:device_profile.[numa node id]' to generate a placement request like above.
I am not quite following the idea(s) proposed here. Cyborg returns only the device-related request groups. The un-numbered request group in the flavor is not touched by Cyborg. Secondly, if you use the ‘accel:’ stuff in the flavor to decide NUMA affinity, how will you pass that to Placement? This thread is about the syntax of the GET /a-c call.
For example, if it is PCI device under first numa node, the extra spec will be 'accel:device_profile.0=<profile_name>' the cyborg can return a simple request 'resources=CYBORG_PCI_XX_DEVICE:1', then we merge this into the request group 'resources1=VCPU:2,MEMORY_MB:128,CYBORG_PCI_XX_DEVICE:1'. If the pci device has a special trait, then cyborg should return request group as 'resources1=CYBORG_PCI_XX_DEVICE:1&required=SOME_TRAIT', then nova merge this into placement request as 'resources1.1'.
Sorry, I don’t follow this either. The request groups have entries like ‘resources:CUSTOM_FOO=1’, not 'resources=CYBORG_PCI_XX_DEVICE:1'. So, I don’t see where to stick the NUMA node #. Anyways, for Cyborg, it seems to me that there is a fairly straightforward scheme to address NUMA affinity: annotate the device’s nested RP with a trait indicating which NUMA node it belongs to (e.g. CUSTOM_NUMA_NODE_0), and use that to guide scheduling. This should be a valid use of traits because it expresses a property of the resource provider and is used for scheduling (only). As for how the annotation is done, it could be automated. The operator’s tool that configures a device to affinitize with a NUMA node (by setting MSI-X vectors, etc.) also invokes a Cyborg API (yet to be written) with the NUMA node # -- that would identify the device RP and update Placement with that trait. The tool needs to ensure that the device has been discovered by Cyborg and updated in Placement before invoking the API. Regards, Sundar