I have a further question. When defining custom resources using Placement and Provider Config, does the service responsible for resource allocation and update of allocations to instances need to be implemented independently by the creator of the custom resource?

Provider config is the mechanism to allow you to only inject additional resource providers/traits/etc to compute nodes, so you still need an external machanism to add the relevant items to that config file. There was a spec to add mechanism to make nova delegate management of subsect of host resources to RMD[1], but this was abandoned due to the provider config and preference to avoid implementing the built-in logic to pull resources from the RMD which is quite specific to Intel's architecture.

Alternatively, does a mechanism exist within OpenStack components (e.g., Nova) to support this? AFAIK, no, and a complete external mechanism may be needed.

I'm struggling to decide on the implementation approach. Should the resource allocation process be handled as a plugin (separable from the OpenStack codebase), or should I modify the libvirt driver to add the processing for allocating LLC and memory bandwidth?

If the available resources can be obtained by a more generic way (like libvirt api or kernel api, without relying on ARM specific software) then it'd be acceptable to implement the logic to detect available resources and update resource providers within libvirt driver. [1] https://blueprints.launchpad.net/nova/+spec/rmd-base-enablement In addition there is no mechanism to change domain xml according to custom resources atm, so you may have to implement the feature within nova (or create a modified version of libvirt driver which sounds non-ideal). On 12/18/24 5:21 PM, Junya Noguchi (Fujitsu) wrote:

Dear Mr. Stanley

Thank you for your response.

I understand that I can define custom resources using Placement and Provider Config[1][2]. I've also confirmed that Intel RMD (Resource Manager Daemon) is mentioned as a use case, but it seems there's no RMD integration for OpenStack[3].

I have a further question. When defining custom resources using Placement and Provider Config, does the service responsible for resource allocation and update of allocations to instances need to be implemented independently by the creator of the custom resource? Alternatively, does a mechanism exist within OpenStack components (e.g., Nova) to support this? For example, is there a function that calls a custom hook function when creating a domain in nova-compute?

I'm struggling to decide on the implementation approach. Should the resource allocation process be handled as a plugin (separable from the OpenStack codebase), or should I modify the libvirt driver to add the processing for allocating LLC and memory bandwidth?

[1] https://docs.openstack.org/nova/latest/admin/managing-resource-providers.htm... [2] https://docs.openstack.org/placement/latest/ [3] https://networkbuilders.intel.com/docs/networkbuilders/resource-management-d... 3.1

Best regards, Junya Noguchi.

On 18/12/2024 08:21, Junya Noguchi (Fujitsu) wrote:

Dear Mr. Stanley

Thank you for your response.

I understand that I can define custom resources using Placement and Provider Config[1][2]. I've also confirmed that Intel RMD (Resource Manager Daemon) is mentioned as a use case, but it seems there's no RMD integration for OpenStack[3].

I have a further question. When defining custom resources using Placement and Provider Config, does the service responsible for resource allocation and update of allocations to instances need to be implemented independently by the creator of the custom resource? Alternatively, does a mechanism exist within OpenStack components (e.g., Nova) to support this? For example, is there a function that calls a custom hook function when creating a domain in nova-compute?

I'm struggling to decide on the implementation approach. Should the resource allocation process be handled as a plugin (separable from the OpenStack codebase), or should I modify the libvirt driver to add the processing for allocating LLC and memory bandwidth?

nova does not allow any external process to modify the xml of an instance. so eitehr the cache allocation needs to be done entirly externally of nova without modifying the xml via an external agent via something like the kernel resoucectl virtual file system or nova need to be modifed to staticaly configure it vai the xml. we do not supprot any kind of hook point or plugin workflow to modify the domain xml. We also do not currently plan to exted the provider config feature to allow you register a dynamic handler for the custom resouce. ultimately im not convinced we should add CAT/RDT type functionality in the near term. The RDT technology is really not well designed. intel CAT (cache allocation technology) shoudl be called CCT (Cache Confinmenet Technology) but that does not have the same marketing appeal. instead of allcoatiog cache ways to processes and the hardware preventing other proicess form using that cache you are actually confining a process to a set of cache ways prevetnign it form using other cacheways. other processes are free to use the cache ways you configed the process too as well. there is no isolation at the hardware level so you can still get noisy neighbour effect form unconfied proceses. that mean you need to first reserve cache ways for exclusive use by openstack (preventing all host proces form usign them) and then config individual instance to the now free cache ways. the same applies to intels memomory bandwith allcoation feature. At least that was the case 5-6  years ago when i last worked on this. That makes this feature quite unsuited for a cloud envionment and very hared to integrated Into nova static resouce allocation architure. That why we proposed two versions originally, 1.)  a simple static allcoation modle via libvirt. This ultimately did not really deliver enough value to warrant complexity. 2 external dynamic agent like RMD. ultimatly RMD was a slignly nicer approch even if it was very racy in that the RMD demaon could both mange the cache asignement of the background system processes and the vm but then you end up with nova/libvirt and RMD seperatlly trying to optimise the placemtn of the vm without knowwign about each other. im not saying that there is no way to supprot this in nova but it would be very difficult to do so. likely its more then one cycles works. before talking about an implementation i think it would be better to reflect on what because your trying to enable that migh inform a desing approch and the limitation of CAT/RDT may mean that it will never fullfile your usecase.

[1] https://docs.openstack.org/nova/latest/admin/managing-resource-providers.htm... [2] https://docs.openstack.org/placement/latest/ [3] https://networkbuilders.intel.com/docs/networkbuilders/resource-management-d... 3.1

Best regards, Junya Noguchi.

Dear Mooney Our understanding of the CAT/MBA specification is as follows. Could you please verify it?

instead of allcoatiog cache ways to processes and the hardware preventing other proicess form using that cache you are actually confining a process to a set of cache ways prevetnign it form using other cacheways. other processes are free to use the cache ways you configed the process too as well. there is no isolation at the hardware level so you can still get noisy neighbour effect form unconfied proceses.

that mean you need to first reserve cache ways for exclusive use by openstack (preventing all host proces form usign them) and then config individual instance to the now free cache ways.

Regarding cache allocation, it is possible to allocate cache resources exclusively to each cgroup. In resctrl, by setting the "mode" to "exclusive", we can assign cache resources to each cgroup exclusively, preventing other cgroups from accessing them.[1] However, this requires creating cgroups, assigning cache resources, and properly assigning processes to the cgroups via resctrl. If one of your primary concerns is the requirement for dynamic hardware control within OpenStack to enable exclusive cache resource allocation, exclusive cache allocation without it is possible now, depending on the implementation.

the same applies to intels memomory bandwith allcoation feature.

Each cgroup can be assigned a maximum bandwidth, expressed as a percentage of the total bandwidth or in MBps [1]. If the total assigned bandwidth exceeds 100%, contention may occur. We can exclusively allocate bandwidth by implementing a mechanism that keeps total assigned bandwidth below 100%. [1] https://www.kernel.org/doc/Documentation/x86/resctrl.rst Please let me know if you have any concerns or comments based on the information above. regards, Taketani

Dear Mooney I've read the libvirt pqos feature spec[1] that you shared during the weekly meeting. I want to understand correctly why the development on this feature is currently stalled, and I would appreciate community feedbacks on the best approach for supporting RDT/MPAM. 1. My understanding of the libvirt pqos feature proposal The current libvirt pqos specification[1] is not designed that CPUs can be assigned to specific cache banks. To resolve an above problem, improving the libvirt pqos spec will be restarted if the feature of NUMA topology with resource providers[2] is complete. However, significant refactoring in placement is currently constrained because placement has scalability problems[3]. Nova community is actively addressing them. Is my understanding correct? Please point out any misunderstandings. Any comments or clarifications would also be appreciated. 2. Which solution for supporting RDT/MPAM feature is the best to communities I think the optimal solution involves resolving scalability problems[3], accelerating development of NUMA topology with RPs[2] and the libvirt pqos feature[1]. However, I assume this will take considerable time(5-6cycles?). Therefore, I would like to confirm whether proposing a new design without resource providers (e.g., developing NUMA topology and pqos feature within nova[4]) would be an acceptable alternative. [1] https://review.opendev.org/c/openstack/nova-specs/+/662264/1 [2] https://review.opendev.org/c/openstack/nova-specs/+/728009 [3] https://review.opendev.org/c/openstack/nova-specs/+/938070 [4] https://review.opendev.org/q/topic:%22bp/libvirt-smarter-cpu-placement%22 Regards, Taketani