Open Stack

On 31.05.19 09:51, Tim Bell wrote:
> Chris,
> 
>  From the CERN set up, I think there are dedicated cells for NUMA optimised configurations (but maybe one of the engineers on the team could confirm to be sure)

This is correct: we have dedicated cells for NUMA aware guests (and 
hence do not mix NUMA aware and NUMA unaware guests on the same set
of hosts).

Arne

> 
>      Q: How important, in your cloud, is it to co-locate guests needing a
>         NUMA topology with guests that do not? A review of documentation
>         (upstream and vendor) shows differing levels of recommendation on
>         this, but in many cases the recommendation is to not do it.
>      
> A: no co-location currently
> 
>      Q: If your answer to the above is "we must be able to do that": How
>         important is it that your cloud be able to pack workloads as tight
>         as possible? That is: If there are two NUMA nodes and each has 2
>         VCPU free, should a 4 VCPU demanding non-NUMA workload be able to
>         land there? Or would you prefer that not happen?
> 
> A: not applicable
>      
> Q:   If the answer to the first question is "we can get by without
>         that" is it satisfactory to be able to configure some hosts as NUMA
>         aware and others as not, as described in the "NUMA topology with
>         RPs" spec [1]? In this set up some non-NUMA workloads could end up
>         on a NUMA host (unless otherwise excluded by traits or aggregates),
>         but only when there was contiguous resource available.
> 
> A: I think this would be OK
> 
> Tim
> -----Original Message-----
> From: Chris Dent <cdent+os at anticdent.org>
> Reply-To: "openstack-discuss at lists.openstack.org" <openstack-discuss at lists.openstack.org>
> Date: Thursday, 30 May 2019 at 14:57
> To: "OpenStack-discuss at lists.openstack.org" <OpenStack-discuss at lists.openstack.org>
> Subject: [ops][nova][placement] NUMA topology vs non-NUMA workloads
> 
>      
>      This message is primarily addressed at operators, and of those,
>      operators who are interested in effectively managing and mixing
>      workloads that care about NUMA with workloads that do not. There are
>      some questions within, after some background to explain the issue.
>      
>      At the PTG, Nova and Placement developers made a commitment to more
>      effectively manage NUMA topologies within Nova and Placement. On the
>      placement side this resulted in a spec which proposed several
>      features that would enable more expressive queries when requesting
>      allocation candidates (places for workloads to go), resulting in
>      fewer late scheduling failures.
>      
>      At first there was one spec that discussed all the features. This
>      morning it was split in two because one of the features is proving
>      hard to resolve. Those two specs can be found at:
>      
>      * https://review.opendev.org/658510 (has all the original discussion)
>      * https://review.opendev.org/662191 (the less contentious features split out)
>      
>      After much discussion, we would prefer to not do the feature
>      discussed in 658510. Called 'can_split', it would allow specified
>      classes of resource (notably VCPU and memory) to be split across
>      multiple numa nodes when each node can only contribute a portion of
>      the required resources and where those resources are modelled as
>      inventory on the NUMA nodes, not the host at large.
>      
>      While this is a good idea in principle it turns out (see the spec)
>      to cause many issues that require changes throughout the ecosystem,
>      for example enforcing pinned cpus for workloads that would normally
>      float. It's possible to make the changes, but it would require
>      additional contributors to join the effort, both in terms of writing
>      the code and understanding the many issues.
>      
>      So the questions:
>      
>      * How important, in your cloud, is it to co-locate guests needing a
>         NUMA topology with guests that do not? A review of documentation
>         (upstream and vendor) shows differing levels of recommendation on
>         this, but in many cases the recommendation is to not do it.
>      
>      * If your answer to the above is "we must be able to do that": How
>         important is it that your cloud be able to pack workloads as tight
>         as possible? That is: If there are two NUMA nodes and each has 2
>         VCPU free, should a 4 VCPU demanding non-NUMA workload be able to
>         land there? Or would you prefer that not happen?
>      
>      * If the answer to the first question is "we can get by without
>         that" is it satisfactory to be able to configure some hosts as NUMA
>         aware and others as not, as described in the "NUMA topology with
>         RPs" spec [1]? In this set up some non-NUMA workloads could end up
>         on a NUMA host (unless otherwise excluded by traits or aggregates),
>         but only when there was contiguous resource available.
>      
>      This latter question articulates the current plan unless responses
>      to this message indicate it simply can't work or legions of
>      assistance shows up. Note that even if we don't do can_split, we'll
>      still be enabling significant progress with the other features
>      described in the second spec [2].
>      
>      Thanks for your help in moving us in the right direction.
>      
>      [1] https://review.opendev.org/552924
>      [2] https://review.opendev.org/662191
>      --
>      Chris Dent                       ٩◔̯◔۶           https://anticdent.org/
>      freenode: cdent
>