Open Stack

On Mon, 2020-10-19 at 20:38 -0500, Eric K. Miller wrote:
> > hw:numa_nodes=1 does not enable per numa node memory tracking
> > to resolve your OOM issue you need to set hw:mem_page_size=small or
> > hw:mem_page_size=any
> 
> Ah!  That's what I was looking for! :)  Thank you Sean!
> 
> > the reason that it is always selection  numa 0 is that nova is taking the list of
> > host numa nodes and checkign each one using itertools.permutations.
> > that always checks the numa nodes in a stable order starting with numa node
> > 0.
> > 
> > since you have jsut set hw:numa_nodes=1 without requesting any numa
> > specifc resouces e.g. memory or cpus  numa node 0 will effectivly always fit
> > the
> > vm
> 
> Makes sense.
> 
> > when you set hw:numa_nodes=1 and nothing else the scudler will only
> > reject a node if the number of cpus on the numa node is less that the
> > number the
> > vm requests. it will not check the memory availabel on the numa node since
> > you did not ask nova to do that via hw:mem_page_size
> > 
> > effectivly if you are using any numa feature in nova and do not set
> > hw:mem_page_size then your flavor is misconfigured as it will not request
> > numa
> > local memory trackign to be enabled.
> 
> Good to know.
> 
> So it sounds like by setting the hw:mem_page_size parameter (probably best to choose "small" as a general default), NUMA node 0 will fill up, and
> then NUMA node 1 will be considered.  In other words, VMs will NOT be provisioned in a "round-robin" fashion between NUMA nodes.  Do I understand
> that correctly?

yes you do https://bugs.launchpad.net/nova/+bug/1893121 basically tracks this. i fundemetally belive this is a performance bug
not a feature althogh others disagree. This is why we have always recommended you set hw:numa_nodes to the number of numa nodes on the host if you
can. the excpetion to that is for workloads that dont support numa awareness in which case you shoudl only deviate form this advice if you mesusre
a perfromace degreadation. with the default behavior you will saturate once numa node before usign the other this
lead to pessimising your memory badnwidth and cpu perfromacne. tl;dr since all the vms are being packed on the first numa node/socket it will
effectivly leave the scond socket ideal and load up the first socket causeing the process to not be able to turbo bost as agressivly as if you had
spread the vms between each numa node evenly.

as you get to higher utilisation that is less of an issue but its a non zero effect on lightly utilised cloud sicne we spread host by default
and is potentially less energy effincet as the termal load will also not be spread between the cpus.
> 
> > you do not need to use hugepages but you do need to enable per numa
> > node memory tracking with hw:mem_page_size=small (use non hugepage
> > typicaly 4k
> > pages) or hw:mem_page_size=any  which basicaly is the same as small but
> > the image can requst hugepages if it wantws too. if you  set small in the
> > flaor but large in the image that is an error. if you set any in the falvor the
> > image can set any value it like like small or large or an explcit page
> > size and the schduler will honour that.
> > 
> > if you know you wnat the flavor to use small pages then you shoudl just set
> > small explictly.
> 
> Also good to know.  Thanks again!
i have added this topic to the ptg etherpad in https://etherpad.opendev.org/p/nova-wallaby-ptg line 170 ish as part of the numa in placement section.

> 
> Eric