On Sat, 2019-04-13 at 02:12 +0000, Manuel Sopena Ballesteros wrote:

Hi,

considering my hosts have numa architecture.

what is the difference between "numa topology" a numa toplogy refers to the topology of your memory contorlers and therefore effect any operation involving memory acess includeing but not limited to memroy mapped io to hardware devices such as nics or gpus.

if we say a vm or host has 2 numa node that means that it has two indepenent meoroy controlers. this is pretty common these these days. by having multiple memory controlers the total memory of bandwith of the system is doubled however the distance (the phyical lenght of the copper traces) between a cpu and a memory contoler or a device and a memory contoler will be differenet between each of the two memory contolers as the cpu core or device will phycially postioned closer to one of them. basicly because the speed of light is not infiite if you have more then 1 memory contorler then acessing different parts of memory will have different latency depening on the memroy adress, the device that is doing the access and the contoler that that device is closets and if that contoler is the one that manages that area of memory.. so put simply when we are talking about numa we are talking about memory, and when we talk about numa affinity we are talking about minimising that latency between a device(cpu or mmio device like a nic) and that memeory by selecting a device who local memory contoler "owns" that memory therefore resulting in the shortest path. to do this in openstack if you add hw:numa_node=1 we will ensure the cpus and pci passthough devices that are used by a guest come the same numa node(memory contorler) as the ram for that guest, thereby optimising for the minium memory latency.

and "cpu pinning"? would "cpu pinning" make any difference if "numa topology" is already in place? cpu pinning can althouth the effect will be less then from enableing a numa toplogy for io intesive workloads.

when you enable cpu pinning in opentack we also create a numa topology of 1 implcitly. you can specify multiple numa node but as a result you get all the benifts of numa affity automatically when you enable cpu pinning. we should have kept these seperate but that is the legacy we have today. where cpu pinning helps is if your workload is compute bound. when you enable a numa afinity in openstack we confine the guest cpus to float over the cpus with a numa node that is local to the momory of the guest. while this soft pinning results in a perfromance boost for io operattions it dose not imporve cpu bound performacne. enabling cpu pinning will pin each guest cpu to a host hardware thread and addtionaly prevent oversubciption by other pinned cpus. i.e. we will never pin to guest vcpus to the same hardawere thread. with numa the guest not only floats over host cpus but the host cpus can be over subscribed.

thank you very much

one other think you should look into is the use of huge page memory. like cpu pinnign enableing hugepages creats an implict numa toplogy of 1 numa node if you do not override it with hw:numa_nodes. useing hugepage memory for the guest via hw:mem_page_size will futher improve guest memory intensive workload by miniuming tlb cache misses optimising virtual memory to phyical memory address traslation.

also as i have mentioned in the past notices like the one below raise legal ambigurity of if they are allowed to be sent to the list. with a strict reading it woudl not allow the mailinglist to forward the email. i am choosing to read this as all subsribers or the mailing list are the intended recipients. such disclaimer are almost allway unenforceabel and are effectivly spam so while it is not strictly against our mailing list etiquette https://wiki.openstack.org/wiki/MailingListEtiquette it is still best avoided.

NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.