<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 21, 2022 at 6:21 PM Sean Mooney <<a href="mailto:smooney@redhat.com">smooney@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Mon, 2022-02-21 at 12:24 +0500, Ammad Syed wrote:<br>
> Hi,,<br>
> <br>
> I am having trouble with my compute node that nova-compute and ovs process<br>
> are being killed by OOM. I have alot memory available in system.<br>
> <br>
> # free -g<br>
> total used free shared buff/cache<br>
> available<br>
> Mem: 1006 121 881 0 2<br>
> 879<br>
> Swap: 7 0 7<br>
> <br>
> But I am seeing process are being killed in dmesg.<br>
> <br>
> [Sat Feb 19 03:46:26 2022] Memory cgroup out of memory: Killed process<br>
> 2080898 (ovs-vswitchd) total-vm:9474284kB, anon-rss:1076384kB,<br>
> file-rss:11700kB, shmem-rss:0kB, UID:0 pgtables:2776kB oom_score_adj:0<br>
> [Sat Feb 19 03:47:01 2022] Memory cgroup out of memory: Killed process<br>
> 2081218 (ovs-vswitchd) total-vm:9475332kB, anon-rss:1096988kB,<br>
> file-rss:11700kB, shmem-rss:0kB, UID:0 pgtables:2780kB oom_score_adj:0<br>
> [Sat Feb 19 03:47:06 2022] Memory cgroup out of memory: Killed process<br>
> 2081616 (ovs-vswitchd) total-vm:9473252kB, anon-rss:1073052kB,<br>
> file-rss:11700kB, shmem-rss:0kB, UID:0 pgtables:2784kB oom_score_adj:0<br>
> [Sat Feb 19 03:47:16 2022] Memory cgroup out of memory: Killed process<br>
> 2081940 (ovs-vswitchd) total-vm:9471236kB, anon-rss:1070920kB,<br>
> file-rss:11700kB, shmem-rss:0kB, UID:0 pgtables:2776kB oom_score_adj:0<br>
> [Sat Feb 19 03:47:16 2022] Memory cgroup out of memory: Killed process 6098<br>
> (nova-compute) total-vm:3428356kB, anon-rss:279920kB, file-rss:9868kB,<br>
> shmem-rss:0kB, UID:64060 pgtables:1020kB oom_score_adj:0<br>
> [Mon Feb 21 11:15:08 2022] Memory cgroup out of memory: Killed process<br>
> 2082296 (ovs-vswitchd) total-vm:9475372kB, anon-rss:1162636kB,<br>
> file-rss:11700kB, shmem-rss:0kB, UID:0 pgtables:2864kB oom_score_adj:0<br>
> <br>
> Any advice on how to fix this ? Also any best practices document on<br>
> configuring memory optimizations in nova compute node.<br>
so one thing to note is that the OOM reaper service runs per numa node<br>
so the gloable free memory values are not really what you need to look at.<br>
<br>
croups/systemd also provide ways to limit the max memory a process/cgroup tree can consume<br>
<br>
so your first stp shoudl be to determin if the oom event was triggered by exaustign the memroy in<br>
a specific numa node or if you are hitting a differnt cgroup memory limit.<br></blockquote><div><br></div><div>As the logs suggest, it looks like the memory of cgroup is exhausted. The memory of system.slice cgroup is 4G and user.slice is 2G by default. I have increased system.slice to 64GB.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
in terms fo how to optimisze memroy it really depend on what your end goal is here.<br>
<br>
obviously we do not want ovs or nova-compute to be killsed in general.<br>
the virtaul and reseden memory for ovs is in the 10 to 1 GB range so that is not excessively large.<br>
<br>
that also should not be anywhere near the numa limit but if you were incorrectly creating numa affinged<br>
vms without seting hw:mem_page_size via nova then that could perhaps tirgger out of memory events<br></blockquote><div><br></div><div>Currently I am not using any page_size in my flavors. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
effectivly with openenstack if your vm is numa affined eiter explictly via hw:numa_nodes extra specs or implictly via<br>
cpu pinning or otherwise then you must define that the memory is tracked using the numa aware path which requires<br>
you do defien hw:mem_page_size in the flaovr or hw_mem_page_size in the image.<br></blockquote><div><br></div><div>I am only using CPU soft pinning (vcpu placement) i.e cpu_shared_set. However I have only configured hw:cpu_sockets='2' in flavors to make two sockets for VM. This helps in effective cpu utilization in windows hosts. However in VM I can only see one numa node of memory. Will this possibly cause trouble ?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
if you do not want ot use hugepages hw:mem_page_size=small is a good default but just be aware that if the<br>
vm has a numa topology then memory over subsction is not supproted in openstack. i.e. you cannot use cpu pinning<br>
or any other feature that requriees a numa topoplogy like virtual persistent memory and also use memory over subscription.<br></blockquote><div><br></div><div>Got it.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
assuming these event do not corralate with vm boots then i woudl investiagte the cgroup memory limits you set on teh ovs and compute service<br>
cgroups. if they are correatlted with vm boots check if the vm is numa affined and if it is which page size is requested.<br>
ir its hw:mem_page_size=small then you might need to use the badly named reserved_huge_pages config option to reserve 4k pages for the host per numa<br>
node.<br>
<a href="https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.reserved_huge_pages" rel="noreferrer" target="_blank">https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.reserved_huge_pages</a><br>
e.g. reserve 4G on node 0 and 1<br>
reserved_huge_pages = node:0,size:4,count:1048576<br>
reserved_huge_pages = node:1,size:4,count:1048576<br>
the sum of all the 4k page size reservation should equal the value of reserved_host_memory_mb<br>
<a href="https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.reserved_host_memory_mb" rel="noreferrer" target="_blank">https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.reserved_host_memory_mb</a></blockquote><div><br></div><div>Currently I have <a href="https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.reserved_host_memory_mb" rel="noreferrer" target="_blank">reserved_host_memory_mb</a> with 64GB memory reserved for host and no oversubscription i.e memory over provisioning factor set to 1.0 in compute nodes.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
<br>
this is only really needed where you are usign numa instances since reserved_host_memory_mb dose not account<br>
for the host numa toplogy so it will not prevent the numa node form being exausted.<br>
<br>
if you are usign a amd epyc system and the NPS( numa per socket) bios option set to say 4 or 8 <br>
on a dual or single socket system respecivly then the the 1TB of ram you have on the host woudl be devided<br>
into numa nodes of 128GB each which is very close to the 121 used you have when you star seeing issues.<br></blockquote><div><br></div><div>Yes I am using epyc system and checked in BIOS NPS is set to 1. <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
nova currently tries to fill the numa nodes in order when you have numa instances too which causes the OOM issue to manifest much sooner then<br>
people often exepct due to the per numa nature of OOM reaper.<br>
<br>
that may not help you in your case but that is how i would approch tracking down this issue.<br></blockquote><div><br></div><div>This indeed helped a lot. It's been last 18 hours, and no OOM Killed observed till now.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> <br>
> Ammad<br>
<br>
</blockquote></div><br clear="all"><div><br></div></div>