<div dir="ltr">How much are you reserving for Openstack vs the VM?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 25, 2022 at 2:19 PM hai wu <<a href="mailto:haiwu.us@gmail.com">haiwu.us@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Understand. The same concern is also raised in the following redhat<br>
KB: <a href="https://access.redhat.com/solutions/4670201" rel="noreferrer" target="_blank">https://access.redhat.com/solutions/4670201</a>.<br>
<br>
But we could also protect some critical openstack services, like<br>
neutron, libvirtd, via the same way by setting OOMScoreAdjust for<br>
those to be -1000. If we do that, we should probably be ok. We protect<br>
both critical openstack services, and all openstack VMs in this way.<br>
<br>
On Thu, Jul 21, 2022 at 6:42 AM Sean Mooney <<a href="mailto:smooney@redhat.com" target="_blank">smooney@redhat.com</a>> wrote:<br>
><br>
> On Wed, 2022-07-20 at 20:25 -0500, hai wu wrote:<br>
> > You are correct, there's no way to set OOMScoreAdjust for<br>
> > machine.slice. It errored out when trying to do that, with "Unknown<br>
> > assignment" error..<br>
><br>
> if you mess with the cgroups behind novas back then any hope of support you have with<br>
> your vendor or updstream is gone.<br>
><br>
> you shoudl really find out why your running out of memroy.<br>
><br>
> it ususllay means you have not configured nova and the host correctly.<br>
><br>
> most often this hapens becuase peopel use cpu pinning wiht out enable per<br>
> numa node memory memory tracking by setting a page size.<br>
><br>
> it also could be because you have not allcoated enough swap.<br>
><br>
> so before you try to adjust things with cgroups yourslef or explore other options you shoudl determin why<br>
> the host is runnign out of memroy.<br>
><br>
> if you prevent ti from kill the gues i have see it kill ovs or nova iteslf before where the guest were<br>
> unkillable or unlkely to be killed because they used hugepages.<br>
><br>
> so you will likely jsut shift the problem else where that will be more impactful.<br>
><br>
> ><br>
> > On Wed, Jul 20, 2022 at 6:48 PM hai wu <<a href="mailto:haiwu.us@gmail.com" target="_blank">haiwu.us@gmail.com</a>> wrote:<br>
> > ><br>
> > > In this case there's no memory oversubscription. This oom killer event<br>
> > > happened when we did "swapoff -a; swapon -a" to push processes in swap<br>
> > > back to memory, which is very strange.<br>
> > ><br>
> > > On Wed, Jul 20, 2022 at 6:39 PM Clark Boylan <<a href="mailto:cboylan@sapwetik.org" target="_blank">cboylan@sapwetik.org</a>> wrote:<br>
> > > ><br>
> > > > On Wed, Jul 20, 2022, at 4:04 PM, hai wu wrote:<br>
> > > > > After installing some systemd package, and starting up machine.slice,<br>
> > > > > systemd-machined, and hard rebooting the vm from openstack side, I<br>
> > > > > could now see the VM showing up under machine.slice. all vms were<br>
> > > > > showing up under libvirtd.service, which is under system.slice.<br>
> > > > ><br>
> > > > > What are the benefits of running libvirt managed guest instances under<br>
> > > > > machine.slice?<br>
> > > ><br>
> > > > You can use machine.slice to set system resource options that each sub slice inherits. Those options are documented at <a href="https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html#" rel="noreferrer" target="_blank">https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html#</a> (per my earlier link <a href="https://www.freedesktop.org/software/systemd/man/systemd.slice.html" rel="noreferrer" target="_blank">https://www.freedesktop.org/software/systemd/man/systemd.slice.html</a>). I don't see OOMScoreAdjust listed there so I am unsure if you can actually set it via this method.<br>
> > > ><br>
> > > > That all said, if you are oversubscribing memory this is likely to always be an issue. If you adjust the oom score for your VMs then the oomkiller is just going to find other victims to kill. Losing your nova compute agent or NetworkManager or iscsid may be just as problematic. Instead, I suspect that you may need to stop oversubscribing memory.<br>
> > > ><br>
> > > > ><br>
> > > > > On Wed, Jul 20, 2022 at 5:53 PM Clark Boylan <<a href="mailto:cboylan@sapwetik.org" target="_blank">cboylan@sapwetik.org</a>> wrote:<br>
> > > > > ><br>
> > > > > > On Wed, Jul 20, 2022, at 3:17 PM, hai wu wrote:<br>
> > > > > > > Is there any configuration file that is needed to ensure guest domains<br>
> > > > > > > are under systemd machine.slice? not seeing anything under<br>
> > > > > > > machine.slice ..<br>
> > > > > ><br>
> > > > > > I think that <a href="https://www.freedesktop.org/software/systemd/man/systemd.slice.html" rel="noreferrer" target="_blank">https://www.freedesktop.org/software/systemd/man/systemd.slice.html</a> and <a href="https://libvirt.org/cgroups.html" rel="noreferrer" target="_blank">https://libvirt.org/cgroups.html</a> covers this for libvirt managed VMs.<br>
> > > > > ><br>
> > > > > > ><br>
> > > > > > > On Wed, Jul 20, 2022 at 3:33 PM Dmitriy Rabotyagov<br>
> > > > > > > <<a href="mailto:noonedeadpunk@gmail.com" target="_blank">noonedeadpunk@gmail.com</a>> wrote:<br>
> > > > > > > ><br>
> > > > > > > > I believe you can decrease OOMScoreAdjust for systemd machines.slice, under which guest domains are to reduce chances of oom killing them.<br>
> > > > > > > ><br>
> > > > > > > > ср, 20 июл. 2022 г., 21:52 hai wu <<a href="mailto:haiwu.us@gmail.com" target="_blank">haiwu.us@gmail.com</a>>:<br>
> > > > > > > > ><br>
> > > > > > > > > nova hypervisor sometimes oom would kill some openstack guests.<br>
> > > > > > > > ><br>
> > > > > > > > > Is it possible to not allow kernel to oom kill any openstack guests?<br>
> > > > > > > > > ram is not oversubscribed much ..<br>
> > > > > > > > ><br>
> > > > > ><br>
> > > ><br>
> ><br>
><br>
<br>
</blockquote></div>