<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<div class="moz-cite-prefix">On 2017-03-23 15:15, Edmund Rhudy
(BLOOMBERG/ 120 PARK) wrote:<br>
</div>
<blockquote
cite="mid:58D41EC2020905000039011C_0_85604@msllnjpmsgsv06"
type="cite">
<title></title>
<!-- rte-version 0.2 9947551637294008b77bce25eb683dac -->
<div class="rte-style-maintainer" style="white-space: pre-wrap; font-size: small; font-family: 'Courier New', Courier, 'BB.FixedWidth';" data-color="global-default" bbg-color="default" data-bb-font-size="medium" bbg-font-size="medium" bbg-font-family="fixed-width">What sort of memory overcommit value are you running Nova with? The scheduler looks at an instance's reservation rather than how much memory is actually being used by QEMU when making a decision, as far as I'm aware (but please correct me if I am wrong on this point). If the HV has 128GB of memory, the instance has a reservation of 96GB, you have 16GB reserved via reserved_host_memory_mb, ram_allocation_ratio is set to 1.0, and you try to launch an instance from a flavor with 32GB of memory, it will fail to pass RamFilter in the scheduler and the scheduler will not consider it a valid host for placement. (I am assuming you are using FilterScheduler still, as I know nothing about the new placement API or what parts of it do and don't work in Newton.)
</div>
</blockquote>
The overcommit value is set to 1.5 in the scheduler. It's not the
scheduler that was preventing the instance from being provisionned,
it was qemu returning that there was not enough ram when libvirt was
trying to provision the instance (that error was not handled well by
openstack, btw, but that's something else). So the instance does
pass every filter. It just ends up in error when getting provisioned
in the compute node because of a lack of ram, with the actual full
error message only visible in the QEMU logs.<br>
<blockquote
cite="mid:58D41EC2020905000039011C_0_85604@msllnjpmsgsv06"
type="cite">
<div class="rte-style-maintainer" style="white-space: pre-wrap; font-size: small; font-family: 'Courier New', Courier, 'BB.FixedWidth';" data-color="global-default" bbg-color="default" data-bb-font-size="medium" bbg-font-size="medium" bbg-font-family="fixed-width"><div>
</div><div>As far as why the memory didn't automatically get reclaimed, maybe KVM will only reclaim empty pages and memory fragmentation in the guest prevented it from doing so? It might also not actively try to reclaim memory unless it comes under pressure to do so, because finding empty pages and returning them to the host may be a somewhat time-consuming operation.
</div></div>
</blockquote>
<br>
That's entirely possible, but according to the doc, libvirt is
supposed to have a memory balloon function that does the operation
of reclaiming empty pages from guest processes, or so I understand.
Now, how this function works is not exactly clear to me, or even if
nova uses it or not. Another user suggested it might not be
automatic, which is in accordance to what you're conjecturing.<br>
<blockquote
cite="mid:58D41EC2020905000039011C_0_85604@msllnjpmsgsv06"
type="cite">
<div class="rte-style-maintainer" style="white-space: pre-wrap; font-size: small; font-family: 'Courier New', Courier, 'BB.FixedWidth';" data-color="global-default" bbg-color="default" data-bb-font-size="medium" bbg-font-size="medium" bbg-font-family="fixed-width"><div><div class="rte-style-maintainer" data-bb-font-size="medium" bbg-color="default" bbg-font-family="fixed-width" bbg-font-size="medium" style="font-size: small; font-family: 'Courier New', Courier, 'BB.FixedWidth';">
<div class="bbg-rte-fold-content" data-header="From: jp.methot@planethoster.info" data-digest="From: jp.methot@planethoster.info" style=""><div class="bbg-rte-fold-summary">From: <a class="moz-txt-link-abbreviated" href="mailto:jp.methot@planethoster.info">jp.methot@planethoster.info</a> </div><div>Subject: Re: [Openstack-operators] Memory usage of guest vms, ballooning and nova
</div></div><blockquote>Hi,
This is indeed linux, CentOS 7 to be more precise, using qemu-kvm as
hypervisor. The used ram was in the used column. While we have made
adjustments by moving and resizing the specific guest that was using 96
GB (verified in top), the ram usage is still fairly high for the amount
of allocated ram.
Currently the ram usage looks like this :
total used free shared buff/cache
available
Mem: 251G 190G 60G 42M 670M 60G
Swap: 952M 707M 245M
I have 188.5GB of ram allocated to 22 instances on this node. I believe
it's unrealistic to think that all these 22 instances have cached/are
using up all their ram at this time.
On 2017-03-23 13:07, Kris G. Lindgren wrote:
> Sorry for the super stupid question.
>
> But if this is linux are you sure that the memory is not actually being consumed via buffers/cache?
>
> free -m
> total used free shared buff/cache available
> Mem: 128751 27708 2796 4099 98246 96156
> Swap: 8191 0 8191
>
> Shows that of 128GB 27GB is used, but buffers/cache consumes 98GB of ram.
>
> ___________________________________________________________________
> Kris Lindgren
> Senior Linux Systems Engineer
> GoDaddy
>
> On 3/23/17, 11:01 AM, "Jean-Philippe Methot" <<a moz-do-not-send="true" spellcheck="false" bbg-destination="mailto:rte:bind" href="mailto:jp.methot@planethoster.info" data-destination="mailto:rte:bind">jp.methot@planethoster.info</a>> wrote:
>
> Hi,
>
> Lately, on my production openstack Newton setup, I've ran into a
> situation that defies my assumptions regarding memory management on
> Openstack compute nodes and I've been looking for explanations.
> Basically, we had a VM with a flavor that limited it to 96 GB of ram,
> which, to be quite honest, we never thought we could ever reach. This is
> a very important VM where we wanted to avoid running out of memory at
> all cost. The VM itself generally uses about 12 GB of ram.
>
> We were surprised when we noticed yesterday that this VM, which has been
> running for several months, was using all its 96 GB on the compute host.
> Despite that, in the guest, the OS was indicating a memory usage of
> about 12 GB. The only explanation I see to this is that at some point in
> time, the host had to allocate all the 96GB of ram to the VM process and
> it never took back the allocated ram. This prevented the creation of
> more guests on the node as it was showing it didn't have enough memory left.
>
> Now, I was under the assumption that memory ballooning was integrated
> into nova and that the amount of allocated memory to a specific guest
> would deflate once that guest did not need the memory. After
> verification, I've found blueprints for it, but I see no trace of any
> implementation anywhere.
>
> I also notice that on most of our compute nodes, the amount of ram used
> is much lower than the amount of ram allocated to VMs, which I do
> believe is normal.
>
> So basically, my question is, how does openstack actually manage ram
> allocation? Will it ever take back the unused ram of a guest process?
> Can I force it to take back that ram?
>
> --
> Jean-Philippe Méthot
> Openstack system administrator
> PlanetHoster inc.
> <a moz-do-not-send="true" bbg-destination="rte:bind" spellcheck="false" href="http://www.planethoster.net" data-destination="rte:bind">www.planethoster.net</a>
>
>
> _______________________________________________
> OpenStack-operators mailing list
> <a moz-do-not-send="true" spellcheck="false" bbg-destination="mailto:rte:bind" href="mailto:OpenStack-operators@lists.openstack.org" data-destination="mailto:rte:bind">OpenStack-operators@lists.openstack.org</a>
> <a moz-do-not-send="true" bbg-destination="rte:bind" spellcheck="false" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" data-destination="rte:bind">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a>
>
>
--
Jean-Philippe Méthot
Openstack system administrator
PlanetHoster inc.
<a moz-do-not-send="true" bbg-destination="rte:bind" spellcheck="false" href="http://www.planethoster.net" data-destination="rte:bind">www.planethoster.net</a>
_______________________________________________
OpenStack-operators mailing list
<a moz-do-not-send="true" spellcheck="false" bbg-destination="mailto:rte:bind" href="mailto:OpenStack-operators@lists.openstack.org" data-destination="mailto:rte:bind">OpenStack-operators@lists.openstack.org</a>
<a moz-do-not-send="true" bbg-destination="rte:bind" spellcheck="false" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" data-destination="rte:bind">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a>
</blockquote>
</div></div></div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Jean-Philippe Méthot
Openstack system administrator
PlanetHoster inc.
<a class="moz-txt-link-abbreviated" href="http://www.planethoster.net">www.planethoster.net</a>
</pre>
</body>
</html>