<div dir="ltr">In addition to what Kris said, here are two other ways to see memory usage of qemu processes:<div><br></div><div>The first is with "nova diagnostics <uuid>". By default this is an admin-only command.</div><div><br></div><div>The second is by running "virsh dommemstat <instance-id>" directly on the compute node.</div><div><br></div><div>Note that it's possible for the used memory (rss) to be greater than the available memory. When this happens, I believe it is due to the qemu process consuming more memory than the actual vm itself -- so the instance has consumed all available memory, plus qemu itself needs some to function properly. Someone please correct me if I'm wrong.</div><div><br></div><div>Hope that helps,</div><div>Joe</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 23, 2015 at 10:12 AM, Kris G. Lindgren <span dir="ltr"><<a href="mailto:klindgren@godaddy.com" target="_blank">klindgren@godaddy.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div>
<div style="font-family:Calibri,sans-serif;font-size:14px;color:rgb(0,0,0)">
Not totally sure I am following - the output of free would help a lot.</div>
<div style="font-family:Calibri,sans-serif;font-size:14px;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,sans-serif;font-size:14px;color:rgb(0,0,0)">
However, the number you should be caring about is free +buffers/cache. The reason for you discrepancy is you are including the cached in memory file system content that linux does in order to improve performance. On boxes with enough ram this can easily be
60+ GB. When the system comes under memory pressure (from applications or the kernel wanting more memory) the kernel will remove any cached filesystem items to free up memory for processes. This link [1] has a pretty good description of what I am talking
about.</div>
<div style="font-family:Calibri,sans-serif;font-size:14px;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,sans-serif;font-size:14px;color:rgb(0,0,0)">
Either way, if you want to test to make sure this is a case of filesystem caching you can run:</div>
<div style="font-family:Calibri,sans-serif;font-size:14px">
<p style="margin:0px;font-size:12px;font-family:Menlo">echo 3 > /proc/sys/vm/drop_caches </p>
</div>
<div style="font-family:Calibri,sans-serif;font-size:14px;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,sans-serif;font-size:14px;color:rgb(0,0,0)">
Which will tell linux to drop all filesystem cache from memory, and I bet a ton of your memory will show up. Note: in doing so - you will affect the performance of the box. Since what use to be an in memory lookup will now have to go to the filesystem. However,
over time the cache will re-establish. You can find more examples of how caching interacts with other part of the linux memory system here: [2]</div>
<div style="font-family:Calibri,sans-serif;font-size:14px;color:rgb(0,0,0)">
<br>
</div>
<div><font face="Calibri,sans-serif">To your question about qemu process.. </font>If you use ps aux, the columns VSZ and RSS will tell you are wanting. VSZ is the virtual size (how much memory the process has asked the kernel for). RSS is resident set side,
or that actual amount of non-swapped memory the process is using. </div>
<div style="font-family:Calibri,sans-serif;font-size:14px;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,sans-serif;font-size:14px;color:rgb(0,0,0)">
[1] - <a href="http://www.linuxatemyram.com/" target="_blank">http://www.linuxatemyram.com/</a> </div>
<div style="font-family:Calibri,sans-serif;font-size:14px;color:rgb(0,0,0)">
[2] - <a href="http://www.linuxatemyram.com/play.html" target="_blank">http://www.linuxatemyram.com/play.html</a></div>
<div style="font-family:Calibri,sans-serif;font-size:14px;color:rgb(0,0,0)">
<div>
<div>____________________________________________</div>
<div> </div>
<div>Kris Lindgren</div>
<div>Senior Linux Systems Engineer</div>
<div>GoDaddy, LLC.</div>
</div>
<div><br>
</div>
</div>
</div>
<div style="font-family:Calibri,sans-serif;font-size:14px;color:rgb(0,0,0)">
<br>
</div>
<span style="font-family:Calibri,sans-serif;font-size:14px;color:rgb(0,0,0)">
<div style="font-family:Calibri;font-size:11pt;text-align:left;color:black;BORDER-BOTTOM:medium none;BORDER-LEFT:medium none;PADDING-BOTTOM:0in;PADDING-LEFT:0in;PADDING-RIGHT:0in;BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;PADDING-TOP:3pt">
<span style="font-weight:bold">From: </span>Mike Leong <<a href="mailto:leongmzlist@gmail.com" target="_blank">leongmzlist@gmail.com</a>><br>
<span style="font-weight:bold">Date: </span>Tuesday, June 23, 2015 at 9:44 AM<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.openstack.org</a>" <<a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>[Openstack-operators] Instance memory overhead<br>
</div><div><div class="h5">
<div><br>
</div>
<div>
<div>
<div dir="ltr">My instances are using much more memory that expected. The amount free memory (free + cached) is under 3G on my servers even though the compute nodes are configured to reserve 32G.
<div><br>
</div>
<div>Here's my setup:
<div>Release: Ice House<br>
</div>
<div>Server mem: 256G</div>
<div>Qemu version: 2.0.0+dfsg-2ubuntu1.1</div>
<div>Networking: Contrail 1.20</div>
<div>Block storage: Ceph 0.80.7</div>
<div>Hypervisor OS: Ubuntu 12.04</div>
<div>memory over-provisioning is disabled</div>
<div>kernel version: 3.11.0-26-generic</div>
<div><br>
</div>
<div>On nova.conf</div>
<div>
<div>reserved_host_memory_mb = 32768</div>
</div>
<div><br>
</div>
<div>Info on instances:</div>
<div>- root volume is file backed (qcow2) on the hypervisor local storage</div>
<div>- each instance has a rbd volume mounted from Ceph</div>
<div>- no swap file/partition</div>
<div><br>
</div>
<div>I've confirmed, via nova-compute.log, that nova is respecting the reserved_host_memory_mb directive and is not over-provisioning. On some hypervisors, nova-compute says there's 4GB available for use even though the OS has less that 4G left (free +cached)!</div>
<div><br>
</div>
<div>I've also summed up the memory from /etc/libvir/qemu/*.xml files and the total looks good.<br>
</div>
<div><br>
</div>
<div>Each hypervisor hosts about 45-50 instances.</div>
<div><br>
</div>
<div>Is there good way to calculate the actual usage of each QEMU process?</div>
<div><br>
</div>
<div>PS: I've tried free, summing up RSS, and smem but none of them can tell me where's the missing mem.</div>
<div><br>
</div>
<div>thx</div>
<div>mike</div>
</div>
</div>
</div>
</div>
</div></div></span>
</div>
<br>_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br></blockquote></div><br></div>