<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>A few thoughts:</div>
<div><br>
</div>
<ol>
<li>Are you actively swapping pages in and out? I'm going to guess not since you have a lot of RAM available and the swap space hasn't been freed. This would lead me to believe there was an event in the past that forced the kernel to move some pages out to
swap space but they haven't been needed, so they've been left there.. The package 'dstat' is helpful to determine if you're paging.</li><li>Do you overcommit your ram? If so, you may have booted a RAM heavy VM at one point that pushed another process into swap. Perhaps look at lowering the overcommit ratio.</li><li>You can lower the how aggressive the kernel moves things in and out of swap space. The file /proc/sys/vm/swapiness controls this within the kernel. You can try lowering it which will cause the kernel to not be so aggressive about moving things in to swap.
0 is the lowest value.</li></ol>
<div>Paging in and out is what kills performance. Stuff in swap doesn't generally hurt performance until it needs to be accessed and has to be moved back into RAM. Constant paging in and out is a huge problem especially on a hypervisor with busy disks though.</div>
<div><br>
</div>
<div><br>
</div>
<div><span style="font-size: medium; ">Ryan Richard | RHCA</span></div>
<div>
<div><span style="font-size: medium; ">Rackspace US, Inc.</span></div>
<br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Joe Topjian <<a href="mailto:joe.topjian@cybera.ca">joe.topjian@cybera.ca</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, October 16, 2013 3:46 PM<br>
<span style="font-weight:bold">To: </span>Samuel Winchenbach <<a href="mailto:swinchen@gmail.com">swinchen@gmail.com</a>><br>
<span style="font-weight:bold">Cc: </span>"<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>" <<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [Openstack-operators] Strange memory usage<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">Hello,
<div><br>
</div>
<div>I'm not exactly sure what the next steps would be. :( </div>
<div><br>
</div>
<div>If your qemu processes are being reported as the high-swap consumers, it might be a more site-specific or instance-specific issue rather than something with OpenStack itself.</div>
<div><br>
</div>
<div>From the process string you gave, it looks like a normal running kvm instance to me... I can't see anything unusual. </div>
<div><br>
</div>
<div>If these instances aren't doing anything that would cause them to use a high amount of memory to the point of swapping, I'd start looking into KVM and reasons why it would want to swap.</div>
<div><br>
</div>
<div>Joe</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Oct 16, 2013 at 2:25 PM, Samuel Winchenbach <span dir="ltr">
<<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_default" style="font-family:'courier new',monospace">Hi Joe. I have deleted that instance (it was no longer needed). Here is the next one down on the list:</div>
<div class="gmail_default" style="font-family:'courier new',monospace"><br>
</div>
<div class="gmail_default"><font face="courier new,monospace">119 6560 3.4 2.6 9247872 1749336 ? Sl Oct15 41:33 qemu-system-x86_64 -machine accel=kvm:tcg -name instance-000001b3 -S -M pc-i440fx-1.4 -cpu Opteron_G3,+nodeid_msr,+wdt,+skinit,+ibs,+osvw,+3dnowprefetch,+cr8legacy,+extapic,+cmp_legacy,+3dnow,+3dnowext,+pdpe1gb,+fxsr_opt,+mmxext,+ht,+vme
-m 4096 -smp 4,sockets=4,cores=1,threads=1 -uuid 9627e75b-8d3b-4482-be43-8aba176a3f1f -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=2013.1.3,serial=49434d53-0200-9010-2500-109025007800,uuid=9627e75b-8d3b-4482-be43-8aba176a3f1f
-no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-000001b3.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
-drive file=/os-grizzly/nova/instances/9627e75b-8d3b-4482-be43-8aba176a3f1f/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/os-grizzly/nova/instances/9627e75b-8d3b-4482-be43-8aba176a3f1f/disk.swap,if=none,id=drive-virtio-disk1,format=qcow2,cache=none
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=31 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:45:63:f0,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/os-grizzly/nova/instances/9627e75b-8d3b-4482-be43-8aba176a3f1f/console.log
-device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc
<a href="http://0.0.0.0:6" target="_blank">0.0.0.0:6</a> -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6</font><br>
</div>
</div>
<div class="HOEnZb">
<div class="h5">
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Oct 16, 2013 at 4:07 PM, Joe Topjian <span dir="ltr">
<<a href="mailto:joe.topjian@cybera.ca" target="_blank">joe.topjian@cybera.ca</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hello,
<div><br>
</div>
<div>"ps auxww | grep 19587", or some variation of the command, should give you the full command. Can you trace it back to an individual instance?</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Joe</div>
</div>
<div class="gmail_extra">
<div><br>
<br>
<div class="gmail_quote">On Wed, Oct 16, 2013 at 11:50 AM, Samuel Winchenbach <span dir="ltr">
<<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_default" style="font-family:'courier new',monospace">Joe: <a href="http://pastie.org/pastes/8407072/text" target="_blank">http://pastie.org/pastes/8407072/text</a></div>
<div class="gmail_default" style="font-family:'courier new',monospace"><br>
</div>
<div class="gmail_default" style="font-family:'courier new',monospace">It really looks like qemu-system-x86 is the culprit. mysqld (Percona/Galera) is using a fair amount too.</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
</div>
<div>-- <br>
<div dir="ltr">Joe Topjian
<div>Systems Architect</div>
<div>Cybera Inc.</div>
<div><br>
</div>
<div><a href="http://www.cybera.ca" target="_blank">www.cybera.ca</a></div>
<div><br>
</div>
<div><font color="#666666"><span>Cybera</span><span> is a not-for-profit organization that works to spur and support innovation, for the economic benefit of Alberta, through the use of cyberinfrastructure.</span></font></div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">Joe Topjian
<div>Systems Architect</div>
<div>Cybera Inc.</div>
<div><br>
</div>
<div><a href="http://www.cybera.ca" target="_blank">www.cybera.ca</a></div>
<div><br>
</div>
<div><font color="#666666"><span>Cybera</span><span> is a not-for-profit organization that works to spur and support innovation, for the economic benefit of Alberta, through the use of cyberinfrastructure.</span></font></div>
</div>
</div>
</div>
</div>
</span>
</body>
</html>