<div dir="ltr">Hello,<div><br></div><div>I'm not exactly sure what the next steps would be. :( </div><div><br></div><div>If your qemu processes are being reported as the high-swap consumers, it might be a more site-specific or instance-specific issue rather than something with OpenStack itself.</div>
<div><br></div><div>From the process string you gave, it looks like a normal running kvm instance to me... I can't see anything unusual. </div><div><br></div><div>If these instances aren't doing anything that would cause them to use a high amount of memory to the point of swapping, I'd start looking into KVM and reasons why it would want to swap.</div>
<div><br></div><div>Joe</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Oct 16, 2013 at 2:25 PM, Samuel Winchenbach <span dir="ltr"><<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:'courier new',monospace">Hi Joe. I have deleted that instance (it was no longer needed). Here is the next one down on the list:</div>
<div class="gmail_default" style="font-family:'courier new',monospace">
<br></div><div class="gmail_default"><font face="courier new, monospace">119 6560 3.4 2.6 9247872 1749336 ? Sl Oct15 41:33 qemu-system-x86_64 -machine accel=kvm:tcg -name instance-000001b3 -S -M pc-i440fx-1.4 -cpu Opteron_G3,+nodeid_msr,+wdt,+skinit,+ibs,+osvw,+3dnowprefetch,+cr8legacy,+extapic,+cmp_legacy,+3dnow,+3dnowext,+pdpe1gb,+fxsr_opt,+mmxext,+ht,+vme -m 4096 -smp 4,sockets=4,cores=1,threads=1 -uuid 9627e75b-8d3b-4482-be43-8aba176a3f1f -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=2013.1.3,serial=49434d53-0200-9010-2500-109025007800,uuid=9627e75b-8d3b-4482-be43-8aba176a3f1f -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-000001b3.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/os-grizzly/nova/instances/9627e75b-8d3b-4482-be43-8aba176a3f1f/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/os-grizzly/nova/instances/9627e75b-8d3b-4482-be43-8aba176a3f1f/disk.swap,if=none,id=drive-virtio-disk1,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=31 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:45:63:f0,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/os-grizzly/nova/instances/9627e75b-8d3b-4482-be43-8aba176a3f1f/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc <a href="http://0.0.0.0:6" target="_blank">0.0.0.0:6</a> -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6</font><br>
</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Oct 16, 2013 at 4:07 PM, Joe Topjian <span dir="ltr"><<a href="mailto:joe.topjian@cybera.ca" target="_blank">joe.topjian@cybera.ca</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hello,<div><br></div><div>"ps auxww | grep 19587", or some variation of the command, should give you the full command. Can you trace it back to an individual instance?</div>
<div><br></div><div>Thanks,</div>
<div>Joe</div></div><div class="gmail_extra"><div><br><br><div class="gmail_quote">On Wed, Oct 16, 2013 at 11:50 AM, Samuel Winchenbach <span dir="ltr"><<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:'courier new',monospace">Joe: <a href="http://pastie.org/pastes/8407072/text" target="_blank">http://pastie.org/pastes/8407072/text</a></div>
<div class="gmail_default" style="font-family:'courier new',monospace">
<br></div><div class="gmail_default" style="font-family:'courier new',monospace">It really looks like qemu-system-x86 is the culprit. mysqld (Percona/Galera) is using a fair amount too.</div></div>
</blockquote></div><br><br clear="all"><div><br></div></div><div>-- <br><div dir="ltr">Joe Topjian<div>Systems Architect</div><div>Cybera Inc.</div><div><br></div><div><a href="http://www.cybera.ca" target="_blank">www.cybera.ca</a></div>
<div><br></div><div><font color="#666666"><span>Cybera</span><span> is a not-for-profit organization that works to spur and support innovation, for the economic benefit of Alberta, through the use of cyberinfrastructure.</span></font></div>
</div>
</div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Joe Topjian<div>Systems Architect</div><div>Cybera Inc.</div><div><br></div><div><a href="http://www.cybera.ca" target="_blank">www.cybera.ca</a></div>
<div><br></div><div><font color="#666666"><span>Cybera</span><span> is a not-for-profit organization that works to spur and support innovation, for the economic benefit of Alberta, through the use of cyberinfrastructure.</span></font></div>
</div>
</div>