Open Stack

Here's mine on a relatively idle new compute node I'm setting up. It has 3
instances running on it, one of which is receiving a constant stream of
netflow traffic to store.

http://paste.openstack.org/show/48708/


On Fri, Oct 18, 2013 at 9:48 AM, Samuel Winchenbach <swinchen at gmail.com>wrote:

> Just out of curiosity would someone else mind showing me what the output
> of their "dstat -sv" looks like?  This is mine:
> http://paste.openstack.org/show/48705/
>
> I am concerned by the procs:new column.  That seems to be the number of
> new processes since the last step?
>
> Sam
>
>
> On Wed, Oct 16, 2013 at 7:36 PM, Mike Wilson <geekinutah at gmail.com> wrote:
>
>> I want to add a little bit to what Ryan Richard is saying. Things
>> swapping when there is plenty of free memory isn't a behavior that is
>> specific to OpenStack or qemu, it is part of the way the linux kernel
>> behaves. You can tune overcommit ratios and swapiness, but even with  wilth
>> swapiness set to 0 any overcommit_ratio greater than 1 will _still_ swap
>> things occasionally. There are 3 ways I am aware of that will ensure that
>> you _never_ use swap.
>>
>> 1. Use hugepages to back your applications, currently the linux kernel
>> doesn't know how to swap hugepages. There are other disadvantages and
>> advantages to using hugepages, I don't recommend this as a
>> one-size-fits-all solution. This definitely merits more understanding and
>> research before you decide to go this route. Qemu is happy to  use
>> hugepages memory to back instances. Good luck deduping those hugepages
>> though, last I heard neither ksmd nor anything else was able to do that.
>>
>> 2. You can apply patches to qemu and other applications to use mlock,
>> mlockall systems calls and the kernel will not swap that memory region
>> ever. If you search around there are a few patches that attempt to do just
>> this floating around on mailing lists and random git repos. Again, your
>> mileage may vary with this approach, I believe we do something close to
>> this in our production environment. But we weren't able to use what we
>> found publicly, we had to write our own. Sorry, the guy that wrote them is
>> on vacation so I can't just provide you with a link to our patch.
>>
>> 3. Turn off swap devices. This is generally not the answer. But if you
>> never ever see a legitimate case to swap rather than just die, then it
>> could be :-).
>>
>> -Mike Wilson
>>
>>
>>
>>
>> On Wed, Oct 16, 2013 at 5:12 PM, Ryan Richard <ryan.richard at rackspace.com
>> > wrote:
>>
>>>  A few thoughts:
>>>
>>>
>>>    1. Are you actively swapping pages in and out? I'm going to guess
>>>    not since you have a lot of RAM available and the swap space hasn't been
>>>    freed. This would lead me to believe there was an event in the past that
>>>    forced the kernel to move some pages out to swap space but they haven't
>>>    been needed, so they've been left there.. The package 'dstat' is helpful to
>>>    determine if you're paging.
>>>    2. Do you overcommit your ram? If so, you may have booted a RAM
>>>    heavy VM at one point that pushed another process into swap. Perhaps look
>>>    at lowering the overcommit ratio.
>>>    3. You can lower the how aggressive the kernel moves things in and
>>>    out of swap space. The file /proc/sys/vm/swapiness controls this within the
>>>    kernel. You can try lowering it which will cause the kernel to not be so
>>>    aggressive about moving things in to swap. 0 is the lowest value.
>>>
>>> Paging in and out is what kills performance. Stuff in swap doesn't
>>> generally hurt performance until it needs to be accessed and has to be
>>> moved back into RAM. Constant paging in and out is a huge problem
>>> especially on a hypervisor with busy disks though.
>>>
>>>
>>>  Ryan Richard | RHCA
>>>  Rackspace US, Inc.
>>>
>>>   From: Joe Topjian <joe.topjian at cybera.ca>
>>> Date: Wednesday, October 16, 2013 3:46 PM
>>> To: Samuel Winchenbach <swinchen at gmail.com>
>>> Cc: "openstack-operators at lists.openstack.org" <
>>> openstack-operators at lists.openstack.org>
>>> Subject: Re: [Openstack-operators] Strange memory usage
>>>
>>>   Hello,
>>>
>>>  I'm not exactly sure what the next steps would be. :(
>>>
>>>  If your qemu processes are being reported as the high-swap consumers,
>>> it might be a more site-specific or instance-specific issue rather than
>>> something with OpenStack itself.
>>>
>>>  From the process string you gave, it looks like a normal running kvm
>>> instance to me... I can't see anything unusual.
>>>
>>>  If these instances aren't doing anything that would cause them to use
>>> a high amount of memory to the point of swapping, I'd start looking into
>>> KVM and reasons why it would want to swap.
>>>
>>>  Joe
>>>
>>>
>>> On Wed, Oct 16, 2013 at 2:25 PM, Samuel Winchenbach <swinchen at gmail.com>wrote:
>>>
>>>>  Hi Joe.  I have deleted that instance (it was no longer needed).
>>>>  Here is the next one down on the list:
>>>>
>>>>  119       6560  3.4  2.6 9247872 1749336 ?     Sl   Oct15  41:33
>>>> qemu-system-x86_64 -machine accel=kvm:tcg -name instance-000001b3 -S -M
>>>> pc-i440fx-1.4 -cpu
>>>> Opteron_G3,+nodeid_msr,+wdt,+skinit,+ibs,+osvw,+3dnowprefetch,+cr8legacy,+extapic,+cmp_legacy,+3dnow,+3dnowext,+pdpe1gb,+fxsr_opt,+mmxext,+ht,+vme
>>>> -m 4096 -smp 4,sockets=4,cores=1,threads=1 -uuid
>>>> 9627e75b-8d3b-4482-be43-8aba176a3f1f -smbios type=1,manufacturer=OpenStack
>>>> Foundation,product=OpenStack
>>>> Nova,version=2013.1.3,serial=49434d53-0200-9010-2500-109025007800,uuid=9627e75b-8d3b-4482-be43-8aba176a3f1f
>>>> -no-user-config -nodefaults -chardev
>>>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-000001b3.monitor,server,nowait
>>>> -mon chardev=charmonitor,id=monitor,mode=control -rtc
>>>> base=utc,driftfix=slew -no-shutdown -device
>>>> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
>>>> file=/os-grizzly/nova/instances/9627e75b-8d3b-4482-be43-8aba176a3f1f/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
>>>> -device
>>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>>>> -drive
>>>> file=/os-grizzly/nova/instances/9627e75b-8d3b-4482-be43-8aba176a3f1f/disk.swap,if=none,id=drive-virtio-disk1,format=qcow2,cache=none
>>>> -device
>>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1
>>>> -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=31 -device
>>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:45:63:f0,bus=pci.0,addr=0x3
>>>> -chardev
>>>> file,id=charserial0,path=/os-grizzly/nova/instances/9627e75b-8d3b-4482-be43-8aba176a3f1f/console.log
>>>> -device isa-serial,chardev=charserial0,id=serial0 -chardev
>>>> pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1
>>>> -device usb-tablet,id=input0 -vnc 0.0.0.0:6 -k en-us -vga cirrus
>>>> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
>>>>
>>>>
>>>> On Wed, Oct 16, 2013 at 4:07 PM, Joe Topjian <joe.topjian at cybera.ca>wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>>  "ps auxww | grep 19587", or some variation of the command, should
>>>>> give you the full command. Can you trace it back to an individual instance?
>>>>>
>>>>>  Thanks,
>>>>> Joe
>>>>>
>>>>>
>>>>> On Wed, Oct 16, 2013 at 11:50 AM, Samuel Winchenbach <
>>>>> swinchen at gmail.com> wrote:
>>>>>
>>>>>>  Joe: http://pastie.org/pastes/8407072/text
>>>>>>
>>>>>>  It really looks like qemu-system-x86 is the culprit.   mysqld
>>>>>> (Percona/Galera) is using a fair amount too.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>  --
>>>>> Joe Topjian
>>>>> Systems Architect
>>>>> Cybera Inc.
>>>>>
>>>>>  www.cybera.ca
>>>>>
>>>>>  Cybera is a not-for-profit organization that works to spur and
>>>>> support innovation, for the economic benefit of Alberta, through the use
>>>>> of cyberinfrastructure.
>>>>>
>>>>
>>>>
>>>
>>>
>>>  --
>>> Joe Topjian
>>> Systems Architect
>>> Cybera Inc.
>>>
>>>  www.cybera.ca
>>>
>>>  Cybera is a not-for-profit organization that works to spur and support
>>> innovation, for the economic benefit of Alberta, through the use
>>> of cyberinfrastructure.
>>>
>>> _______________________________________________
>>> OpenStack-operators mailing list
>>> OpenStack-operators at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>
>>>
>>
>


-- 
Joe Topjian
Systems Architect
Cybera Inc.

www.cybera.ca

Cybera is a not-for-profit organization that works to spur and support
innovation, for the economic benefit of Alberta, through the use
of cyberinfrastructure.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20131018/87498d53/attachment.html>