Open Stack

Hi,

I was able to resolve the performance related issue by changing the cpu
allocation ratio from 16 to lower value.

Thanks

On Tue, Dec 20, 2016 at 6:09 PM, Vikram Chhibber <vikram.chhibber at gmail.com>
wrote:

> Thanks Konstantin.
> This does not look like any specific issue with the boot of the instance.
> The issue is that overall system gets slow as multiple instances are
> running.
> Since this is nested virtualization, I also checked the kvm acceleration
> in the first leve VM which looks ok to me.
>
> The strange thing is that there are too many interrupts of type
> "Rescheduling interrupts".
>
> If I bring up one instance, things are fast and here is the vmstat:
> r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id
> wa st
>
> 1  1      0 2714652 270040 656980    0    0     0    32  514  825  0 16 80
>  4  0
>
> 3  0      0 2714652 270040 656980    0    0     0    32  573  904  1 20 76
>  3  0
>
> 0  0      0 2713908 270044 656996    0    0     0    32  526  829  1 15 80
>  2  1
>
> 0  0      0 2713940 270044 656996    0    0     0    32  478  795  1 11 85
>  3  0
>
>
> vmstat when two instances are running.
>
> r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id
> wa st
>
> 3  1      0 2715332 269700 656128    0    0     0    56  504  779  3 32 64
>  1  0
>
> 4  0      0 2715704 269700 656128    0    0     0    48  536  801  4 34 57
>  5  0
>
> 2  0      0 2715752 269700 656128    0    0     0    36  554  920  3 34 58
>  6  0
>
> 21  2      0 2713700 269708 656124    0    0     0   176  601  837 11 39
> 44  5  1
>
>
> Thanks
>
>
>
>
> On Mon, Dec 19, 2016 at 1:58 AM, Kostyantyn Volenbovskyi <
> volenbovsky at yandex.ru> wrote:
>
>> Hi,
>>
>> a few thoughts:
>> -analyze what exactly time is spent on using Nova logs
>> (api+scheduler+conductor+compute)+Neutron (agent, bind port
>> stuff)+Libvirt/QEMU logs. Use req-id and UUID of instance to identify the
>> ’slow’ case across logs.
>> Take a look at the create instance’ flow that is present in bunch of
>> websites in internet and make a draft of distribution of time for each stage
>> (side note: maybe there is tool that will allow such thing?). Turn on
>> debug logging in Nova and Neutron to narrow it down in case necessary .
>> -compare CPU/RAM consumption in normal case and ‘slow’ case’
>>
>> How do I bound my vCPU all the way to the physical host?
>>
>> Answer is CPU pinning. But yup, that is about CPU utilization on host ,
>> not related to time of VM launch.
>> Note that for general-purpose case CPU pinning can be an overkill in a
>> way that general-purpose relies
>> on scheduling by host OS and it should not be problematic. And most
>> configurations run with oversubscription of CPU 16 which is default (1
>> physical CPU=16 virtual)
>> 'Specific purpose' cases are like NFV/Telco where CPU pinning is
>> much-much more used.
>>
>> BR,
>> Konstantin
>>
>>
>> On Dec 19, 2016, at 5:27 AM, Vikram Chhibber <vikram.chhibber at gmail.com>
>> wrote:
>>
>> Hi All,
>>
>> I am using Kilo release 1:2015.1.4-0ubuntu2 for my lab deployment on
>> single physical host. The issue is that I am getting very low performance
>> when I launch multiple VM instances. The first one boots up within seconds
>> but the second takes noticeable time. If I instantiate the third instance
>> when two instances are already running, it may take 30 minutes to come up.
>> Moreover, the CPU idle % for a given instance keeps decreasing as number of
>> running instances increase.
>> I am suspecting that this behavior is because be lack of bounding of
>> vCPUs with physical CPUs.
>> Because of single node installation, the compute node itself becomes
>> virtualized and run my instances within it. How do I bound my vCPU all the
>> way to the physical host? I did not see any documentation regarding this
>> for Kilo release and there is no mention of bounding CPU for the
>> virtualized compute node on single node installation.
>>
>> This is the specification of the host:
>> Architecture:          x86_64
>> CPU op-mode(s):        32-bit, 64-bita
>> Byte Order:            Little Endian
>> CPU(s):                36
>> On-line CPU(s) list:   0-35
>> Thread(s) per core:    2
>> Core(s) per socket:    18
>> Socket(s):             1
>> NUMA node(s):          1
>> Vendor ID:             GenuineIntel
>> CPU family:            6
>> Model:                 63
>> Stepping:              2
>> CPU MHz:               1200.000
>> BogoMIPS:              4599.94
>> Virtualization:        VT-x
>> L1d cache:             32K
>> L1i cache:             32K
>> L2 cache:              256K
>> L3 cache:              46080K
>> NUMA node0 CPU(s):     0-35
>>
>>
>> Spec. for the compute node:
>> Architecture:          x86_64
>> CPU op-mode(s):        32-bit, 64-bit
>> Byte Order:            Little Endian
>> CPU(s):                26
>> On-line CPU(s) list:   0-25
>> Thread(s) per core:    1
>> Core(s) per socket:    1
>> Socket(s):             26
>> NUMA node(s):          1
>> Vendor ID:             GenuineIntel
>> CPU family:            6
>> Model:                 6
>> Stepping:              3
>> CPU MHz:               2299.996
>> BogoMIPS:              4599.99
>> Virtualization:        VT-x
>> Hypervisor vendor:     KVM
>> Virtualization type:   full
>> L1d cache:             32K
>> L1i cache:             32K
>> L2 cache:              4096K
>> NUMA node0 CPU(s):     0-25
>>
>>
>> Spec for my instance:
>> Architecture:          x86_64
>> CPU op-mode(s):        32-bit, 64-bit
>> Byte Order:            Little Endian
>> CPU(s):                4
>> On-line CPU(s) list:   0-3
>> Thread(s) per core:    1
>> Core(s) per socket:    1
>> Socket(s):             4
>> NUMA node(s):          1
>> Vendor ID:             GenuineIntel
>> CPU family:            15
>> Model:                 6
>> Stepping:              1
>> CPU MHz:               2299.994
>> BogoMIPS:              4599.98
>> Virtualization:        VT-x
>> Hypervisor vendor:     KVM
>> Virtualization type:   full
>> L1d cache:             32K
>> L1i cache:             32K
>> L2 cache:              4096K
>> NUMA node0 CPU(s):     0-3
>>
>> Thanks
>> _______________________________________________
>> Mailing list: http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>> Post to     : openstack at lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20161221/72f3b244/attachment.html>