<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi, <div class=""><br class=""></div><div class="">a few thoughts:</div><div class="">-analyze what exactly time is spent on using Nova logs (api+scheduler+conductor+compute)+Neutron (agent, bind port stuff)+Libvirt/QEMU logs. Use req-id and UUID of instance to identify the ’slow’ case across logs.</div><div class="">Take a look at the create instance’ flow that is present in bunch of websites in internet and make a draft of distribution of time for each stage</div><div class="">(side note: maybe there is tool that will allow such thing?). Turn on debug logging in Nova and Neutron to narrow it down in case necessary .</div><div class="">-compare CPU/RAM consumption in normal case and ‘slow’ case’</div><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div dir="ltr" class=""><span id="gmail-docs-internal-guid-389f7d4c-1556-5399-346e-592fd070c4dd" class=""><div class="" style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;"><span class="" style="font-size: 14.6667px; font-family: arial; vertical-align: baseline; white-space: pre-wrap;">How do I bound my vCPU all the way to the physical host?</span></div></span></div></blockquote></div><div class="">Answer is CPU pinning. But yup, that is about CPU utilization on host , not related to time of VM launch.</div><div class="">Note that for general-purpose case CPU pinning can be an overkill in a way that general-purpose relies</div><div class="">on scheduling by host OS and it should not be problematic. And most configurations run with oversubscription of CPU 16 which is default (1 physical CPU=16 virtual)</div><div class="">'Specific purpose' cases are like NFV/Telco where CPU pinning is much-much more used.</div><div class=""><br class=""></div><div class="">BR, </div><div class="">Konstantin</div><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Dec 19, 2016, at 5:27 AM, Vikram Chhibber <<a href="mailto:vikram.chhibber@gmail.com" class="">vikram.chhibber@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><span id="gmail-docs-internal-guid-389f7d4c-1556-5399-346e-592fd070c4dd" class=""><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Hi All,</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class=""><br class=""></span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">I am using Kilo release 1:2015.1.4-0ubuntu2 for my lab deployment on single physical host. The issue is that I am getting very low performance when I launch multiple VM instances. The first one boots up within seconds but the second takes noticeable time. If I instantiate the third instance when two instances are already running, it may take 30 minutes to come up. Moreover, the CPU idle % for a given instance keeps decreasing as number of running instances increase.</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">I am suspecting that this behavior is because be lack of bounding of vCPUs with physical CPUs.</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Because of single node installation, the compute node itself becomes virtualized and run my instances within it. How do I bound my vCPU all the way to the physical host? I did not see any documentation regarding this for Kilo release and there is no mention of bounding CPU for the virtualized compute node on single node installation.</span></div><br class=""><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">This is the specification of the host:</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Architecture: x86_64</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">CPU op-mode(s): 32-bit, 64-bita</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Byte Order: Little Endian</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">CPU(s): 36</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">On-line CPU(s) list: 0-35</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Thread(s) per core: 2</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Core(s) per socket: 18</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Socket(s): 1</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">NUMA node(s): 1</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Vendor ID: GenuineIntel</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">CPU family: 6</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Model: 63</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Stepping: 2</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">CPU MHz: 1200.000</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">BogoMIPS: 4599.94</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Virtualization: VT-x</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">L1d cache: 32K</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">L1i cache: 32K</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">L2 cache: 256K</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">L3 cache: 46080K</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">NUMA node0 CPU(s): 0-35</span></div><br class=""><br class=""><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Spec. for the compute node:</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Architecture: x86_64</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">CPU op-mode(s): 32-bit, 64-bit</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Byte Order: Little Endian</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">CPU(s): 26</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">On-line CPU(s) list: 0-25</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Thread(s) per core: 1</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Core(s) per socket: 1</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Socket(s): 26</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">NUMA node(s): 1</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Vendor ID: GenuineIntel</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">CPU family: 6</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Model: 6</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Stepping: 3</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">CPU MHz: 2299.996</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">BogoMIPS: 4599.99</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Virtualization: VT-x</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Hypervisor vendor: KVM</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Virtualization type: full</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">L1d cache: 32K</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">L1i cache: 32K</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">L2 cache: 4096K</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">NUMA node0 CPU(s): 0-25</span></div><br class=""><br class=""><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Spec for my instance:</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Architecture: x86_64</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">CPU op-mode(s): 32-bit, 64-bit</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Byte Order: Little Endian</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">CPU(s): 4</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">On-line CPU(s) list: 0-3</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Thread(s) per core: 1</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Core(s) per socket: 1</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Socket(s): 4</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">NUMA node(s): 1</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Vendor ID: GenuineIntel</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">CPU family: 15</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Model: 6</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Stepping: 1</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">CPU MHz: 2299.994</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">BogoMIPS: 4599.98</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Virtualization: VT-x</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Hypervisor vendor: KVM</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Virtualization type: full</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">L1d cache: 32K</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">L1i cache: 32K</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">L2 cache: 4096K</span></div><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">NUMA node0 CPU(s): 0-3</span></span><br class=""><div class=""><span class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class=""><br class=""></span></span></div><div class=""><span class=""><span style="font-size:14.6667px;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="">Thanks</span></span></div></div>
_______________________________________________<br class="">Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br class="">Post to : <a href="mailto:openstack@lists.openstack.org" class="">openstack@lists.openstack.org</a><br class="">Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br class=""></div></blockquote></div><br class=""></div></body></html>