Increasing the RPC timeout should help. I have seen this problem in nova-network in the past. Vish suggestion sounds good.<div><br></div><div>Recently we launched by mistake 128 VMs in a production environment of a customer: 0 errors. They are using 12 cores and several gigs for the nova-network servers with dual 10G pipes. So hardware matters, of course.</div>
<div><br></div><div>My two cents,</div><div>Diego</div><div><div><div> -- <br><span style="border-collapse:separate;font-family:Times"><span style="border-collapse:collapse;font-family:arial,sans-serif"><div align="left" style="font-size:13px">
<div><font><span lang="ES" style="font-family:Arial">Diego Parrilla<br><a href="http://www.stackops.com/" title="file:///C:/Documents%20and%20Settings/carolina.capsir.per1/Application%20Data/Microsoft/Signatures/www.garrigues.com
www.garrigues.com" style="color:rgb(7,77,143)" target="_blank"><span title="file:///C:/Documents%20and%20Settings/carolina.capsir.per1/Application%20Data/Microsoft/Signatures/www.garrigues.com"></span></a></span></font><font face="Arial" style="font-family:arial,helvetica,sans-serif"><b>CEO</b><font size="1"><br>
</font></font><span style="border-collapse:separate;font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"></span></span><b><font face="Arial" style="font-family:arial,helvetica,sans-serif"><a href="http://www.stackops.com/" target="_blank"><b>www.stackops.com</b></a> | </font></b><font face="Arial" style="font-family:arial,helvetica,sans-serif"><font size="1"> <a href="mailto:diego.parrilla@stackops.com" target="_blank">diego.parrilla@stackops.com</a></font></font><span style="border-collapse:separate;font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><font color="#004438" face="Arial"><b><b><span lang="EN-GB" style="font-size:10pt"></span></b></b></font></span></span><font face="Arial" style="font-family:arial,helvetica,sans-serif"><font size="1"> | </font></font><span style="border-collapse:separate;font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><font face="Arial" style="font-family:arial,helvetica,sans-serif"><font size="1">+34 649 94 43 29 | <a>skype:diegoparrilla</a></font></font></span></span><font face="Arial" style="font-family:arial,helvetica,sans-serif"><a href="http://www.stackops.com/" target="_blank"><b><br>
</b></a></font></div></div><div style="font-size:13px"><font color="#004438" face="Arial"><b><p><span style="border-collapse:separate;color:rgb(0,0,0);font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><span style="border-collapse:separate;font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><font color="#004438" face="Arial"><b><b><span lang="EN-GB" style="font-size:10pt"><img src="http://stackops.s3-external-3.amazonaws.com/STACKOPSLOGO-ICON.png"></span></b></b></font></span></span></span></span></p>
</b></font></div></span></span><div><br></div></div>
<br><br><div class="gmail_quote">On Tue, Feb 19, 2013 at 10:09 AM, gtt116 <span dir="ltr"><<a href="mailto:gtt116@126.com" target="_blank">gtt116@126.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Hi Diego<br>
<br>
Thanks for you reply.<br>
How many hosts do you have? I have 4 hosts. And in this bug,
<a href="https://bugs.launchpad.net/nova/+bug/1094226" target="_blank">https://bugs.launchpad.net/nova/+bug/1094226</a>,
The N is 20. In my environment N is about 16. <br>
<br>
I found that nova-network is too busy to deal with so many rpc
request at the same time. The Rabbitmq is strong enough in the
scenario.<br>
<br>
于 2013年02月19日 16:54, Diego Parrilla Santamaría 写道:<br>
</div><div><div class="h5">
<blockquote type="cite">Hi gtt,
<div><br>
</div>
<div>what does it mean for you 'lots of instance simultaneously'?
100, 1000, 10000, more?</div>
<div><br>
</div>
<div>We have launched >100 (but less than <1000)
simultaneously without any issue. Rabbit running in a multicore
with several gigs of RAM with out of the box configuration.</div>
<div><br>
</div>
<div>Cheers</div>
<div>Diego</div>
<div>
<div> -- <br>
<span style="border-collapse:separate;font-family:Times"><span style="border-collapse:collapse;font-family:arial,sans-serif">
<div style="font-size:13px" align="left">
<div><font><span style="font-family:Arial" lang="ES">Diego
Parrilla<br>
<a href="http://www.stackops.com/" title="file:///C:/Documents%20and%20Settings/carolina.capsir.per1/Application%20Data/Microsoft/Signatures/www.garrigues.comwww.garrigues.com" style="color:rgb(7,77,143)" target="_blank"><span title="file:///C:/Documents%20and%20Settings/carolina.capsir.per1/Application%20Data/Microsoft/Signatures/www.garrigues.com"></span></a></span></font><font style="font-family:arial,helvetica,sans-serif" face="Arial"><b>CEO</b><font size="1"><br>
</font></font><span style="border-collapse:separate;font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"></span></span><b><font style="font-family:arial,helvetica,sans-serif" face="Arial"><a href="http://www.stackops.com/" target="_blank"><b>www.stackops.com</b></a> | </font></b><font style="font-family:arial,helvetica,sans-serif" face="Arial"><font size="1"> <a href="mailto:diego.parrilla@stackops.com" target="_blank">diego.parrilla@stackops.com</a></font></font><span style="border-collapse:separate;font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><font color="#004438" face="Arial"><b><b><span style="font-size:10pt" lang="EN-GB"></span></b></b></font></span></span><font style="font-family:arial,helvetica,sans-serif" face="Arial"><font size="1"> | </font></font><span style="border-collapse:separate;font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><font style="font-family:arial,helvetica,sans-serif" face="Arial"><font size="1"><a href="tel:%2B34%20649%2094%2043%2029" value="+34649944329" target="_blank">+34 649 94 43 29</a> | <a>skype:diegoparrilla</a></font></font></span></span><font style="font-family:arial,helvetica,sans-serif" face="Arial"><a href="http://www.stackops.com/" target="_blank"><b><br>
</b></a></font></div>
</div>
<div style="font-size:13px"><font color="#004438" face="Arial"><b>
<p><span style="border-collapse:separate;font-size:medium;font-family:Times"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><span style="border-collapse:separate;font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><font color="#004438" face="Arial"><b><b><span style="font-size:10pt" lang="EN-GB"><img></span></b></b></font></span></span></span></span></p>
</b></font></div>
</span></span>
<div><br>
</div>
</div>
<br>
<br>
<div class="gmail_quote">On Tue, Feb 19, 2013 at 9:35 AM, gtt116
<span dir="ltr"><<a href="mailto:gtt116@126.com" target="_blank">gtt116@126.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Hi all,<br>
<br>
When create lots of instance simultaneously, there will be
lots of instance in ERROR state. And most of them are
caused by network rpc request timeout. This result is not
so graceful. <br>
<br>
I think it will be better if scheduler keep a queue of
creating request. when he find all the hosts are busy
enough(compute_node.current_workload reach some value),
stop cast the request to host temporarily, until he found
some host free enough. In this way, we can make sure
booting lots of instances simultaneously results in active
instances rather than lots of ERROR instance. but will
cause a small weak point, if the top value of
current_workload small enough, create instance processing
will be slow.<br>
<br>
Do you have another quick fix?<br>
<br>
Thanks,<span><font color="#888888"><br>
<pre cols="72">--
best regards,
gtt</pre>
</font></span></div>
<br>
_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<br>
</div></div><span class="HOEnZb"><font color="#888888"><pre cols="72">--
best regards,
gtt</pre>
</font></span></div>
</blockquote></div><br></div></div>