<div dir="ltr">We recently had a power outage, and perhaps one of the scenarios of controller capacity planning is starting all of the compute nodes at once or in large batches (when power was restored).<div><br></div><div>We painfully learned about our nova-conductor being low on workers/cores, but still we doubted whether it was a problem of our deployment. Now we know nova-conductor is very resource hungry.</div><div><br></div><div>Official recommendations about node ratios would be very appreciated.</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 19, 2015 at 8:36 PM, Rochelle Grober <span dir="ltr"><<a href="mailto:rochelle.grober@huawei.com" target="_blank">rochelle.grober@huawei.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p>Sorry this doesn't thread properly, but cut and pasted out of the digest...<u></u><u></u></p><span class="">
<p style="margin-left:.5in"><u></u> <u></u></p>
<p style="margin-left:.5in">> As providing OpenStack community with understandable recommendations
<u></u><u></u></p>
<p style="margin-left:.5in">> and instructions on performant OpenStack cloud deployments is part of
<u></u><u></u></p>
<p style="margin-left:.5in">> Performance Team mission, I'm kindly asking you to share your
<u></u><u></u></p>
<p style="margin-left:.5in">> experience on safe cloud deployment ratio between various types of
<u></u><u></u></p>
<p style="margin-left:.5in">> nodes you're having right now and the possible issues you observed (as
<u></u><u></u></p>
<p style="margin-left:.5in">> an example: discussed GoDaddy's cloud is having 3 conductor boxes vs
<u></u><u></u></p>
<p style="margin-left:.5in">> 250 computes in the cell, and there was an opinion that it's simply
<u></u><u></u></p>
<p style="margin-left:.5in">> not enough and that's it).<u></u><u></u></p>
<p style="margin-left:.5in"><u></u> <u></u></p>
<p style="margin-left:.5in">That was my opinion, and it was based on an apparently incorrect assumption that they had a lot of things coming and going on their cloud. I think they've demonstrated at this point that (other issues<u></u><u></u></p>
<p style="margin-left:.5in">aside) three is enough for them, given their environment, workload, and configuration.<u></u><u></u></p>
<p style="margin-left:.5in"><u></u> <u></u></p>
</span><p>This information is great for building rules of thumb, so to speak. GoDaddy has an example configuraton that is adequate for low frequency construct/destruct (low number of vm create/destroy) cloud architectures. This provides a lower
bounds and might be representative of a lot of enterprise cloud deployments.<u></u><u></u></p><span class="">
<p style="margin-left:.5in"><u></u> <u></u></p>
<p style="margin-left:.5in">The problem with coming up with any sort of metric that will apply to everyone is that it's highly variable. If you have 250 compute nodes and never create or destroy any instances, you'll be able to get away
with<u></u><u></u></p>
<p style="margin-left:.5in">*many* fewer conductors than if you have a very active cloud. Similarly, during a live upgrade (or following any upgrade where we do some online migration of data), your conductor load will be higher than normal.
Of course, 4-core and 96-core conductor nodes aren't equal either.<u></u><u></u></p>
<p style="margin-left:.5in"><u></u> <u></u></p>
</span><p>And here we have another rule of thumb, but no numbers put to it yet. If you have a low frequency construct/destruct cloud model, you will need to temporarily increase your number of conductors by {x amount OR x%} when performing OpenStack
live upgrades.<u></u><u></u></p><span class="">
<p style="margin-left:.5in"><u></u> <u></u></p>
<p style="margin-left:.5in">So, by all means, we should gather information on what people are doing successfully, but keep in mind that it depends *a lot* on what sort of workloads the cloud is supporting.<u></u><u></u></p>
<p style="margin-left:.5in"><u></u> <u></u></p>
</span><p>Right, but we can start applying fuzzy logic (the human kind, not machine) and get a better understanding of working configurations and *<b>why</b>* they work, then start examining where the transition states between configurations are.
You need data before you can create information ;-)<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>--Rocky<u></u><u></u></p>
<p style="margin-left:.5in"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-left:.5in">--Dan<u></u><u></u></p>
</div>
</div>
<br>_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br></blockquote></div><br></div>