<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style id="owaParaStyle" type="text/css">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body ocsi="0" fpstyle="1">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">You can default it to the number of cores, but please make it configurable. Some ops cram lots of services onto one node, and one service doesn't get to monopolize all cores.<br>
<br>
Thanks,<br>
Kevin<br>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div style="direction: ltr;" id="divRpF690427"><font face="Tahoma" size="2" color="#000000"><b>From:</b> Angus Salkeld [asalkeld@mirantis.com]<br>
<b>Sent:</b> Tuesday, September 01, 2015 4:53 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [Heat] convergence rally test results (so far)<br>
</font><br>
</div>
<div></div>
<div>
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr">On Tue, Sep 1, 2015 at 10:45 PM Steven Hardy <<a href="mailto:shardy@redhat.com" target="_blank">shardy@redhat.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
On Fri, Aug 28, 2015 at 01:35:52AM +0000, Angus Salkeld wrote:<br>
> Hi<br>
> I have been running some rally tests against convergence and our existing<br>
> implementation to compare.<br>
> So far I have done the following:<br>
> 1. defined a template with a resource<br>
> groupA <a href="https://github.com/asalkeld/convergence-rally/blob/master/templates/resource_group_test_resource.yaml.template" rel="noreferrer" target="_blank">
https://github.com/asalkeld/convergence-rally/blob/master/templates/resource_group_test_resource.yaml.template</a><br>
> 2. the inner resource looks like<br>
> this:A <a href="https://github.com/asalkeld/convergence-rally/blob/master/templates/server_with_volume.yaml.templateA" rel="noreferrer" target="_blank">
https://github.com/asalkeld/convergence-rally/blob/master/templates/server_with_volume.yaml.templateA</a> (it<br>
> uses TestResource to attempt to be a reasonable simulation of a<br>
> server+volume+floatingip)<br>
> 3. defined a rally<br>
> job:A <a href="https://github.com/asalkeld/convergence-rally/blob/master/increasing_resources.yamlA" rel="noreferrer" target="_blank">
https://github.com/asalkeld/convergence-rally/blob/master/increasing_resources.yamlA</a> that<br>
> creates X resources then updates to X*2 then deletes.<br>
> 4. I then ran the above with/without convergence and with 2,4,8<br>
> heat-engines<br>
> Here are the results compared:<br>
> <a href="https://docs.google.com/spreadsheets/d/12kRtPsmZBl_y78aw684PTBg3op1ftUYsAEqXBtT800A/edit?usp=sharing" rel="noreferrer" target="_blank">
https://docs.google.com/spreadsheets/d/12kRtPsmZBl_y78aw684PTBg3op1ftUYsAEqXBtT800A/edit?usp=sharing</a><br>
> Some notes on the results so far:<br>
> * A convergence with only 2 engines does suffer from RPC overload (it<br>
> gets message timeouts on larger templates). I wonder if this is the<br>
> problem in our convergence gate...<br>
> * convergence does very well with a reasonable number of engines<br>
> running.<br>
> * delete is slightly slower on convergence<br>
> Still to test:<br>
> * the above, but measure memory usage<br>
> * many small templates (run concurrently)<br>
<br>
So, I tried running my many-small-templates here with convergence enabled:<br>
<br>
<a href="https://bugs.launchpad.net/heat/+bug/1489548" rel="noreferrer" target="_blank">https://bugs.launchpad.net/heat/+bug/1489548</a><br>
<br>
In heat.conf I set:<br>
<br>
max_resources_per_stack = -1<br>
convergence_engine = true<br>
<br>
Most other settings (particularly RPC and DB settings) are defaults.<br>
<br>
Without convergence (but with max_resources_per_stack disabled) I see the<br>
time to create a ResourceGroup of 400 nested stacks (each containing one<br>
RandomString resource) is about 2.5 minutes (core i7 laptop w/SSD, 4 heat<br>
workers e.g the default for a 4 core machine).<br>
<br>
With convergence enabled, I see these errors from sqlalchemy:<br>
<br>
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 652, in<br>
_checkout\n fairy = _ConnectionRecord.checkout(pool)\n', u' File<br>
"/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 444, in<br>
checkout\n rec = pool._do_get()\n', u' File<br>
"/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 980, in<br>
_do_get\n (self.size(), self.overflow(), self._timeout))\n',<br>
u'TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection<br>
timed out, timeout 30\n'].<br>
<br>
I assume this means we're loading the DB much more in the convergence case<br>
and overflowing the QueuePool?<br>
</blockquote>
<div><br>
</div>
<div>Yeah, looks like it.</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<br>
This seems to happen when the RPC call from the ResourceGroup tries to<br>
create some of the 400 nested stacks.<br>
<br>
Interestingly after this error, the parent stack moves to CREATE_FAILED,<br>
but the engine remains (very) busy, to the point of being partially<br>
responsive, so it looks like maybe the cancel-on-fail isnt' working (I'm<br>
assuming it isn't error_wait_time because the parent stack has been marked<br>
FAILED and I'm pretty sure it's been more than 240s).<br>
<br>
I'll dig a bit deeper when I get time, but for now you might like to try<br>
the stress test too. It's a bit of a synthetic test, but it turns out to<br>
be a reasonable proxy for some performance issues we observed when creating<br>
large-ish TripleO deployments (which also create a large number of nested<br>
stacks concurrently).<br>
</blockquote>
<div><br>
</div>
<div>Thanks a lot for testing Steve! I'll make 2 bugs for what you have raised</div>
<div>1. limit the number of resource actions in parallel (maybe base on the number of cores)</div>
<div>2. the cancel on fail error</div>
<div><br>
</div>
<div>-Angus</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<br>
Steve<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</body>
</html>