<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Date: Wed, 06 Aug 2014 09:44:12 -0400<br>
From: Sean Dague <<a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a>><br>
To: <a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a><br>
Subject: Re: [openstack-dev] Which program for Rally<br>
Message-ID: <<a href="mailto:53E2312C.8000309@dague.net" target="_blank">53E2312C.8000309@dague.net</a>><br>
Content-Type: text/plain; charset=utf-8<br>
<br>Like the fact that right now the rally team is proposing gate jobs which<br>
have some overlap to the existing largeops jobs. Did they start a<br>
conversation about it? Nope. They just went off to do their thing<br>
instead. <a href="https://review.openstack.org/#/c/112251/" target="_blank">https://review.openstack.org/#/c/112251/</a><br></blockquote><div><br></div><div>Hi Sean, <br><br></div><div>Appreciate your analysis <br></div>
<div>Here is a comparison of the tempest largeops job and similar in Rally.<br><br></div><div><div>What large-ops job provides as of now:</div><div>Running hard-coded configured benchmarks (in gates), that are taken from tempest repo.</div>
<div>eg: "run 100 vms by one request". End result is a +1
or -1 which doesnt really reflect much in terms of the performance stats
and regressions in performance.</div><div> </div><div> </div><div>What Rally job provides:</div><div>(example in glance: <a href="https://github.com/openstack/glance/tree/master/rally-scenarios" target="_blank">https://github.com/openstack/glance/tree/master/rally-scenarios</a>)</div>
<div> </div><div>1) Projects can specify which benchmarks to run:</div><div><a href="https://github.com/openstack/glance/blob/master/rally-scenarios/glance.yaml" target="_blank">https://github.com/openstack/glance/blob/master/rally-scenarios/glance.yaml</a></div>
<div> </div><div>2) Projects can specify conditions of passing and inputs to benchmarks</div><div>(e.g. there is no iteration of benchmark failed and average duration of iteration is less then X)</div>
<div><a href="https://github.com/stackforge/rally/blob/master/rally-scenarios/rally.yaml#L11-L12" target="_blank">https://github.com/stackforge/rally/blob/master/rally-scenarios/rally.yaml#L11-L12</a></div><div> </div><div>
</div><div>3) Projects can create any number of benchmarks inside their source tree</div><div>(so they don't need to merge anything to rally)</div><div><a href="https://github.com/openstack/glance/tree/master/rally-scenarios/plugins" target="_blank">https://github.com/openstack/glance/tree/master/rally-scenarios/plugins</a></div>
<div> </div><div>4) Users are getting automated reports of all benchmarks:</div><div><a href="http://logs.openstack.org/81/112181/2/check/gate-rally-dsvm-rally/78b1146/rally-plot/results.html.gz" target="_blank">http://logs.openstack.org/81/112181/2/check/gate-rally-dsvm-rally/78b1146/rally-plot/results.html.gz</a></div>
<div> </div><div>5) Users can easily install Rally (with this script <a href="https://github.com/stackforge/rally/blob/master/install_rally.sh" target="_blank">https://github.com/stackforge/rally/blob/master/install_rally.sh</a>)</div>
<div>and test benchmark locally, using the same benchmark configuration as in gate.</div><div> </div><div> 6) Rally jobs (benchmarks) give you capabilities to
check for SLAs in your gates itself, which helps immensly to gauge
impact of your proposed change on the current code in terms of
performance and SLA</div><div> </div><div>Basically with Rally job, one can benchmark changes and compare them with master in gates.</div><div>Using below approach:</div><div> </div><div>
1) Put patch set 1 that changes rally benchmark configuration and probably adds some benchmark</div><div>Get base results</div><div> </div><div>2) Put patch set2 that includes point 1 + changes that fixes issue</div>
<div>Get new results</div><div> </div><div>3) Compare results and if new results are better push patch set 3 that removes changes in task and merge it.<br><br></div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
So now we're going to run 2 jobs that do very similar things, with<br>
different teams adjusting the test loads. Which I think is basically<br>
madness.<br>
<br>
-Sean<br>
<br>
--<br>
Sean Dague<br>
<a href="http://dague.net" target="_blank">http://dague.net</a><br></blockquote><div><br><div>Rally jobs allows every project to chose what benchmarks to have (as a plugins in their source tree) & run in gates.</div>
<div> </div><div>Rally is trying to be as open as possible by helping
projects define and set their own benchmarks in their gates which they
have full control over.</div><div> </div><div>I think this is very important point as it simplifies a lot work on performance issues. Hopefully we can discuss out these issues in IRC or someplace so that we are all on same page in terms of details about what Rally does and what it doesnt do.<br>
</div><div> </div><div>Rohan Kanade</div><div>Senior Software Engineer, Red Hat</div> </div></div><br></div></div>