<div dir="ltr">David, <div><br></div><div><br></div><div>Yes, that is the goal. </div><div>We should be able to get information about "how" it works, not only that it works. <br></div><div><br></div><div>And get such interesting results:</div>
<div><a href="https://wiki.openstack.org/wiki/Rally#How_influence_amqp_rpc_single_reply_queue_on_performance">https://wiki.openstack.org/wiki/Rally#How_influence_amqp_rpc_single_reply_queue_on_performance</a><br></div><div>
<br></div><div><br></div><div><br></div><div>Best regards,</div><div>Boris Pavlovic</div><div>---</div><div>Mirantis Inc. </div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Oct 18, 2013 at 8:57 PM, David Kranz <span dir="ltr"><<a href="mailto:dkranz@redhat.com" target="_blank">dkranz@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><div><div class="h5">
<div>On 10/18/2013 12:17 PM, Boris Pavlovic
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">John,
<div><br>
</div>
<div>
<div class="gmail_default" style="font-size:13px;font-family:'courier new',monospace">Actually
seems like a pretty good suggestion IMO, at least something
worth some investigation and consideration before quickly
discounting it. Rather than "that's not what tempest is",
maybe it's something tempest "could do". Don't know, not
saying one way or the other, just wondering if it's worth
some investigation or thought.</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>These investigations I made before start working around
Rally. It was about 3 months ago. </div>
<div>It is not "quickly discounting" I didn't have yesterday
time to make long response, so I will write it today: </div>
<div><br>
</div>
<div>I really don't like to make a copy of another projects, so
I tried to reuse all projects & libs that we already
have. <br>
</div>
<div><br>
</div>
<div>To explain why we shouldn't merge Rally & Tempest in
one project (and should keep both) we should analyze their
use cases. </div>
<div><br>
</div>
<div><br>
</div>
<div>1. DevStack - one "click" and get your OpenStack cloud from
sources</div>
<div><br>
</div>
<div>2. Tempest - one "click" and get your OpenStack Cloud
verified </div>
<div>
<br>
</div>
<div>Both of these projects are great, because they are very
useful and solve complicated tasks without "pain" for end
user. (and I like them)</div>
<div><br>
</div>
<div>3. Rally is also one "click" system that solve OpenStack
benchmarking.</div>
<div><br>
</div>
<div>To clarify situation. We should analyze what I mean by one
"click" benchmarking and what are the use cases.</div>
<div><br>
</div>
<div>Use Cases: </div>
<div>1. Investigate how deployments influence on OS performance
(find the set of good OpenStack deployment architectures)</div>
<div>2. Automatically get numbers & profiling info about how
your changes influence on OS performance</div>
<div>3. Using Rally profiler detect scale & performance
issues.</div>
<div>Like here when we are trying to delete 3 VMs by one request
they are deleted one by one because of DB lock on quotas table
<a href="http://37.58.79.43:8080/traces/0011f252c9d98e31" target="_blank">http://37.58.79.43:8080/traces/0011f252c9d98e31</a></div>
<div>4. Determine maximal load that could handle production
cloud</div>
<div><br>
</div>
<div>To cover these cases we should actually test OpenStack
deployments making simultaneously OpenStack API calls. </div>
<div><br>
</div>
<div>So to get results we have to: </div>
<div>1. Deploy OpenStack cloud somewhere. (Or get existing
cloud)</div>
<div>2. Verify It</div>
<div>3. Run Benchmarks </div>
<div>4. Collect all results & present it in human readable
form. </div>
<div><br>
</div>
<div><br>
</div>
<div>The goal of Rally was designed to automate these steps:</div>
<div>1.a Use existing cloud. </div>
<div>1.b.1 Automatically get (virtual) Servers from (soft layer,
Amazon, RackSpace or you private public cloud, or OpenStack
cloud)</div>
<div>1.b.2 DeployOpenStack on these servers from source (using
Devstack, Anvli, Fuel or TrippleO...). </div>
<div>1.b.3 Patch this OpenStack with tomograph to get profiling
information (I hope we will merge these patches into
upstream). </div>
<div>2. Using tempest verify this cloud (we are going to switch
from fuel-ostf-tests)</div>
<div>3. Run specified parametrized (to be able to make different
load) benchmark scenarios </div>
<div>4. Collect all information about execution & present it
in human readable form. (Tomograph, Zipking, matplotlib...)</div>
<div><br>
</div>
<div><br>
</div>
<div>So I am not sure that we should put inside Tempest Rally,
because Rally use tempest. It is something like putting Nova
into Cinder =)</div>
<div>Also putting Tempest into Rally is not a good idea. (same
as putting Cinder back to Nova).</div>
<div><br>
</div>
<div><br>
</div>
<div>Best regards,</div>
<div>Boris Pavlovic</div>
<div>---</div>
<div>Mirantis Inc. </div>
<div><br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Oct 17, 2013 at 11:56 PM,
John Griffith <span dir="ltr"><<a href="mailto:john.griffith@solidfire.com" target="_blank">john.griffith@solidfire.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div class="gmail_default" style="font-family:'courier new',monospace"><br>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Thu, Oct 17, 2013 at
1:44 PM, Jay Pipes <span dir="ltr"><<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div>On 10/17/2013 03:32 PM, Boris Pavlovic
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Jay,<br>
<br>
<br>
Or, alternately, just have Rally as part
of Tempest.<br>
<br>
<br>
Actually, tempest is used only to verify
that cloud works properly.<br>
And verification is only small part of the
Rally.<br>
<br>
At this moment we are using
fuel-ostf-tests, but we are going to use<br>
tempest to verify cloud.<br>
</blockquote>
<br>
</div>
OK, cool... was just a suggestion :) Tempest
has a set of stress tests [1] which are kind
of related, which is the only reason I brought
it up.<br>
<br>
Best,<br>
-jay<br>
<br>
[1] <a href="https://github.com/openstack/tempest/tree/master/tempest/stress" target="_blank">https://github.com/openstack/tempest/tree/master/tempest/stress</a>
<div>
<div><br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_default">Actually seems like a pretty good
suggestion IMO, at least something worth some
investigation and consideration before quickly
discounting it. Rather than "that's not what
tempest is", maybe it's something tempest "could
do". Don't know, not saying one way or the other,
just wondering if it's worth some investigation or
thought.</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">By the way, VERY COOL!!</div>
<br>
</div>
</div>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
OpenStack-dev mailing list
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote></div></div>
Thanks, Boris. This is really great. I took a look and there is a
lot of similarity between the tempest stress framework and the rally
benchmark driver and some code could be shared. But that code is
only a very small part of each of these projects and IMO there is no
reason to try and integrate the two more deeply. Rally will be
beneficial to the OpenStack QA community which needs to grow beyond
"OpenStackQA == tempest".<span class="HOEnZb"><font color="#888888"><br>
<br>
-David<br>
</font></span></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>