<div dir="ltr"><div>Hi stackers, </div><div><br></div><div><br></div><div>I would like to put some more details on current situation. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br><span style="font-family:arial,sans-serif;font-size:13px">The issue is with what Rally is in it's<br></span><span style="font-family:arial,sans-serif;font-size:13px">current form. It's scope is too large and monolithic, and it duplicates much of<br>
</span><span style="font-family:arial,sans-serif;font-size:13px">the functionality we either already have or need in current QA or Infra<br></span><span style="font-family:arial,sans-serif;font-size:13px">projects. But, nothing in Rally is designed to be used outside of it. I actually<br>
</span><span style="font-family:arial,sans-serif;font-size:13px">feel pretty strongly that in it's current form Rally should *not* be a part of<br></span><span style="font-family:arial,sans-serif;font-size:13px">any OpenStack program</span></blockquote>
<div><br></div><div>Rally is not just a bunch of scripts like tempest, it's more like Nova, Cinder, and other projects that works out of box and resolve Operators & Dev use cases in one click. </div><div><br></div>
<div>This architectural design is the main key of Rally success, and why we got such large adoption and community.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<span style="font-family:arial,sans-serif;font-size:13px">So I'm opposed to this option. It feels to me like this is only on the table<br></span><span style="font-family:arial,sans-serif;font-size:13px">because the Rally team has not done a great job of communicating or working with<br>
</span><span style="font-family:arial,sans-serif;font-size:13px">anyone else except for when it comes to either push using Rally, or this<br></span><span style="font-family:arial,sans-serif;font-size:13px">conversation about adopting Rally.</span></blockquote>
<div><br></div><div>Actually Rally team done already a bunch of useful work including cross projects and infra stuff. </div><div><br></div><div>Keystone, Glance, Cinder, Neutron and Heat are running rally performance jobs, that can be used for performance testing, benchmarking, regression testing (already now). These jobs supports in-tree plugins for all components (scenarios, load generators, benchmark context) and they can use Rally fully without interaction with Rally team at all. More about these jobs: </div>
<div><a href="https://docs.google.com/a/mirantis.com/document/d/1s93IBuyx24dM3SmPcboBp7N47RQedT8u4AJPgOHp9-A/">https://docs.google.com/a/mirantis.com/document/d/1s93IBuyx24dM3SmPcboBp7N47RQedT8u4AJPgOHp9-A/</a></div><div>
So I really don't see anything like this in tempest (even in observed future) <br></div><div><br></div><div><br></div><div>I would like to mention work on OSprofiler (cross service/project profiler) <a href="https://github.com/stackforge/osprofiler">https://github.com/stackforge/osprofiler</a> (that was done by Rally team) <a href="https://review.openstack.org/#/c/105096/">https://review.openstack.org/#/c/105096/</a></div>
<div>(btw Glance already accepted it <a href="https://review.openstack.org/#/c/105635/">https://review.openstack.org/#/c/105635/</a> )</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<span style="font-family:arial,sans-serif;font-size:13px">My primary concern is the timing for doing all of this work. We're approaching<br></span><span style="font-family:arial,sans-serif;font-size:13px">J-3 and honestly this feels like it would take the better part of an entire<br>
</span><span style="font-family:arial,sans-serif;font-size:13px">cycle to analyze, plan, and then implement. Starting an analysis of how to do<br></span><span style="font-family:arial,sans-serif;font-size:13px">all of the work at this point I feel would just distract everyone from<br>
</span><span style="font-family:arial,sans-serif;font-size:13px">completing our dev goals for the cycle. Probably the Rally team, if they want<br></span><span style="font-family:arial,sans-serif;font-size:13px">to move forward here, should start the analysis of this structural split and we<br>
</span><span style="font-family:arial,sans-serif;font-size:13px">can all pick this up together post-juno</span></blockquote><div><br></div><div><br></div><div>Matt, Sean - seriously community is about convincing people, not about forcing people to do something against their wiliness. You are making huge architectural decisions without deep knowledge about what is Rally, what are use cases, road map, goals and auditory. </div>
<div><br></div><div>IMHO community in my opinion is thing about convincing people. So QA program should convince Rally team (at least me) to do such changes. Key secret to convince me, is to say how this will help OpenStack to perform better.</div>
<div><br></div><div>Currently Rally team see a lot of issues related to this decision:</div><div><br></div><div>1) It breaks already existing performance jobs (Heat, Glance, Cinder, Neutron, Keystone)</div><div><br></div>
<div>2) It breaks functional testing of Rally (that is already done in gates)</div><div><br></div><div>2) It makes Rally team depending on Tempest throughput, and what I heard multiple times from QA team is that performance work is very low priority and that major goals are to keep gates working. So it will slow down work of performance team. </div>
<div><br></div><div>3) Brings a ton of questions what should be in Rally and what should be in Tempest. That are at the moment quite resolved. </div><div><a href="https://docs.google.com/a/pavlovic.ru/document/d/137zbrz0KJd6uZwoZEu4BkdKiR_Diobantu0GduS7HnA/edit#heading=h.9ephr9df0new">https://docs.google.com/a/pavlovic.ru/document/d/137zbrz0KJd6uZwoZEu4BkdKiR_Diobantu0GduS7HnA/edit#heading=h.9ephr9df0new</a><br>
</div><div><br></div><div>4) It breaks existing OpenStack team, that is working 100% on performance, regressions and sla topics. Sorry but there is no such team in tempest. This directory is not active developed:</div><div>
<a href="https://github.com/openstack/tempest/commits/master/tempest/stress">https://github.com/openstack/tempest/commits/master/tempest/stress</a><br></div><div><br></div><div><br></div><div><div>Matt, Sean, David - what are the real goals of merging Rally to Tempest?</div>
<div>I see a huge harm for OpenStack and companies that are using Rally, and don't see actually any benefits. </div></div><div>What I heard for now is something like "this decision will make tempest better"..</div>
<div>But do you care more about Tempest than OpenStack? </div><div><br></div><div><br></div><div>Best regards,</div><div>Boris Pavlovic </div><div> </div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Tue, Aug 12, 2014 at 12:37 AM, David Kranz <span dir="ltr"><<a href="mailto:dkranz@redhat.com" target="_blank">dkranz@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><div><div class="h5">
<div>On 08/11/2014 04:21 PM, Matthew
Treinish wrote:<br>
</div>
<blockquote type="cite">
<p dir="ltr">I apologize for the delay in my response to this
thread, between travelling<br>
and having a stuck 'a' key on my laptop this is the earliest I
could respond.<br>
I opted for a separate branch on this thread to summarize my
views and I'll<br>
respond inline later on some of the previous discussion.</p>
<p dir="ltr">On Wed, Aug 06, 2014 at 12:30:35PM +0200, Thierry
Carrez wrote:<br>
> Hi everyone,<br>
> <br>
> At the TC meeting yesterday we discussed Rally program
request and<br>
> incubation request. We quickly dismissed the incubation
request, as<br>
> Rally appears to be able to live happily on top of
OpenStack and would<br>
> benefit from having a release cycle decoupled from the
OpenStack<br>
> "integrated release".<br>
> <br>
> That leaves the question of the program. OpenStack programs
are created<br>
> by the Technical Committee, to bless existing efforts and
teams that are<br>
> considered *essential* to the production of the "OpenStack"
integrated<br>
> release and the completion of the OpenStack project
mission. There are 3<br>
> ways to look at Rally and official programs at this point:<br>
> <br>
> 1. Rally as an essential QA tool<br>
> Performance testing (and especially performance regression
testing) is<br>
> an essential QA function, and a feature that Rally
provides. If the QA<br>
> team is happy to use Rally to fill that function, then
Rally can<br>
> obviously be adopted by the (already-existing) QA program.
That said,<br>
> that would put Rally under the authority of the QA PTL, and
that raises<br>
> a few questions due to the current architecture of Rally,
which is more<br>
> product-oriented. There needs to be further discussion
between the QA<br>
> core team and the Rally team to see how that could work and
if that<br>
> option would be acceptable for both sides.</p>
<p dir="ltr">So ideally this is where Rally would belong, the
scope of what Rally is<br>
attempting to do is definitely inside the scope of the QA
program. I don't see<br>
any reason why that isn't the case. The issue is with what Rally
is in it's<br>
current form. It's scope is too large and monolithic, and it
duplicates much of<br>
the functionality we either already have or need in current QA
or Infra<br>
projects. But, nothing in Rally is designed to be used outside
of it. I actually<br>
feel pretty strongly that in it's current form Rally should
*not* be a part of<br>
any OpenStack program.</p>
<p dir="ltr">All of the points Sean was making in the other branch
on this thread (which I'll<br>
probably respond to later) are a huge concerns I share with
Rally. He basically<br>
summarized most of my views on the topic, so I'll try not to
rewrite everything.<br>
But, the fact that all of this duplicate functionality was
implemented in a<br>
completely separate manner which is Rally specific and can't
really be used<br>
unless all of Rally is used is of a large concern. What I think
the path<br>
forward here is to have both QA and Rally work together on
getting common<br>
functionality that is re-usable and shareable. Additionally, I
have some<br>
concerns over the methodology that Rally uses for it's
performance measurement.<br>
But, I'll table that discussion because I think it would
partially derail this<br>
discussion.</p>
<p dir="ltr">So one open question is long-term where would this
leave Rally if we want to<br>
bring it in under the QA program. (after splitting up the
functionality to more<br>
conducive with all our existing tools and projects) The one
thing Rally does<br>
here which we don't have an analogous solution for is, for lack
of better term,<br>
the post processing layer. The part that generates the performs
the analysis on<br>
the collected data and generates the graphs. That is something
that we'll have<br>
an eventually need for and that is something that we can work on
turning rally<br>
into as we migrate everything to actually work together.</p>
<p dir="ltr">There are probably also other parts of Rally which
don't fit into an existing<br>
QA program project, (or the QA program in general) and in those
cases we<br>
probably should split them off as smaller projects to implement
that bit. For<br>
example, the SLA stuff Rally has that probably should be a
separate entity as<br>
well, but I'm unsure if that fits under QA program.</p>
<p dir="ltr">My primary concern is the timing for doing all of
this work. We're approaching<br>
J-3 and honestly this feels like it would take the better part
of an entire<br>
cycle to analyze, plan, and then implement. Starting an analysis
of how to do<br>
all of the work at this point I feel would just distract
everyone from<br>
completing our dev goals for the cycle. Probably the Rally team,
if they want<br>
to move forward here, should start the analysis of this
structural split and we<br>
can all pick this up together post-juno.</p>
<p dir="ltr">> <br>
> 2. Rally as an essential operator tool<br>
> Regular benchmarking of OpenStack deployments is a best
practice for<br>
> cloud operators, and a feature that Rally provides. With a
bit of a<br>
> stretch, we could consider that benchmarking is essential
to the<br>
> completion of the OpenStack project mission. That program
could one day<br>
> evolve to include more such "operations best practices"
tools. In<br>
> addition to the slight stretch already mentioned, one
concern here is<br>
> that we still want to have performance testing in QA (which
is clearly<br>
> essential to the production of "OpenStack"). Letting Rally
primarily be<br>
> an operational tool might make that outcome more difficult.<br>
> </p>
<p dir="ltr">So I'm opposed to this option. It feels to me like
this is only on the table<br>
because the Rally team has not done a great job of communicating
or working with<br>
anyone else except for when it comes to either push using Rally,
or this<br>
conversation about adopting Rally.</p>
<p dir="ltr">That being said, looking at a separate "operator
tool" program for Rally doesn't<br>
make much sense to me. There is nothing in Rally that is more or
less operator<br>
tooling specific compared to Tempest or some of the infra
tooling. I fail to see<br>
what in Rally warrants a separate program. To be a bit sardonic,
my question is<br>
if Tempest had a REST API [1][2] then should we move it to the
proposed<br>
operators program too? The other thing, which came out of the
summit, was that<br>
tempest is often used by operators in a loop to get a heartbeat
on their cloud.</p>
<p dir="ltr">My point is that just because a tool is part of the
QA program doesn't mean<br>
it's not useful for operators. I think that's something that
seems to be lost<br>
during this discussion. (or just brushed over) Sure, our first
priority is going<br>
to be on making things work in dev environment and the gate, but
that doesn't<br>
necessarily preclude using things against a production
environment. For tempest<br>
at least, that's something we actually explicitly support. [3]</p>
</blockquote></div></div>
+1<br>
We were a little slow out of the gate (so to speak) on this but are
making progress by eliminating all devstack-specific stuff from
tempest configuration, adding support for non-admin parallel tempest
with multiple users, and in general getting rid of discovered
roadblocks to real use. As has been pointed out before, many folks
use tempest against real clouds, including many members of the
tempest core team. IMO this should be considered an equal priority
with gating a dev environment. The biggest problem with that goal is
that tempest gate jobs do not run in most of the vast number of
actual configurations that most real clouds can use and so it is
hard to keep it working with all configurations. But we should
support these cases as best we can.<span class="HOEnZb"><font color="#888888"><br>
<br>
-David<br>
<br>
</font></span><blockquote type="cite"><div><div class="h5">
<p dir="ltr">Maybe, one day there will be a need for a program
like this, but I'm just not<br>
seeing it here with Rally.</p>
<p dir="ltr">> 3. Let Rally be a product on top of OpenStack<br>
> The last option is to not have Rally in any program, and
not consider it<br>
> *essential* to the production of the "OpenStack" integrated
release or<br>
> the completion of the OpenStack project mission. Rally can
happily exist<br>
> as an operator tool on top of OpenStack. It is built as a
monolithic<br>
> product: that approach works very well for external
complementary<br>
> solutions... Also be more integrated in OpenStack or part
of the<br>
> OpenStack programs might come at a cost (slicing some
functionality out<br>
> of rally to make it more a framework and less a product)
that might not<br>
> be what its authors want.</p>
<p dir="ltr">Honestly, if the Rally team wants the project to
remain in it's current form and<br>
scope then I agree that it belongs outside of OpenStack. It
definitely feels<br>
like a product to me, and there is nothing stopping them from
continuing to<br>
operate as they do now on top of OpenStack. I'm sorry, but the
fact that the<br>
docs in the rally tree has a section for user testimonials [4] I
feel speaks a<br>
lot about the intent of the project.</p>
<p dir="ltr">> <br>
> Let's explore each option to see which ones are viable, and
the pros and<br>
> cons of each.<br>
> </p>
<p dir="ltr">I apologize if any of this is somewhat incoherent,
I'm still a bit jet-lagged<br>
so I'm not sure that I'm making much sense.</p>
<p dir="ltr">-Matt Treinish</p>
<p dir="ltr">[1] <a href="https://review.openstack.org/#/c/96163/" target="_blank">https://review.openstack.org/#/c/96163/</a><br>
[2] <a href="https://review.openstack.org/#/c/101093/" target="_blank">https://review.openstack.org/#/c/101093/</a><br>
[3]
<a href="http://docs.openstack.org/developer/tempest/overview.html#design-principles" target="_blank">http://docs.openstack.org/developer/tempest/overview.html#design-principles</a><br>
[4]
<a href="http://git.openstack.org/cgit/stackforge/rally/tree/doc/user_stories" target="_blank">http://git.openstack.org/cgit/stackforge/rally/tree/doc/user_stories</a><br>
</p>
<br>
<fieldset></fieldset>
<br>
</div></div><div class=""><pre>_______________________________________________
OpenStack-dev mailing list
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</div></blockquote>
<br>
</div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>