<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 08/11/2014 04:21 PM, Matthew
Treinish wrote:<br>
</div>
<blockquote
cite="mid:21f0cf0f-a118-4b9e-838a-4bac956b556a@email.android.com"
type="cite">
<p dir="ltr">I apologize for the delay in my response to this
thread, between travelling<br>
and having a stuck 'a' key on my laptop this is the earliest I
could respond.<br>
I opted for a separate branch on this thread to summarize my
views and I'll<br>
respond inline later on some of the previous discussion.</p>
<p dir="ltr">On Wed, Aug 06, 2014 at 12:30:35PM +0200, Thierry
Carrez wrote:<br>
> Hi everyone,<br>
> <br>
> At the TC meeting yesterday we discussed Rally program
request and<br>
> incubation request. We quickly dismissed the incubation
request, as<br>
> Rally appears to be able to live happily on top of
OpenStack and would<br>
> benefit from having a release cycle decoupled from the
OpenStack<br>
> "integrated release".<br>
> <br>
> That leaves the question of the program. OpenStack programs
are created<br>
> by the Technical Committee, to bless existing efforts and
teams that are<br>
> considered *essential* to the production of the "OpenStack"
integrated<br>
> release and the completion of the OpenStack project
mission. There are 3<br>
> ways to look at Rally and official programs at this point:<br>
> <br>
> 1. Rally as an essential QA tool<br>
> Performance testing (and especially performance regression
testing) is<br>
> an essential QA function, and a feature that Rally
provides. If the QA<br>
> team is happy to use Rally to fill that function, then
Rally can<br>
> obviously be adopted by the (already-existing) QA program.
That said,<br>
> that would put Rally under the authority of the QA PTL, and
that raises<br>
> a few questions due to the current architecture of Rally,
which is more<br>
> product-oriented. There needs to be further discussion
between the QA<br>
> core team and the Rally team to see how that could work and
if that<br>
> option would be acceptable for both sides.</p>
<p dir="ltr">So ideally this is where Rally would belong, the
scope of what Rally is<br>
attempting to do is definitely inside the scope of the QA
program. I don't see<br>
any reason why that isn't the case. The issue is with what Rally
is in it's<br>
current form. It's scope is too large and monolithic, and it
duplicates much of<br>
the functionality we either already have or need in current QA
or Infra<br>
projects. But, nothing in Rally is designed to be used outside
of it. I actually<br>
feel pretty strongly that in it's current form Rally should
*not* be a part of<br>
any OpenStack program.</p>
<p dir="ltr">All of the points Sean was making in the other branch
on this thread (which I'll<br>
probably respond to later) are a huge concerns I share with
Rally. He basically<br>
summarized most of my views on the topic, so I'll try not to
rewrite everything.<br>
But, the fact that all of this duplicate functionality was
implemented in a<br>
completely separate manner which is Rally specific and can't
really be used<br>
unless all of Rally is used is of a large concern. What I think
the path<br>
forward here is to have both QA and Rally work together on
getting common<br>
functionality that is re-usable and shareable. Additionally, I
have some<br>
concerns over the methodology that Rally uses for it's
performance measurement.<br>
But, I'll table that discussion because I think it would
partially derail this<br>
discussion.</p>
<p dir="ltr">So one open question is long-term where would this
leave Rally if we want to<br>
bring it in under the QA program. (after splitting up the
functionality to more<br>
conducive with all our existing tools and projects) The one
thing Rally does<br>
here which we don't have an analogous solution for is, for lack
of better term,<br>
the post processing layer. The part that generates the performs
the analysis on<br>
the collected data and generates the graphs. That is something
that we'll have<br>
an eventually need for and that is something that we can work on
turning rally<br>
into as we migrate everything to actually work together.</p>
<p dir="ltr">There are probably also other parts of Rally which
don't fit into an existing<br>
QA program project, (or the QA program in general) and in those
cases we<br>
probably should split them off as smaller projects to implement
that bit. For<br>
example, the SLA stuff Rally has that probably should be a
separate entity as<br>
well, but I'm unsure if that fits under QA program.</p>
<p dir="ltr">My primary concern is the timing for doing all of
this work. We're approaching<br>
J-3 and honestly this feels like it would take the better part
of an entire<br>
cycle to analyze, plan, and then implement. Starting an analysis
of how to do<br>
all of the work at this point I feel would just distract
everyone from<br>
completing our dev goals for the cycle. Probably the Rally team,
if they want<br>
to move forward here, should start the analysis of this
structural split and we<br>
can all pick this up together post-juno.</p>
<p dir="ltr">> <br>
> 2. Rally as an essential operator tool<br>
> Regular benchmarking of OpenStack deployments is a best
practice for<br>
> cloud operators, and a feature that Rally provides. With a
bit of a<br>
> stretch, we could consider that benchmarking is essential
to the<br>
> completion of the OpenStack project mission. That program
could one day<br>
> evolve to include more such "operations best practices"
tools. In<br>
> addition to the slight stretch already mentioned, one
concern here is<br>
> that we still want to have performance testing in QA (which
is clearly<br>
> essential to the production of "OpenStack"). Letting Rally
primarily be<br>
> an operational tool might make that outcome more difficult.<br>
> </p>
<p dir="ltr">So I'm opposed to this option. It feels to me like
this is only on the table<br>
because the Rally team has not done a great job of communicating
or working with<br>
anyone else except for when it comes to either push using Rally,
or this<br>
conversation about adopting Rally.</p>
<p dir="ltr">That being said, looking at a separate "operator
tool" program for Rally doesn't<br>
make much sense to me. There is nothing in Rally that is more or
less operator<br>
tooling specific compared to Tempest or some of the infra
tooling. I fail to see<br>
what in Rally warrants a separate program. To be a bit sardonic,
my question is<br>
if Tempest had a REST API [1][2] then should we move it to the
proposed<br>
operators program too? The other thing, which came out of the
summit, was that<br>
tempest is often used by operators in a loop to get a heartbeat
on their cloud.</p>
<p dir="ltr">My point is that just because a tool is part of the
QA program doesn't mean<br>
it's not useful for operators. I think that's something that
seems to be lost<br>
during this discussion. (or just brushed over) Sure, our first
priority is going<br>
to be on making things work in dev environment and the gate, but
that doesn't<br>
necessarily preclude using things against a production
environment. For tempest<br>
at least, that's something we actually explicitly support. [3]</p>
</blockquote>
+1<br>
We were a little slow out of the gate (so to speak) on this but are
making progress by eliminating all devstack-specific stuff from
tempest configuration, adding support for non-admin parallel tempest
with multiple users, and in general getting rid of discovered
roadblocks to real use. As has been pointed out before, many folks
use tempest against real clouds, including many members of the
tempest core team. IMO this should be considered an equal priority
with gating a dev environment. The biggest problem with that goal is
that tempest gate jobs do not run in most of the vast number of
actual configurations that most real clouds can use and so it is
hard to keep it working with all configurations. But we should
support these cases as best we can.<br>
<br>
-David<br>
<br>
<blockquote
cite="mid:21f0cf0f-a118-4b9e-838a-4bac956b556a@email.android.com"
type="cite">
<p dir="ltr">Maybe, one day there will be a need for a program
like this, but I'm just not<br>
seeing it here with Rally.</p>
<p dir="ltr">> 3. Let Rally be a product on top of OpenStack<br>
> The last option is to not have Rally in any program, and
not consider it<br>
> *essential* to the production of the "OpenStack" integrated
release or<br>
> the completion of the OpenStack project mission. Rally can
happily exist<br>
> as an operator tool on top of OpenStack. It is built as a
monolithic<br>
> product: that approach works very well for external
complementary<br>
> solutions... Also be more integrated in OpenStack or part
of the<br>
> OpenStack programs might come at a cost (slicing some
functionality out<br>
> of rally to make it more a framework and less a product)
that might not<br>
> be what its authors want.</p>
<p dir="ltr">Honestly, if the Rally team wants the project to
remain in it's current form and<br>
scope then I agree that it belongs outside of OpenStack. It
definitely feels<br>
like a product to me, and there is nothing stopping them from
continuing to<br>
operate as they do now on top of OpenStack. I'm sorry, but the
fact that the<br>
docs in the rally tree has a section for user testimonials [4] I
feel speaks a<br>
lot about the intent of the project.</p>
<p dir="ltr">> <br>
> Let's explore each option to see which ones are viable, and
the pros and<br>
> cons of each.<br>
> </p>
<p dir="ltr">I apologize if any of this is somewhat incoherent,
I'm still a bit jet-lagged<br>
so I'm not sure that I'm making much sense.</p>
<p dir="ltr">-Matt Treinish</p>
<p dir="ltr">[1] <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/96163/">https://review.openstack.org/#/c/96163/</a><br>
[2] <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/101093/">https://review.openstack.org/#/c/101093/</a><br>
[3]
<a class="moz-txt-link-freetext" href="http://docs.openstack.org/developer/tempest/overview.html#design-principles">http://docs.openstack.org/developer/tempest/overview.html#design-principles</a><br>
[4]
<a class="moz-txt-link-freetext" href="http://git.openstack.org/cgit/stackforge/rally/tree/doc/user_stories">http://git.openstack.org/cgit/stackforge/rally/tree/doc/user_stories</a><br>
</p>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>