Open Stack

I apologize for the delay in my response to this thread, between travelling
and having a stuck 'a' key on my laptop this is the earliest I could respond.
I opted for a separate branch on this thread to summarize my views and I'll
respond inline later on some of the previous discussion.

On Wed, Aug 06, 2014 at 12:30:35PM +0200, Thierry Carrez wrote:
> Hi everyone,
> 
> At the TC meeting yesterday we discussed Rally program request and
> incubation request. We quickly dismissed the incubation request, as
> Rally appears to be able to live happily on top of OpenStack and would
> benefit from having a release cycle decoupled from the OpenStack
> "integrated release".
> 
> That leaves the question of the program. OpenStack programs are created
> by the Technical Committee, to bless existing efforts and teams that are
> considered *essential* to the production of the "OpenStack" integrated
> release and the completion of the OpenStack project mission. There are 3
> ways to look at Rally and official programs at this point:
> 
> 1. Rally as an essential QA tool
> Performance testing (and especially performance regression testing) is
> an essential QA function, and a feature that Rally provides. If the QA
> team is happy to use Rally to fill that function, then Rally can
> obviously be adopted by the (already-existing) QA program. That said,
> that would put Rally under the authority of the QA PTL, and that raises
> a few questions due to the current architecture of Rally, which is more
> product-oriented. There needs to be further discussion between the QA
> core team and the Rally team to see how that could work and if that
> option would be acceptable for both sides.

So ideally this is where Rally would belong, the scope of what Rally is
attempting to do is definitely inside the scope of the QA program. I don't see
any reason why that isn't the case. The issue is with what Rally is in it's
current form. It's scope is too large and monolithic, and it duplicates much of
the functionality we either already have or need in current QA or Infra
projects. But, nothing in Rally is designed to be used outside of it. I actually
feel pretty strongly that in it's current form Rally should *not* be a part of
any OpenStack program.

All of the points Sean was making in the other branch on this thread (which I'll
probably respond to later) are a huge concerns I share with Rally. He basically
summarized most of my views on the topic, so I'll try not to rewrite everything.
But, the fact that all of this duplicate functionality was implemented in a
completely separate manner which is Rally specific and can't really be used
unless all of Rally is used is of a large concern. What I think the path
forward here is to have both QA and Rally work together on getting common
functionality that is re-usable and shareable. Additionally, I have some
concerns over the methodology that Rally uses for it's performance measurement.
But, I'll table that discussion because I think it would partially derail this
discussion.

So one open question is long-term where would this leave Rally if we want to
bring it in under the QA program. (after splitting up the functionality to more
conducive with all our existing tools and projects) The one thing Rally does
here which we don't have an analogous solution for is, for lack of better term,
the post processing layer. The part that generates the performs the analysis on
the collected data and generates the graphs. That is something that we'll have
an eventually need for and that is something that we can work on turning rally
into as we migrate everything to actually work together.

There are probably also other parts of Rally which don't fit into an existing
QA program project, (or the QA program in general) and in those cases we
probably should split them off as smaller projects to implement that bit. For
example, the SLA stuff Rally has that probably should be a separate entity as
well, but I'm unsure if that fits under QA program.

My primary concern is the timing for doing all of this work. We're approaching
J-3 and honestly this feels like it would take the better part of an entire
cycle to analyze, plan, and then implement. Starting an analysis of how to do
all of the work at this point I feel would just distract everyone from
completing our dev goals for the cycle. Probably the Rally team, if they want
to move forward here, should start the analysis of this structural split and we
can all pick this up together post-juno.

> 
> 2. Rally as an essential operator tool
> Regular benchmarking of OpenStack deployments is a best practice for
> cloud operators, and a feature that Rally provides. With a bit of a
> stretch, we could consider that benchmarking is essential to the
> completion of the OpenStack project mission. That program could one day
> evolve to include more such "operations best practices" tools. In
> addition to the slight stretch already mentioned, one concern here is
> that we still want to have performance testing in QA (which is clearly
> essential to the production of "OpenStack"). Letting Rally primarily be
> an operational tool might make that outcome more difficult.
> 

So I'm opposed to this option. It feels to me like this is only on the table
because the Rally team has not done a great job of communicating or working with
anyone else except for when it comes to either push using Rally, or this
conversation about adopting Rally.

That being said, looking at a separate "operator tool" program for Rally doesn't
make much sense to me. There is nothing in Rally that is more or less operator
tooling specific compared to Tempest or some of the infra tooling. I fail to see
what in Rally warrants a separate program. To be a bit sardonic, my question is
if Tempest had a REST API [1][2] then should we move it to the proposed
operators program too? The other thing, which came out of the summit, was that
tempest is often used by operators in a loop to get a heartbeat on their cloud.

My point is that just because a tool is part of the QA program doesn't mean
it's not useful for operators. I think that's something that seems to be lost
during this discussion. (or just brushed over) Sure, our first priority is going
to be on making things work in dev environment and the gate, but that doesn't
necessarily preclude using things against a production environment. For tempest
at least, that's something we actually explicitly support. [3]

Maybe, one day there will be a need for a program like this, but I'm just not
seeing it here with Rally.

> 3. Let Rally be a product on top of OpenStack
> The last option is to not have Rally in any program, and not consider it
> *essential* to the production of the "OpenStack" integrated release or
> the completion of the OpenStack project mission. Rally can happily exist
> as an operator tool on top of OpenStack. It is built as a monolithic
> product: that approach works very well for external complementary
> solutions... Also be more integrated in OpenStack or part of the
> OpenStack programs might come at a cost (slicing some functionality out
> of rally to make it more a framework and less a product) that might not
> be what its authors want.

Honestly, if the Rally team wants the project to remain in it's current form and
scope then I agree that it belongs outside of OpenStack. It definitely feels
like a product to me, and there is nothing stopping them from continuing to
operate as they do now on top of OpenStack. I'm sorry, but the fact that the
docs in the rally tree has a section for user testimonials [4] I feel speaks a
lot about the intent of the project.

> 
> Let's explore each option to see which ones are viable, and the pros and
> cons of each.
> 

I apologize if any of this is somewhat incoherent, I'm still a bit jet-lagged
so I'm not sure that I'm making much sense.

-Matt Treinish

[1] https://review.openstack.org/#/c/96163/
[2] https://review.openstack.org/#/c/101093/
[3] http://docs.openstack.org/developer/tempest/overview.html#design-principles
[4] http://git.openstack.org/cgit/stackforge/rally/tree/doc/user_stories
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140811/ab6d5d53/attachment.html>