Open Stack

On 08/06/2014 06:30 AM, Thierry Carrez wrote:
> Hi everyone,
> 
> At the TC meeting yesterday we discussed Rally program request and
> incubation request. We quickly dismissed the incubation request, as
> Rally appears to be able to live happily on top of OpenStack and would
> benefit from having a release cycle decoupled from the OpenStack
> "integrated release".
> 
> That leaves the question of the program. OpenStack programs are created
> by the Technical Committee, to bless existing efforts and teams that are
> considered *essential* to the production of the "OpenStack" integrated
> release and the completion of the OpenStack project mission. There are 3
> ways to look at Rally and official programs at this point:
> 
> 1. Rally as an essential QA tool
> Performance testing (and especially performance regression testing) is
> an essential QA function, and a feature that Rally provides. If the QA
> team is happy to use Rally to fill that function, then Rally can
> obviously be adopted by the (already-existing) QA program. That said,
> that would put Rally under the authority of the QA PTL, and that raises
> a few questions due to the current architecture of Rally, which is more
> product-oriented. There needs to be further discussion between the QA
> core team and the Rally team to see how that could work and if that
> option would be acceptable for both sides.
> 
> 2. Rally as an essential operator tool
> Regular benchmarking of OpenStack deployments is a best practice for
> cloud operators, and a feature that Rally provides. With a bit of a
> stretch, we could consider that benchmarking is essential to the
> completion of the OpenStack project mission. That program could one day
> evolve to include more such "operations best practices" tools. In
> addition to the slight stretch already mentioned, one concern here is
> that we still want to have performance testing in QA (which is clearly
> essential to the production of "OpenStack"). Letting Rally primarily be
> an operational tool might make that outcome more difficult.
> 
> 3. Let Rally be a product on top of OpenStack
> The last option is to not have Rally in any program, and not consider it
> *essential* to the production of the "OpenStack" integrated release or
> the completion of the OpenStack project mission. Rally can happily exist
> as an operator tool on top of OpenStack. It is built as a monolithic
> product: that approach works very well for external complementary
> solutions... Also be more integrated in OpenStack or part of the
> OpenStack programs might come at a cost (slicing some functionality out
> of rally to make it more a framework and less a product) that might not
> be what its authors want.
> 
> Let's explore each option to see which ones are viable, and the pros and
> cons of each.

My feeling right now is that Rally is trying to accomplish too much at
the start (both #1 and #2).  I would rather see the project focus on
doing one of them as best as it can before increasing scope.

It's my opinion that #1 is the most important thing that Rally can be
doing to help ensure the success of OpenStack, so I'd like to explore
the "Rally as a QA tool" in more detail to start with.

>From the TC meeting, it seems that the QA group (via sdague, at least)
has provided some feedback to Rally over the last several months.  I
would really like to see an analysis and write-up from the QA group on
the current state of Rally and how it may (or may not) be able to serve
the performance QA needs.

-- 
Russell Bryant