Open Stack

On 08/11/2014 04:21 PM, Matthew Treinish wrote:
>
> I apologize for the delay in my response to this thread, between 
> travelling
> and having a stuck 'a' key on my laptop this is the earliest I could 
> respond.
> I opted for a separate branch on this thread to summarize my views and 
> I'll
> respond inline later on some of the previous discussion.
>
> On Wed, Aug 06, 2014 at 12:30:35PM +0200, Thierry Carrez wrote:
> > Hi everyone,
> >
> > At the TC meeting yesterday we discussed Rally program request and
> > incubation request. We quickly dismissed the incubation request, as
> > Rally appears to be able to live happily on top of OpenStack and would
> > benefit from having a release cycle decoupled from the OpenStack
> > "integrated release".
> >
> > That leaves the question of the program. OpenStack programs are created
> > by the Technical Committee, to bless existing efforts and teams that are
> > considered *essential* to the production of the "OpenStack" integrated
> > release and the completion of the OpenStack project mission. There are 3
> > ways to look at Rally and official programs at this point:
> >
> > 1. Rally as an essential QA tool
> > Performance testing (and especially performance regression testing) is
> > an essential QA function, and a feature that Rally provides. If the QA
> > team is happy to use Rally to fill that function, then Rally can
> > obviously be adopted by the (already-existing) QA program. That said,
> > that would put Rally under the authority of the QA PTL, and that raises
> > a few questions due to the current architecture of Rally, which is more
> > product-oriented. There needs to be further discussion between the QA
> > core team and the Rally team to see how that could work and if that
> > option would be acceptable for both sides.
>
> So ideally this is where Rally would belong, the scope of what Rally is
> attempting to do is definitely inside the scope of the QA program. I 
> don't see
> any reason why that isn't the case. The issue is with what Rally is in 
> it's
> current form. It's scope is too large and monolithic, and it 
> duplicates much of
> the functionality we either already have or need in current QA or Infra
> projects. But, nothing in Rally is designed to be used outside of it. 
> I actually
> feel pretty strongly that in it's current form Rally should *not* be a 
> part of
> any OpenStack program.
>
> All of the points Sean was making in the other branch on this thread 
> (which I'll
> probably respond to later) are a huge concerns I share with Rally. He 
> basically
> summarized most of my views on the topic, so I'll try not to rewrite 
> everything.
> But, the fact that all of this duplicate functionality was implemented 
> in a
> completely separate manner which is Rally specific and can't really be 
> used
> unless all of Rally is used is of a large concern. What I think the path
> forward here is to have both QA and Rally work together on getting common
> functionality that is re-usable and shareable. Additionally, I have some
> concerns over the methodology that Rally uses for it's performance 
> measurement.
> But, I'll table that discussion because I think it would partially 
> derail this
> discussion.
>
> So one open question is long-term where would this leave Rally if we 
> want to
> bring it in under the QA program. (after splitting up the 
> functionality to more
> conducive with all our existing tools and projects) The one thing 
> Rally does
> here which we don't have an analogous solution for is, for lack of 
> better term,
> the post processing layer. The part that generates the performs the 
> analysis on
> the collected data and generates the graphs. That is something that 
> we'll have
> an eventually need for and that is something that we can work on 
> turning rally
> into as we migrate everything to actually work together.
>
> There are probably also other parts of Rally which don't fit into an 
> existing
> QA program project, (or the QA program in general) and in those cases we
> probably should split them off as smaller projects to implement that 
> bit. For
> example, the SLA stuff Rally has that probably should be a separate 
> entity as
> well, but I'm unsure if that fits under QA program.
>
> My primary concern is the timing for doing all of this work. We're 
> approaching
> J-3 and honestly this feels like it would take the better part of an 
> entire
> cycle to analyze, plan, and then implement. Starting an analysis of 
> how to do
> all of the work at this point I feel would just distract everyone from
> completing our dev goals for the cycle. Probably the Rally team, if 
> they want
> to move forward here, should start the analysis of this structural 
> split and we
> can all pick this up together post-juno.
>
> >
> > 2. Rally as an essential operator tool
> > Regular benchmarking of OpenStack deployments is a best practice for
> > cloud operators, and a feature that Rally provides. With a bit of a
> > stretch, we could consider that benchmarking is essential to the
> > completion of the OpenStack project mission. That program could one day
> > evolve to include more such "operations best practices" tools. In
> > addition to the slight stretch already mentioned, one concern here is
> > that we still want to have performance testing in QA (which is clearly
> > essential to the production of "OpenStack"). Letting Rally primarily be
> > an operational tool might make that outcome more difficult.
> >
>
> So I'm opposed to this option. It feels to me like this is only on the 
> table
> because the Rally team has not done a great job of communicating or 
> working with
> anyone else except for when it comes to either push using Rally, or this
> conversation about adopting Rally.
>
> That being said, looking at a separate "operator tool" program for 
> Rally doesn't
> make much sense to me. There is nothing in Rally that is more or less 
> operator
> tooling specific compared to Tempest or some of the infra tooling. I 
> fail to see
> what in Rally warrants a separate program. To be a bit sardonic, my 
> question is
> if Tempest had a REST API [1][2] then should we move it to the proposed
> operators program too? The other thing, which came out of the summit, 
> was that
> tempest is often used by operators in a loop to get a heartbeat on 
> their cloud.
>
> My point is that just because a tool is part of the QA program doesn't 
> mean
> it's not useful for operators. I think that's something that seems to 
> be lost
> during this discussion. (or just brushed over) Sure, our first 
> priority is going
> to be on making things work in dev environment and the gate, but that 
> doesn't
> necessarily preclude using things against a production environment. 
> For tempest
> at least, that's something we actually explicitly support. [3]
>
+1
We were a little slow out of the gate (so to speak) on this but are 
making progress by eliminating all devstack-specific stuff from tempest 
configuration, adding support for non-admin parallel tempest with 
multiple users, and in general getting rid of discovered roadblocks to 
real use. As has been pointed out before, many folks use tempest against 
real clouds, including many members of the tempest core team. IMO this 
should be considered an equal priority with gating a dev environment. 
The biggest problem with that goal is that tempest gate jobs do not run 
in most of the vast number of actual configurations that most real 
clouds can use and so it is hard to keep it working with all 
configurations. But we should support these cases as best we can.

  -David

> Maybe, one day there will be a need for a program like this, but I'm 
> just not
> seeing it here with Rally.
>
> > 3. Let Rally be a product on top of OpenStack
> > The last option is to not have Rally in any program, and not consider it
> > *essential* to the production of the "OpenStack" integrated release or
> > the completion of the OpenStack project mission. Rally can happily exist
> > as an operator tool on top of OpenStack. It is built as a monolithic
> > product: that approach works very well for external complementary
> > solutions... Also be more integrated in OpenStack or part of the
> > OpenStack programs might come at a cost (slicing some functionality out
> > of rally to make it more a framework and less a product) that might not
> > be what its authors want.
>
> Honestly, if the Rally team wants the project to remain in it's 
> current form and
> scope then I agree that it belongs outside of OpenStack. It definitely 
> feels
> like a product to me, and there is nothing stopping them from 
> continuing to
> operate as they do now on top of OpenStack. I'm sorry, but the fact 
> that the
> docs in the rally tree has a section for user testimonials [4] I feel 
> speaks a
> lot about the intent of the project.
>
> >
> > Let's explore each option to see which ones are viable, and the pros and
> > cons of each.
> >
>
> I apologize if any of this is somewhat incoherent, I'm still a bit 
> jet-lagged
> so I'm not sure that I'm making much sense.
>
> -Matt Treinish
>
> [1] https://review.openstack.org/#/c/96163/
> [2] https://review.openstack.org/#/c/101093/
> [3] 
> http://docs.openstack.org/developer/tempest/overview.html#design-principles
> [4] http://git.openstack.org/cgit/stackforge/rally/tree/doc/user_stories
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140811/37ba0a45/attachment.html>