Open Stack

Hi stackers,


I would like to put some more details on current situation.

>
> The issue is with what Rally is in it's
> current form. It's scope is too large and monolithic, and it duplicates
> much of
> the functionality we either already have or need in current QA or Infra
> projects. But, nothing in Rally is designed to be used outside of it. I
> actually
> feel pretty strongly that in it's current form Rally should *not* be a
> part of
> any OpenStack program


Rally is not just a bunch of scripts like tempest, it's more like Nova,
Cinder, and other projects that works out of box and resolve Operators &
Dev use cases in one click.

This architectural design is the main key of Rally success, and why we got
such large adoption and community.

So I'm opposed to this option. It feels to me like this is only on the table
> because the Rally team has not done a great job of communicating or
> working with
> anyone else except for when it comes to either push using Rally, or this
> conversation about adopting Rally.


Actually Rally team done already a bunch of useful work including cross
projects and infra stuff.

Keystone, Glance, Cinder, Neutron and Heat are running rally performance
jobs, that can be used for performance testing, benchmarking, regression
testing (already now). These jobs supports in-tree plugins for all
components (scenarios, load generators, benchmark context) and they can use
Rally fully without interaction with Rally team at all. More about these
jobs:
https://docs.google.com/a/mirantis.com/document/d/1s93IBuyx24dM3SmPcboBp7N47RQedT8u4AJPgOHp9-A/
So I really don't see anything like this in tempest (even in observed
future)


I would like to mention work on OSprofiler (cross service/project profiler)
https://github.com/stackforge/osprofiler (that was done by Rally team)
https://review.openstack.org/#/c/105096/
(btw Glance already accepted it https://review.openstack.org/#/c/105635/ )


My primary concern is the timing for doing all of this work. We're
> approaching
> J-3 and honestly this feels like it would take the better part of an entire
> cycle to analyze, plan, and then implement. Starting an analysis of how to
> do
> all of the work at this point I feel would just distract everyone from
> completing our dev goals for the cycle. Probably the Rally team, if they
> want
> to move forward here, should start the analysis of this structural split
> and we
> can all pick this up together post-juno



Matt, Sean - seriously community is about convincing people, not about
forcing people to do something against their wiliness.  You are making huge
architectural decisions without deep knowledge about what is Rally, what
are use cases, road map, goals and auditory.

IMHO community in my opinion is thing about convincing people. So QA
program should convince Rally team (at least me) to do such changes. Key
secret to convince me, is to say how this will help OpenStack to perform
better.

Currently Rally team see a lot of issues related to this decision:

1) It breaks already existing performance jobs (Heat, Glance, Cinder,
Neutron, Keystone)

2) It breaks functional testing of Rally (that is already done in gates)

2) It makes Rally team depending on Tempest throughput, and what I heard
multiple times from QA team is that performance work is very low priority
and that major goals are to keep gates working. So it will slow down work
of performance team.

3) Brings a ton of questions what should be in Rally and what should be in
Tempest. That are at the moment quite resolved.
https://docs.google.com/a/pavlovic.ru/document/d/137zbrz0KJd6uZwoZEu4BkdKiR_Diobantu0GduS7HnA/edit#heading=h.9ephr9df0new

4) It breaks existing OpenStack team, that is working 100% on performance,
regressions and sla topics. Sorry but there is no such team in tempest.
This directory is not active developed:
https://github.com/openstack/tempest/commits/master/tempest/stress


Matt, Sean, David - what are the real goals of merging Rally to Tempest?
I see a huge harm for OpenStack and companies that are using Rally, and
don't see actually any benefits.
What I heard for now is something like "this decision will make tempest
better"..
But do you care more about Tempest than OpenStack?


Best regards,
Boris Pavlovic




On Tue, Aug 12, 2014 at 12:37 AM, David Kranz <dkranz at redhat.com> wrote:

>  On 08/11/2014 04:21 PM, Matthew Treinish wrote:
>
> I apologize for the delay in my response to this thread, between travelling
> and having a stuck 'a' key on my laptop this is the earliest I could
> respond.
> I opted for a separate branch on this thread to summarize my views and I'll
> respond inline later on some of the previous discussion.
>
> On Wed, Aug 06, 2014 at 12:30:35PM +0200, Thierry Carrez wrote:
> > Hi everyone,
> >
> > At the TC meeting yesterday we discussed Rally program request and
> > incubation request. We quickly dismissed the incubation request, as
> > Rally appears to be able to live happily on top of OpenStack and would
> > benefit from having a release cycle decoupled from the OpenStack
> > "integrated release".
> >
> > That leaves the question of the program. OpenStack programs are created
> > by the Technical Committee, to bless existing efforts and teams that are
> > considered *essential* to the production of the "OpenStack" integrated
> > release and the completion of the OpenStack project mission. There are 3
> > ways to look at Rally and official programs at this point:
> >
> > 1. Rally as an essential QA tool
> > Performance testing (and especially performance regression testing) is
> > an essential QA function, and a feature that Rally provides. If the QA
> > team is happy to use Rally to fill that function, then Rally can
> > obviously be adopted by the (already-existing) QA program. That said,
> > that would put Rally under the authority of the QA PTL, and that raises
> > a few questions due to the current architecture of Rally, which is more
> > product-oriented. There needs to be further discussion between the QA
> > core team and the Rally team to see how that could work and if that
> > option would be acceptable for both sides.
>
> So ideally this is where Rally would belong, the scope of what Rally is
> attempting to do is definitely inside the scope of the QA program. I don't
> see
> any reason why that isn't the case. The issue is with what Rally is in it's
> current form. It's scope is too large and monolithic, and it duplicates
> much of
> the functionality we either already have or need in current QA or Infra
> projects. But, nothing in Rally is designed to be used outside of it. I
> actually
> feel pretty strongly that in it's current form Rally should *not* be a
> part of
> any OpenStack program.
>
> All of the points Sean was making in the other branch on this thread
> (which I'll
> probably respond to later) are a huge concerns I share with Rally. He
> basically
> summarized most of my views on the topic, so I'll try not to rewrite
> everything.
> But, the fact that all of this duplicate functionality was implemented in a
> completely separate manner which is Rally specific and can't really be used
> unless all of Rally is used is of a large concern. What I think the path
> forward here is to have both QA and Rally work together on getting common
> functionality that is re-usable and shareable. Additionally, I have some
> concerns over the methodology that Rally uses for it's performance
> measurement.
> But, I'll table that discussion because I think it would partially derail
> this
> discussion.
>
> So one open question is long-term where would this leave Rally if we want
> to
> bring it in under the QA program. (after splitting up the functionality to
> more
> conducive with all our existing tools and projects) The one thing Rally
> does
> here which we don't have an analogous solution for is, for lack of better
> term,
> the post processing layer. The part that generates the performs the
> analysis on
> the collected data and generates the graphs. That is something that we'll
> have
> an eventually need for and that is something that we can work on turning
> rally
> into as we migrate everything to actually work together.
>
> There are probably also other parts of Rally which don't fit into an
> existing
> QA program project, (or the QA program in general) and in those cases we
> probably should split them off as smaller projects to implement that bit.
> For
> example, the SLA stuff Rally has that probably should be a separate entity
> as
> well, but I'm unsure if that fits under QA program.
>
> My primary concern is the timing for doing all of this work. We're
> approaching
> J-3 and honestly this feels like it would take the better part of an entire
> cycle to analyze, plan, and then implement. Starting an analysis of how to
> do
> all of the work at this point I feel would just distract everyone from
> completing our dev goals for the cycle. Probably the Rally team, if they
> want
> to move forward here, should start the analysis of this structural split
> and we
> can all pick this up together post-juno.
>
> >
> > 2. Rally as an essential operator tool
> > Regular benchmarking of OpenStack deployments is a best practice for
> > cloud operators, and a feature that Rally provides. With a bit of a
> > stretch, we could consider that benchmarking is essential to the
> > completion of the OpenStack project mission. That program could one day
> > evolve to include more such "operations best practices" tools. In
> > addition to the slight stretch already mentioned, one concern here is
> > that we still want to have performance testing in QA (which is clearly
> > essential to the production of "OpenStack"). Letting Rally primarily be
> > an operational tool might make that outcome more difficult.
> >
>
> So I'm opposed to this option. It feels to me like this is only on the
> table
> because the Rally team has not done a great job of communicating or
> working with
> anyone else except for when it comes to either push using Rally, or this
> conversation about adopting Rally.
>
> That being said, looking at a separate "operator tool" program for Rally
> doesn't
> make much sense to me. There is nothing in Rally that is more or less
> operator
> tooling specific compared to Tempest or some of the infra tooling. I fail
> to see
> what in Rally warrants a separate program. To be a bit sardonic, my
> question is
> if Tempest had a REST API [1][2] then should we move it to the proposed
> operators program too? The other thing, which came out of the summit, was
> that
> tempest is often used by operators in a loop to get a heartbeat on their
> cloud.
>
> My point is that just because a tool is part of the QA program doesn't mean
> it's not useful for operators. I think that's something that seems to be
> lost
> during this discussion. (or just brushed over) Sure, our first priority is
> going
> to be on making things work in dev environment and the gate, but that
> doesn't
> necessarily preclude using things against a production environment. For
> tempest
> at least, that's something we actually explicitly support. [3]
>
> +1
> We were a little slow out of the gate (so to speak) on this but are making
> progress by eliminating all devstack-specific stuff from tempest
> configuration, adding support for non-admin parallel tempest with multiple
> users, and in general getting rid of discovered roadblocks to real use. As
> has been pointed out before, many folks use tempest against real clouds,
> including many members of the tempest core team. IMO this should be
> considered an equal priority with gating a dev environment. The biggest
> problem with that goal is that tempest gate jobs do not run in most of the
> vast number of actual configurations that most real clouds can use and so
> it is hard to keep it working with all configurations. But we should
> support these cases as best we can.
>
>  -David
>
>  Maybe, one day there will be a need for a program like this, but I'm
> just not
> seeing it here with Rally.
>
> > 3. Let Rally be a product on top of OpenStack
> > The last option is to not have Rally in any program, and not consider it
> > *essential* to the production of the "OpenStack" integrated release or
> > the completion of the OpenStack project mission. Rally can happily exist
> > as an operator tool on top of OpenStack. It is built as a monolithic
> > product: that approach works very well for external complementary
> > solutions... Also be more integrated in OpenStack or part of the
> > OpenStack programs might come at a cost (slicing some functionality out
> > of rally to make it more a framework and less a product) that might not
> > be what its authors want.
>
> Honestly, if the Rally team wants the project to remain in it's current
> form and
> scope then I agree that it belongs outside of OpenStack. It definitely
> feels
> like a product to me, and there is nothing stopping them from continuing to
> operate as they do now on top of OpenStack. I'm sorry, but the fact that
> the
> docs in the rally tree has a section for user testimonials [4] I feel
> speaks a
> lot about the intent of the project.
>
> >
> > Let's explore each option to see which ones are viable, and the pros and
> > cons of each.
> >
>
> I apologize if any of this is somewhat incoherent, I'm still a bit
> jet-lagged
> so I'm not sure that I'm making much sense.
>
> -Matt Treinish
>
> [1] https://review.openstack.org/#/c/96163/
> [2] https://review.openstack.org/#/c/101093/
> [3]
> http://docs.openstack.org/developer/tempest/overview.html#design-principles
> [4] http://git.openstack.org/cgit/stackforge/rally/tree/doc/user_stories
>
>
> _______________________________________________
> OpenStack-dev mailing listOpenStack-dev at lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140812/e60faf23/attachment.html>