Open Stack

On Wed, 2014-08-06 at 15:48 -0600, John Griffith wrote:
> I have to agree with Duncan here.  I also don't know if I fully
> understand the limit in options.  Stress test seems like it
> could/should be different (again overlap isn't a horrible thing) and I
> don't see it as siphoning off resources so not sure of the issue.
>  We've become quite wrapped up in projects, programs and the like
> lately and it seems to hinder forward progress more than anything
> else.
h
> 
> I'm also not convinced that Tempest is where all things belong, in
> fact I've been thinking more and more that a good bit of what Tempest
> does today should fall more on the responsibility of the projects
> themselves.  For example functional testing of features etc, ideally
> I'd love to have more of that fall on the projects and their
> respective teams.  That might even be something as simple to start as
> saying "if you contribute a new feature, you have to also provide a
> link to a contribution to the Tempest test-suite that checks it".
>  Sort of like we do for unit tests, cross-project tracking is
> difficult of course, but it's a start.  The other idea is maybe
> functional test harnesses live in their respective projects.
> 

Couldn't we reduce the scope of tempest (and rally) : make tempest the
API verification and rally the secenario/performance tester? Make each
tool do less, but better. My point being to split the projects by
functionality so there is less need to share code and stomp on each
other's toes.

> 
> 
> Honestly I think who better to write tests for a project than the
> folks building and contributing to the project.  At some point IMO the
> QA team isn't going to scale.  I wonder if maybe we should be thinking
> about proposals for delineating responsibility and goals in terms of
> functional testing?
> 

This is planned, I believe.

-Angus

> 
> 
> 
> 
> 
> On Wed, Aug 6, 2014 at 12:25 PM, Duncan Thomas
> <duncan.thomas at gmail.com> wrote:
>         I'm not following here - you complain about rally being
>         monolithic,
>         then suggest that parts of it should be baked into tempest - a
>         tool
>         that is already huge and difficult to get into. I'd rather see
>         tools
>         that do one thing well and some overlap than one tool to rule
>         them
>         all.

+1

>         
>         On 6 August 2014 14:44, Sean Dague <sean at dague.net> wrote:
>         > On 08/06/2014 09:11 AM, Russell Bryant wrote:
>         >> On 08/06/2014 06:30 AM, Thierry Carrez wrote:
>         >>> Hi everyone,
>         >>>
>         >>> At the TC meeting yesterday we discussed Rally program
>         request and
>         >>> incubation request. We quickly dismissed the incubation
>         request, as
>         >>> Rally appears to be able to live happily on top of
>         OpenStack and would
>         >>> benefit from having a release cycle decoupled from the
>         OpenStack
>         >>> "integrated release".
>         >>>
>         >>> That leaves the question of the program. OpenStack
>         programs are created
>         >>> by the Technical Committee, to bless existing efforts and
>         teams that are
>         >>> considered *essential* to the production of the
>         "OpenStack" integrated
>         >>> release and the completion of the OpenStack project
>         mission. There are 3
>         >>> ways to look at Rally and official programs at this point:
>         >>>
>         >>> 1. Rally as an essential QA tool
>         >>> Performance testing (and especially performance regression
>         testing) is
>         >>> an essential QA function, and a feature that Rally
>         provides. If the QA
>         >>> team is happy to use Rally to fill that function, then
>         Rally can
>         >>> obviously be adopted by the (already-existing) QA program.
>         That said,
>         >>> that would put Rally under the authority of the QA PTL,
>         and that raises
>         >>> a few questions due to the current architecture of Rally,
>         which is more
>         >>> product-oriented. There needs to be further discussion
>         between the QA
>         >>> core team and the Rally team to see how that could work
>         and if that
>         >>> option would be acceptable for both sides.
>         >>>
>         >>> 2. Rally as an essential operator tool
>         >>> Regular benchmarking of OpenStack deployments is a best
>         practice for
>         >>> cloud operators, and a feature that Rally provides. With a
>         bit of a
>         >>> stretch, we could consider that benchmarking is essential
>         to the
>         >>> completion of the OpenStack project mission. That program
>         could one day
>         >>> evolve to include more such "operations best practices"
>         tools. In
>         >>> addition to the slight stretch already mentioned, one
>         concern here is
>         >>> that we still want to have performance testing in QA
>         (which is clearly
>         >>> essential to the production of "OpenStack"). Letting Rally
>         primarily be
>         >>> an operational tool might make that outcome more
>         difficult.
>         >>>
>         >>> 3. Let Rally be a product on top of OpenStack
>         >>> The last option is to not have Rally in any program, and
>         not consider it
>         >>> *essential* to the production of the "OpenStack"
>         integrated release or
>         >>> the completion of the OpenStack project mission. Rally can
>         happily exist
>         >>> as an operator tool on top of OpenStack. It is built as a
>         monolithic
>         >>> product: that approach works very well for external
>         complementary
>         >>> solutions... Also be more integrated in OpenStack or part
>         of the
>         >>> OpenStack programs might come at a cost (slicing some
>         functionality out
>         >>> of rally to make it more a framework and less a product)
>         that might not
>         >>> be what its authors want.
>         >>>
>         >>> Let's explore each option to see which ones are viable,
>         and the pros and
>         >>> cons of each.
>         >>
>         >> My feeling right now is that Rally is trying to accomplish
>         too much at
>         >> the start (both #1 and #2).  I would rather see the project
>         focus on
>         >> doing one of them as best as it can before increasing
>         scope.
>         >>
>         >> It's my opinion that #1 is the most important thing that
>         Rally can be
>         >> doing to help ensure the success of OpenStack, so I'd like
>         to explore
>         >> the "Rally as a QA tool" in more detail to start with.
>         >
>         > I want to clarify some things. I don't think that rally in
>         it's current
>         > form belongs in any OpenStack project. It's a giant
>         monolythic tool,
>         > which is apparently a design point. That's the wrong design
>         point for an
>         > OpenStack project.
>         >
>         > For instance:
>         >
>         >
>         https://github.com/stackforge/rally/tree/master/rally/benchmark/scenarios should
>         > all be tests in Tempest (and actually today mostly are via
>         API tests).
>         > There is an existing stress framework in Tempest which does
>         the
>         > repetitive looping that rally does on these already. This
>         fact has been
>         > brought up before.
>         >
>         >
>         https://github.com/stackforge/rally/tree/master/rally/verification/verifiers
>         > - should be baked back into Tempest (at least on the results
>         side,
>         > though diving in there now it looks largely duplicative from
>         existing
>         > subunit to html code).
>         >
>         >
>         https://github.com/stackforge/rally/blob/master/rally/db/api.py - is
>         > largely (not entirely) what we'd like from a long term
>         trending piece
>         > that subunit2sql is working on. Again this was just all
>         thrown into the
>         > Rally db instead of thinking about how to split it off.
>         Also, notable
>         > here is there are some fundamental testr bugs (like worker
>         > misallocation) which mean the data is massively dirty today.
>         It would be
>         > good for people to actually work on fixing those things.
>         >
>         > The parts that should stay outside of Tempest are the setup
>         tool
>         > (separation of concerns is that Tempest is the load runner,
>         not the
>         > setup environment) and any of the SLA portions.
>         >
>         > I think rally brings forward a good point about making
>         Tempest easier to
>         > run. But I think that shouldn't be done outside Tempest.
>         Making the test
>         > tool easier to use should be done in the tool itself. If
>         that means
>         > adding a tempest cmd or such, so be it. Note this was a
>         topic for
>         > discussion at last summit:
>         > https://wiki.openstack.org/wiki/Summit/Juno/Etherpads#QA
>         >
>         >> From the TC meeting, it seems that the QA group (via
>         sdague, at least)
>         >> has provided some feedback to Rally over the last several
>         months.  I
>         >> would really like to see an analysis and write-up from the
>         QA group on
>         >> the current state of Rally and how it may (or may not) be
>         able to serve
>         >> the performance QA needs.
>         >
>         > Something that we need to figure out is given where we are
>         in the
>         > release cycle do we want to ask the QA team to go off and do
>         Rally deep
>         > dive now to try to pull it apart into the parts that make
>         sense for
>         > other programs to take in. There are always trade offs.
>         >
>         > Like the fact that right now the rally team is proposing
>         gate jobs which
>         > have some overlap to the existing largeops jobs. Did they
>         start a
>         > conversation about it? Nope. They just went off to do their
>         thing
>         > instead. https://review.openstack.org/#/c/112251/
>         >
>         > So now we're going to run 2 jobs that do very similar
>         things, with
>         > different teams adjusting the test loads. Which I think is
>         basically
>         > madness.
>         >
>         >         -Sean
>         >
>         > --
>         > Sean Dague
>         > http://dague.net
>         >
>         > _______________________________________________
>         > OpenStack-dev mailing list
>         > OpenStack-dev at lists.openstack.org
>         >
>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>         
>         
>         
>         
>         --
>         Duncan Thomas
>         
>         _______________________________________________
>         OpenStack-dev mailing list
>         OpenStack-dev at lists.openstack.org
>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>         
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140807/c0d3ff1e/attachment.pgp>