Open Stack

On 08/06/2014 05:48 PM, John Griffith wrote:
> I have to agree with Duncan here.  I also don't know if I fully 
> understand the limit in options.  Stress test seems like it 
> could/should be different (again overlap isn't a horrible thing) and I 
> don't see it as siphoning off resources so not sure of the issue. 
>  We've become quite wrapped up in projects, programs and the like 
> lately and it seems to hinder forward progress more than anything else.
>
> I'm also not convinced that Tempest is where all things belong, in 
> fact I've been thinking more and more that a good bit of what Tempest 
> does today should fall more on the responsibility of the projects 
> themselves.  For example functional testing of features etc, ideally 
> I'd love to have more of that fall on the projects and their 
> respective teams.  That might even be something as simple to start as 
> saying "if you contribute a new feature, you have to also provide a 
> link to a contribution to the Tempest test-suite that checks it". 
>  Sort of like we do for unit tests, cross-project tracking is 
> difficult of course, but it's a start.  The other idea is maybe 
> functional test harnesses live in their respective projects.
>
> Honestly I think who better to write tests for a project than the 
> folks building and contributing to the project.  At some point IMO the 
> QA team isn't going to scale.  I wonder if maybe we should be thinking 
> about proposals for delineating responsibility and goals in terms of 
> functional testing?
>
>
>
All good points. Your last paragraph was discussed by the QA team 
leading up to and at the Atlanta summit. The conclusion was that the 
api/functional tests focused on a single project should be part of that 
project. As Sean said, we can envision there being half (or some other 
much smaller number) as many such tests in tempest going forward.

Details are under discussion, but the way this is likely to play out is 
that individual projects will start by creating their own functional 
tests outside of tempest. Swift already does this and neutron seems to 
be moving in that direction. There is a spec to break out parts of 
tempest 
(https://github.com/openstack/qa-specs/blob/master/specs/tempest-library.rst) 
into a library that might be used by projects implementing functional 
tests.

Once a project has "sufficient" functional testing, we can consider 
removing its api tests from tempest. This is a bit tricky because 
tempest needs to cover *all* cross-project interactions. In this 
respect, there is no clear line in tempest between scenario tests which 
have this goal explicitly, and api tests which may also involve 
interactions that might not be covered in a scenario. So we will need a 
principled way to make sure there is complete cross-project coverage in 
tempest with a smaller number of api tests.

  -David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140811/3b757f3a/attachment.html>