Open Stack

> From: Malini Kamalambal [mailto:malini.kamalambal at RACKSPACE.COM]
<snip>
> 
> We are talking about different levels of testing,
> 
> 1. Unit tests - which everybody agrees should be in the individual
> project
> itself
> 2. System Tests - 'System' referring to (& limited to), all the
> components
> that make up the project. These are also the functional tests for the
> project.
> 3. Integration Tests - This is to verify that the OS components
> interact
> well and don't break other components -Keystone being the most obvious
> example. This is where I see getting the maximum mileage out of
> Tempest.
> 
> I see value in projects taking ownership of the System Tests - because
> if
> the project is not 'functionally ready', it is not ready to integrate
> with
> other components of Openstack.
> But for this approach to be successful, projects should have diversity
> in
> the team composition - we need more testers who focus on creating these
> tests.
> This will keep the teams honest in their quality standards.

+1000  I love your approach to this.  You are right.  Functional tests for the project, that exist in an environment, but that exercise the intricacies of just the project aren't there for most projects, but really should be.  And these tests should be exercised against new code before the code enters the gerrit/Jenkins stream. But, as Malini points out, it's at most a dream for most projects as the test developers just aren't part of most projects.


> As long as individual projects cannot guarantee functional test
> coverage,
> we will need more tests in Tempest.
> But that will shift focus away from Integration Testing, which can be
> done
> ONLY in Tempest.

+1  This is also an important point.  If functional testing belonged to the projects, then most of these tests would be run before a tempest test was ever run and would not need to be part of the integrated tests, except as a subset that demonstrate the functioning integration with other projects.

> 
> Regardless of whatever we end up deciding, it will be good to have
> these
> discussions sooner than later.
> This will help at least the new projects to move in the right
> direction.

Maybe a summit topic?  How do we push functional testing into the project level development?

--Rocky

> 
> -Malini
> 
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev