Open Stack

We are talking about different levels of testing,

1. Unit tests - which everybody agrees should be in the individual project
itself
2. System Tests - 'System' referring to (& limited to), all the components
that make up the project. These are also the functional tests for the
project.
3. Integration Tests - This is to verify that the OS components interact
well and don't break other components -Keystone being the most obvious
example. This is where I see getting the maximum mileage out of Tempest.

"Its not easy to detect what the integration points with other projects are, any project can use any stable API from any other project. Because of this all OpenStack APIs should fit into this category. "

Any project can use any stable API –but that does not make all API tests , Integration Tests.
A test becomes Integration test when it has two or more projects interacting in the test.

Individual projects should be held accountable to make sure that their API's work – no matter who consumes them.
We should be able to treat the project as a complete system, make API calls and validate that the response matches the API definitions.
Identifying issues earlier in the pipeline reduces the Total Cost of Quality.

I agree that Integration Testing is hard. It is complicated because it requires knowledge of how systems interact with each other – and knowledge comes from a lot of time spent on analysis.
It requires people with project expertise to talk to each other & identify possible test cases.
openstack-qa is the ideal forum to do this.
Holding projects responsible for their functionality will help the QA team focus on complicated integration tests.

"Having a second group writing tests to Nova's public APIs has been really helpful in keeping us honest as well."

Sounds like a testimonial for more project level testing :)


I see value in projects taking ownership of the System Tests - because if
the project is not 'functionally ready', it is not ready to integrate with
other components of Openstack.

"What do you mean by not ready?"

'Functionally Ready' - The units that make up a project can work together as a system,  all API's have been exercised with positive & negative test cases by treating the project as a complete system.
There are no known critical bugs. The point here being identify as many issues as possible, earlier in the game.

But for this approach to be successful, projects should have diversity in
the team composition - we need more testers who focus on creating these
tests.
This will keep the teams honest in their quality standards.

As long as individual projects cannot guarantee functional test coverage,
we will need more tests in Tempest.
But that will shift focus away from Integration Testing, which can be done
ONLY in Tempest.

Regardless of whatever we end up deciding, it will be good to have these
discussions sooner than later.
This will help at least the new projects to move in the right direction.

-Malini








_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140325/e5f8eee5/attachment.html>