<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<span id="OLK_SRC_BODY_SECTION">
<div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb">
<div class="h5"><br>
<br>
<br>
</div>
</div>
We are talking about different levels of testing,<br>
<br>
1. Unit tests - which everybody agrees should be in the individual project<br>
itself<br>
2. System Tests - 'System' referring to (& limited to), all the components<br>
that make up the project. These are also the functional tests for the<br>
project.<br>
3. Integration Tests - This is to verify that the OS components interact<br>
well and don't break other components -Keystone being the most obvious<br>
example. This is where I see getting the maximum mileage out of Tempest.<br>
</blockquote>
<div><br>
</div>
<div>"Its not easy to detect what the integration points with other projects are, any project can use any stable API from any other project. Because of this all OpenStack APIs should fit into this category. "</div>
</div>
</div>
</div>
</div>
</div>
</span>
<div><br>
</div>
<div>Any project can use any stable API –but that does not make all API tests , Integration Tests.</div>
<div>A test becomes Integration test when it has two or more projects interacting in the test.</div>
<div><br>
</div>
<div>Individual projects should be held accountable to make sure that their API's work – no matter who consumes them.</div>
<div>We should be able to treat the project as a complete system, make API calls and validate that the response matches the API definitions.</div>
<div>Identifying issues earlier in the pipeline reduces the Total Cost of Quality.</div>
<div><br>
</div>
<div>I agree that Integration Testing is hard. It is complicated because it requires knowledge of how systems interact with each other – and knowledge comes from a lot of time spent on analysis.</div>
<div>It requires people with project expertise to talk to each other & identify possible test cases.</div>
<div>openstack-qa is the ideal forum to do this.</div>
<div>Holding projects responsible for their functionality will help the QA team focus on complicated integration tests.</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>"Having a second group writing tests to Nova's public APIs has been really helpful in keeping us honest as well."</div>
</div>
</div>
</div>
</div>
</div>
</span>
<div><br>
</div>
<div>Sounds like a testimonial for more project level testing :)</div>
<span id="OLK_SRC_BODY_SECTION">
<div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I see value in projects taking ownership of the System Tests - because if<br>
the project is not 'functionally ready', it is not ready to integrate with<br>
other components of Openstack.<br>
</blockquote>
<div><br>
</div>
<div>"What do you mean by not ready?"</div>
</div>
</div>
</div>
</div>
</div>
</span>
<div><br>
</div>
<div>'Functionally Ready' - The units that make up a project can work together as a system, all API's have been exercised with positive & negative test cases by treating the project as a complete system.</div>
<div>There are no known critical bugs. The point here being identify as many issues as possible, earlier in the game.</div>
<span id="OLK_SRC_BODY_SECTION">
<div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
But for this approach to be successful, projects should have diversity in<br>
the team composition - we need more testers who focus on creating these<br>
tests.<br>
This will keep the teams honest in their quality standards.<br>
<br>
As long as individual projects cannot guarantee functional test coverage,<br>
we will need more tests in Tempest.<br>
But that will shift focus away from Integration Testing, which can be done<br>
ONLY in Tempest.<br>
<br>
Regardless of whatever we end up deciding, it will be good to have these<br>
discussions sooner than later.<br>
This will help at least the new projects to move in the right direction.<br>
<span class="HOEnZb"><font color="#888888"><br>
-Malini<br>
</font></span>
<div class="HOEnZb">
<div class="h5"><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</span>
</body>
</html>