Open Stack

This is really interesting discussion but was thrown off by the different use of ‘functional testing.’ I decided to reconcile it with my understanding and ended up with this two-pager. Sharing it in case it helps:

https://docs.google.com/document/d/1ILxfoJlov9lBfuuZtvwW_7bmlGaYYw0UsE1R9lo6XwQ/edit


Regards,

Mark Maglana
QA/Release Engineer
Nexus IS, Inc.
Single Number Reach: 424-225-1309






On Mar 25, 2014, at 4:55 AM, Malini Kamalambal <malini.kamalambal at RACKSPACE.COM> wrote:

> 
> 
> 
> We are talking about different levels of testing,
> 
> 1. Unit tests - which everybody agrees should be in the individual project
> itself
> 2. System Tests - 'System' referring to (& limited to), all the components
> that make up the project. These are also the functional tests for the
> project.
> 3. Integration Tests - This is to verify that the OS components interact
> well and don't break other components -Keystone being the most obvious
> example. This is where I see getting the maximum mileage out of Tempest.
> 
> "Its not easy to detect what the integration points with other projects are, any project can use any stable API from any other project. Because of this all OpenStack APIs should fit into this category. "
> 
> Any project can use any stable API –but that does not make all API tests , Integration Tests.
> A test becomes Integration test when it has two or more projects interacting in the test.
> 
> Individual projects should be held accountable to make sure that their API's work – no matter who consumes them.
> We should be able to treat the project as a complete system, make API calls and validate that the response matches the API definitions.
> Identifying issues earlier in the pipeline reduces the Total Cost of Quality.
> 
> I agree that Integration Testing is hard. It is complicated because it requires knowledge of how systems interact with each other – and knowledge comes from a lot of time spent on analysis.
> It requires people with project expertise to talk to each other & identify possible test cases.
> openstack-qa is the ideal forum to do this.
> Holding projects responsible for their functionality will help the QA team focus on complicated integration tests.
> 
> "Having a second group writing tests to Nova's public APIs has been really helpful in keeping us honest as well."
> 
> Sounds like a testimonial for more project level testing :)
> 
> 
> I see value in projects taking ownership of the System Tests - because if
> the project is not 'functionally ready', it is not ready to integrate with
> other components of Openstack.
> 
> "What do you mean by not ready?"
> 
> 'Functionally Ready' - The units that make up a project can work together as a system,  all API's have been exercised with positive & negative test cases by treating the project as a complete system.
> There are no known critical bugs. The point here being identify as many issues as possible, earlier in the game.
>  
> But for this approach to be successful, projects should have diversity in
> the team composition - we need more testers who focus on creating these
> tests.
> This will keep the teams honest in their quality standards.
> 
> As long as individual projects cannot guarantee functional test coverage,
> we will need more tests in Tempest.
> But that will shift focus away from Integration Testing, which can be done
> ONLY in Tempest.
> 
> Regardless of whatever we end up deciding, it will be good to have these
> discussions sooner than later.
> This will help at least the new projects to move in the right direction.
> 
> -Malini
> 
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev