Open Stack

On 02/10/2013 12:37 PM, John Griffith wrote:
<snip
> Testing the interaction with other Cinder modules should be done through
> functional tests.  This is where devstack exercises would come in to
> play.  Currently I don't think we put enough emphasis here, when
> somebody submits a bug fix for something we often say "You need a unit
> test to catch this sort of thing in the future", maybe that's true in
> some cases but I'd argue that we should be saying "You need a test in
> devstack or tempest to catch this in the future".  One of the main
> reasons unit tests aren't very helpful here IMO is that in most cases a
> bug results from a refactor or enhancement in the code.  Most of the
> time when folks make these enhancements they also modify the unit test
> to behave the way they "expect" it to and think it should.  The result
> is they just create a unit test that verifies the incorrect behaviour.

On that last statement, I agree. And that can only be caught by 
reviewers. I found some spectacular instances of this in the API audit 
in Nova. :)

The answer is *never* more devstack exercises for this. Those are very 
much basic sanity checks, to ensure the services are started. Please, 
don't submit lots of things there.

The answer might be more Tempest tests. I would highly encourage more 
teams to get engaged there. There are relatively few of us working on 
that code base, more hands would be appreciated. Especially more people 
that are deep into one of the code bases. We've got good Nova and Glance 
domain knowledge cross over with core folks on Tempest, but very much 
lacking on the other projects, so reviewing the efficacy of some of the 
tests is hard. Also realize that dropping a tempest test can't be 
enforced as a simultaneous commit.

I will also have to say that some of the things that we've got in Nova 
which stretch the formal definition of unit tests are extremely useful. 
Like the samples testing, which has been enforcing that our API doesn't 
change without anyone knowing. The actual database migrations are 
definitely outside this scope as well. These things exposed real bugs 
that would have made grizzly deploys extremely difficult.

All that said, the time to catch most of this is at review. And then 
refactor later to deal with the debt that's in there. It's just a lot of 
work and tedious tasks. Just requires people to dive in and help.

	-Sean

-- 
Sean Dague
IBM Linux Technology Center
email: sdague at linux.vnet.ibm.com
alt-email: sldague at us.ibm.com