Open Stack

On 10/30/2013 10:37 PM, Robert Collins wrote:
> This is a bit of a social norms thread....
>
> I've been consistently asking for tests in reviews for a while now,
> and I get the occasional push-back. I think this falls into a few
> broad camps:
>
> A - there is no test suite at all, adding one in unreasonable
> B - this thing cannot be tested in this context (e.g. functional tests
> are defined in a different tree)
> C - this particular thing is very hard to test
> D - testing this won't offer benefit
> E - other things like this in the project don't have tests
> F - submitter doesn't know how to write tests
> G - submitter doesn't have time to write tests
>
> Now, of these, I think it's fine not add tests in cases A, B, C in
> combination with D, and D.
>
> I don't think E, F or G are sufficient reasons to merge something
> without tests, when reviewers are asking for them. G in the special
> case that the project really wants the patch landed - but then I'd
> expect reviewers to not ask for tests or to volunteer that they might
> be optional.
>
> Now, if I'm wrong, and folk have different norms about when to accept
> 'reason X not to write tests' as a response from the submitter -
> please let me know!

First - providing tests with code is a basic design tenant of OpenStack 
- https://wiki.openstack.org/wiki/BasicDesignTenets (I realize we don't 
highlight that as much as we used to, but it really is baked in culture 
to people that have been here for a while).

This is pretty close to my thinking. I like to challenge people on C. 
There was a pretty good case of it late in grizzly or havana where I was 
reviewing some nova virt code and the sumbitter claimed it was too hard 
to test. A week later danpb wrote a 100 line patch that provided the 
entire test framework for the case. So "C" was a pretty BS answer.

Adding tests and test frameworks are a way of paying it forward in the 
project. Much like doing reviews, adding tests is a way to save other 
people on the project time. Good tests pay for themselves very quickly 
in time lost on the project.

So I generally find "I don't want to add tests" the same kind of 
selfishness as "why should I have to review code?".

A good example of the kind of savings we get, over the summer when were 
were regularlly breaking on requirements mismatch (and on the update 
script modifying things incorrectly), I went in and added a testing 
framework to requirements repo, so we knew all the parts were doing 
expected things. Without that we'd still be fighting these things on a 
weekly basis, now it's a solved problem, and we can go fight the next fire.

This was actually as C class problem (which is why no one else had done 
it yet), and required building new devstack job to complete. However the 
benefits were really high on it.

	-Sean

-- 
Sean Dague
http://dague.net