Open Stack

Yeah, email is making this tricky.

I suspect the problem is coming in with our definition of "unit tests". I don't think a unit test should be calling out of the method being tested at all. So anything beyond stubbing out the methods within the method being tested seems like noise to me. What you're describing sounds more like integration tests. I thought the motive of your thread was to create fast/small/readable/non-brittle/maintainable tests. Integration tests, while important, make this goal difficult. So, if we're both talking about real unit tests, I don't seen the benefit of the fake.

As for my example of 123 vs "abc", that was a bad example. Let me rephrase ... in one test I may want to have an environment that has no pre-existing instances in the db. In another test I may want to have an environment with a hundred instances. 

I'd like to understand how configuring the fake for both of these scenarios will be any easier than just having a stub. It seems like an unnecessary abstraction.

-S



________________________________________
From: Soren Hansen [soren at linux2go.dk]
Sent: Tuesday, November 22, 2011 4:37 PM
To: Sandy Walsh
Cc: Jay Pipes; openstack at lists.launchpad.net
Subject: Re: [Openstack] [nova-testing] Efforts for Essex

2011/11/22 Sandy Walsh <sandy.walsh at rackspace.com>:
> I suppose there is a matter of preference here. I prefer to look in the setup() and teardown() methods of my test suite to find out how everything hangs together. Otherwise I have to check nova.TestCase when things break. The closer my test can stay to my expectations from unittest.TestCase the happier I am.

Sorry, I don't follow. The unit tests would use the fake db driver by
default. No per-test-specific setup necessary. Creating the instance
in the fake DB would happen explicitly in the individual tests (by way
of either calling db.instance_create directly, or by way of some
utility function).

> I can't comment on your fake db implementation, but my fear is this scenario:
>
> Test1 assumes db.create_foo() will return 123 and Test2 assumes it will return "abc". How do they both comfortably co-exist? And whatever the mechanism, why is it better than just stubs.Set("db.create_foo", _my_create_foo)?

I'm confused. That's *exactly* what I want to avoid. By everything
sharing the same fake db driver, you can never have one mock that
returns one style of response, and another mock that returns another
style of response.

> It's local and it makes sense in the context of that file.

But it has to make sense globally. If something you're testing only
ever sees an Instance object with a couple of "hardcoded" attributes
on it, because that's what its mock gives it, you'll never know if
it'll fail if it gets a more complete, real Instance object.

--
Soren Hansen        | http://linux2go.dk/
Ubuntu Developer    | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/