Open Stack

If the "known_bugs" list isn't being well received, how about this:

# TODO(ewanm): Enable once bug #21212 is fixed
if False:
            assert(something)

And then put a comment on bug #21212 saying "please also enable the following unit tests when you fix this bug".

Ewan.

From: Justin Santa Barbara [mailto:justin at fathomdb.com]
Sent: 28 February 2011 15:50
To: openstack at lists.launchpad.net
Cc: Clay Gerrard; Dan Prince; Tim Simpson; Ewan Mellor
Subject: Re: [Openstack] How to deal with 'tangential' bugs?

I'm really talking about tests that degrade to work around bugs; I agree that we shouldn't blindly skip whole tests (although when a test is skipped using @unittest.skip, I think it gives a nice indication that it was skipped vs passed?)

For example, consider the OpenStack API authentication issue I gave.

To get a passing unit test with OpenStack authentication, my best bet is to set all three values (username, api_key, api_secret) to the same value.  This, however, is a truly terrible test case.  Having "known_bugs" marks my unit test as being suboptimal; it lets me provide better code in the same place (controlled by the known_bugs setting); and when the bug fixer comes to fix it they easily get a failing unit test that they can use for TDD.

In code, you'd have something like this:

auth_name = random_string(8)
auth_key = random_string(8)
auth_secret = random_string(8)

if 'bug12345' in known_bugs:
  # Set all 3 values the same, so that we can still auth even though secret & key aren't what we'd intuitively expect
  auth_secret = auth_name
  auth_key = auth_name

create_user(auth_name, auth_key, auth_secret)


So I have a weak test case (because it's the best I can do until bug12345 is fixed).  When someone removes 'bug12345' from the known_bug list, the same test case becomes strong.  If someone is ready to fix the OpenStack auth issue, they need only remove bug12345 from the known_bugs list and the tests cases will then fail, so they have ready-made TDD.

Another example:  My unit test might skip over an assertion, even though the spec says it should be the case, because there's a filed bug about it.  I have the choice between not testing it at all (not writing a good unit test), or writing a good unit test with a 'guard' that means that it temporarily degrades to a weaker unit test.

Another example: I encountered an issue with fake_carrot.  Vish fixed it very quickly, but in the meantime my best bet was to use "real" RabbitMQ.  Switching between fake_carrot and "real" RabbitMQ could be driven by a known_bugs flag. This is probably more controversial, but because fake_carrot is supposed to behave the same as carrot, it did mean I could satisfy myself & anyone else that the bug was (probably) with fake_carrot, because it wasn't behaving like the system it was supposed to be faking.


I think in reality there are two alternatives to putting this into a really obvious file like known_bugs.py:
1) Hide a similar flag in the file with the unit test, or comment out the 'good test code'
or
2) Simply not write a thorough test case.

I think both of these alternatives are worse.  I don't love known_bugs.py either, but I think it's the best we've got.  If anyone has a better idea, I'd love to hear it!

Ewan: I think the idea of skipUnless(datetime.now()...) is brilliant.  I'm just thinking of the havoc that could be caused by sneaking an obfuscated version of that into the unit test suite, particularly one that only failed if the username was 'hudson' :)

Justin

On Mon, Feb 28, 2011 at 1:57 PM, Tim Simpson <tim.simpson at rackspace.com<mailto:tim.simpson at rackspace.com>> wrote:
The Skip plugin for nose offers similar functionality which can be used in Python 2.6:
http://somethingaboutorange.com/mrl/projects/nose/0.11.1/plugins/skip.html

Using this you can write decorators that raise SkipTest if a certain criteria isn't met.

________________________________________
From: openstack-bounces+tim.simpson=rackspace.com<http://rackspace.com>@lists.launchpad.net<http://lists.launchpad.net> [openstack-bounces+tim.simpson=rackspace.com<http://rackspace.com>@lists.launchpad.net<http://lists.launchpad.net>] on behalf of Clay Gerrard [clay.gerrard at rackspace.com<mailto:clay.gerrard at rackspace.com>]
Sent: Monday, February 28, 2011 3:09 PM
To: Dan Prince; Justin Santa Barbara
Cc: openstack at lists.launchpad.net<mailto:openstack at lists.launchpad.net>
Subject: Re: [Openstack] How to deal with 'tangential' bugs?

Unittest2 lets you define a test case that is expected to fail:
http://docs.python.org/library/unittest.html#unittest.expectedFailure

new in 2.7, but it could be possible to backport - or do something similar...

May have issues with nose:
http://code.google.com/p/python-nose/issues/detail?id=325
_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to     : openstack at lists.launchpad.net<mailto:openstack at lists.launchpad.net>
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20110301/7cc26d38/attachment.html>