<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-GB link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>If the “known_bugs” list isn’t being well received, how about this:<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal style='text-indent:36.0pt'><span style='font-size:11.0pt;
font-family:Consolas;color:#1F497D'># TODO(ewanm): Enable once bug #21212 is
fixed<o:p></o:p></span></p>
<p class=MsoNormal style='text-indent:36.0pt'><span style='font-size:11.0pt;
font-family:Consolas;color:#1F497D'>if False:<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:Consolas;
color:#1F497D'> assert(something)<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:Consolas;
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>And then put a comment on bug #21212 saying “please also enable
the following unit tests when you fix this bug”.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Ewan.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'>
<p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:
"Tahoma","sans-serif"'>From:</span></b><span lang=EN-US style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'> Justin Santa Barbara
[mailto:justin@fathomdb.com] <br>
<b>Sent:</b> 28 February 2011 15:50<br>
<b>To:</b> openstack@lists.launchpad.net<br>
<b>Cc:</b> Clay Gerrard; Dan Prince; Tim Simpson; Ewan Mellor<br>
<b>Subject:</b> Re: [Openstack] How to deal with 'tangential' bugs?<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>I'm really talking about tests that degrade to work around
bugs; I agree that we shouldn't blindly skip whole tests (although when a test
is skipped using @unittest.skip, I think it gives a nice indication that it was
skipped vs passed?)<o:p></o:p></p>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>For example, consider the <span class=apple-style-span><span
style='font-size:10.0pt;font-family:"Arial","sans-serif"'>OpenStack API
authentication issue I gave.</span></span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal><span class=apple-style-span><i><span style='font-size:10.0pt;
font-family:"Arial","sans-serif"'>To get a passing unit test with OpenStack
authentication, my best bet is to set all three values (username, api_key,
api_secret) to the same value. This, however, is a truly terrible test
case. Having "known_bugs" marks my unit test as being
suboptimal; it lets me provide better code in the same place (controlled by the
known_bugs setting); and when the bug fixer comes to fix it they easily get a
failing unit test that they can use for TDD.</span></i></span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal><span class=apple-style-span><span style='font-family:"Arial","sans-serif"'>In
code, you'd have something like this:</span></span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<div>
<p class=MsoNormal><span class=apple-style-span><span style='font-family:"Arial","sans-serif"'>auth_name
= random_string(8)</span></span><span style='font-family:"Arial","sans-serif"'><o:p></o:p></span></p>
</div>
<div>
<div>
<p class=MsoNormal><span class=apple-style-span><span style='font-family:"Arial","sans-serif"'>auth_key
= random_string(8)</span></span><span style='font-family:"Arial","sans-serif"'><o:p></o:p></span></p>
</div>
<div>
<div>
<p class=MsoNormal><span class=apple-style-span><span style='font-family:"Arial","sans-serif"'>auth_secret
= random_string(8)</span></span><span style='font-family:"Arial","sans-serif"'><o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
<div>
<p class=MsoNormal><span class=apple-style-span><span style='font-family:"Arial","sans-serif"'>if
'bug12345' in known_bugs:</span></span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span class=apple-style-span><span style='font-family:"Arial","sans-serif"'> #
Set all 3 values the same, so that we can still auth even though secret &
key aren't what we'd intuitively expect</span></span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span class=apple-style-span><span style='font-family:"Arial","sans-serif"'> auth_secret
= auth_name</span></span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span class=apple-style-span><span style='font-family:"Arial","sans-serif"'> auth_key
= auth_name</span></span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal><span class=apple-style-span><span style='font-family:"Arial","sans-serif"'>create_user(auth_name,
auth_key, auth_secret)</span></span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-family:"Arial","sans-serif"'><br>
<span class=apple-style-span>So I have a weak test case (because it's the best
I can do until bug12345 is fixed). When someone removes 'bug12345' from
the known_bug list, the same test case becomes strong. If someone is
ready to fix the OpenStack auth issue, they need only remove bug12345 from the
known_bugs list and the tests cases will then fail, so they have ready-made
TDD.</span></span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal><span class=apple-style-span><span style='font-family:"Arial","sans-serif"'>Another
example: My unit test might skip over an assertion, even though the spec
says it should be the case, because there's a filed bug about it. I have
the choice between not testing it at all (not writing a good unit test), or
writing a good unit test with a 'guard' that means that it temporarily degrades
to a weaker unit test.</span></span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal><span class=apple-style-span><span style='font-family:"Arial","sans-serif"'>Another
example: I encountered an issue with fake_carrot. Vish fixed it very
quickly, but in the meantime my best bet was to use "real" RabbitMQ.
Switching between fake_carrot and "real" RabbitMQ could be
driven by a known_bugs flag. This is probably more controversial, but because
fake_carrot is supposed to behave the same as carrot, it did mean I could
satisfy myself & anyone else that the bug was (probably) with fake_carrot,
because it wasn't behaving like the system it was supposed to be faking.</span></span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-family:"Arial","sans-serif"'><br>
<span class=apple-style-span>I think in reality there are two alternatives to
putting this into a really obvious file like known_bugs.py:</span></span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span class=apple-style-span><span style='font-family:"Arial","sans-serif"'>1)
Hide a similar flag in the file with the unit test, or comment out the 'good
test code'</span></span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span class=apple-style-span><span style='font-family:"Arial","sans-serif"'>or</span></span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span class=apple-style-span><span style='font-family:"Arial","sans-serif"'>2)
Simply not write a thorough test case.</span></span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal><span class=apple-style-span><span style='font-family:"Arial","sans-serif"'>I
think both of these alternatives are worse. I don't love known_bugs.py
either, but I think it's the best we've got. If anyone has a better idea,
I'd love to hear it!</span></span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal><span class=apple-style-span><span style='font-family:"Arial","sans-serif"'>Ewan:
I think the idea of skipUnless(datetime.now()...) is brilliant. I'm just
thinking of the havoc that could be caused by sneaking an obfuscated version of
that into the unit test suite, particularly one that only failed if the
username was 'hudson' :)</span></span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>Justin<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal style='margin-bottom:12.0pt'><o:p> </o:p></p>
<div>
<p class=MsoNormal>On Mon, Feb 28, 2011 at 1:57 PM, Tim Simpson <<a
href="mailto:tim.simpson@rackspace.com">tim.simpson@rackspace.com</a>>
wrote:<o:p></o:p></p>
<p class=MsoNormal>The Skip plugin for nose offers similar functionality which
can be used in Python 2.6:<br>
<a
href="http://somethingaboutorange.com/mrl/projects/nose/0.11.1/plugins/skip.html"
target="_blank">http://somethingaboutorange.com/mrl/projects/nose/0.11.1/plugins/skip.html</a><br>
<br>
Using this you can write decorators that raise SkipTest if a certain criteria
isn't met.<br>
<br>
________________________________________<br>
From: openstack-bounces+tim.simpson=<a href="http://rackspace.com"
target="_blank">rackspace.com</a>@<a href="http://lists.launchpad.net"
target="_blank">lists.launchpad.net</a> [openstack-bounces+tim.simpson=<a
href="http://rackspace.com" target="_blank">rackspace.com</a>@<a
href="http://lists.launchpad.net" target="_blank">lists.launchpad.net</a>] on
behalf of Clay Gerrard [<a href="mailto:clay.gerrard@rackspace.com">clay.gerrard@rackspace.com</a>]<br>
Sent: Monday, February 28, 2011 3:09 PM<br>
To: Dan Prince; Justin Santa Barbara<br>
Cc: <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
Subject: Re: [Openstack] How to deal with 'tangential' bugs?<o:p></o:p></p>
<div>
<div>
<p class=MsoNormal><br>
Unittest2 lets you define a test case that is expected to fail:<br>
<a href="http://docs.python.org/library/unittest.html#unittest.expectedFailure"
target="_blank">http://docs.python.org/library/unittest.html#unittest.expectedFailure</a><br>
<br>
new in 2.7, but it could be possible to backport - or do something similar...<br>
<br>
May have issues with nose:<br>
<a href="http://code.google.com/p/python-nose/issues/detail?id=325"
target="_blank">http://code.google.com/p/python-nose/issues/detail?id=325</a><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal>_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><o:p></o:p></p>
</div>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>