<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.hoenzb
{mso-style-name:hoenzb;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
/* List Definitions */
@list l0
{mso-list-id:1542395968;
mso-list-type:hybrid;
mso-list-template-ids:-373683834 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
-->
</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="Section1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D">(Sorry for the top post. It was this or bottom post because of company choice of email systems)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D">Integration tests, corner cases, negative tests. Lots of names that don't have clear definitions in this discussion. But, it seems like the collection of
"negative tests" include both functional and integrated tests. Maybe we should analyze the tests to see where they belong. So, perhaps we can ask an important question here:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:Symbol;color:#1F497D"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D">Does the test provoke a change of behavior outside the designated project, depending on the output(s) from the test action? In other words, if
the test is attempting to generate a 4xx response, does it cause another project or client to act differently than if it returns a 2xx or different 4xx response? An example of this would be the project the response is returned to attempts a retry.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D">This should qualify as a tempest test because it drives actions from multiple projects. Yes, it is a "negative" api test, but run in an integrated system,
it is also a normal integration test that demonstrates the integrated system behaves appropriately on expected error conditions from other projects. Or it throws an exception or an error message and we know we have a problem with either the api or the misbehaving
project.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D">Andrea aptly pointed out that these tests may very well be interoperability tests that should be in a gate somewhere. After all, what are APIs, but defined
interfaces between two components. Sometimes it's between human and software, but most often in OpenStack, it's between two pieces of software written by different groups of developers. Hence, the API *<b>is</b>* the integration point and ideally, all request/response
options should be validated, whether happy path or not so happy.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D">And, oh by the way, "negative" responses are crucial interoperability points for DefCore.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D">--Rocky<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:10.0pt;
font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;
font-family:"Tahoma","sans-serif""> Valeriy Ponomaryov March 16, 2016 11:25 PM</span></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">Main reason to split negative tests out, that I see, is attempt to remove "corner, not integrational" tests out of Tempest tree. If so, then "negative" tests is not proper choice, I think.<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">Tempest has lots of positive "corner" test cases. So, if move something, then we need to move exactly "corner" tests and leave only "integrational" tests of any kind - either it is negative or positive.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">It is said, that doing so, all single-project related tests will be in its own repo as a plugin and only integrational tests will be in Tempest.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Valeriy<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">On Thu, Mar 17, 2016 at 7:31 AM, Qiming Teng <<a href="mailto:tengqim@linux.vnet.ibm.com" target="_blank">tengqim@linux.vnet.ibm.com</a>> wrote:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">><br>
> I'd love to see this idea explored further. What happens if Tempest<br>
> ends up without tests, as a library for shared code as well as a<br>
> centralized place to run tests from via plugins?<br>
><br>
<br>
Also curious about this. It seems weird to separate the 'positive' and<br>
the 'negative' ones, assuming those patches are mostly contributed by<br>
the same group of developers.<br>
<span style="color:#888888"><br>
<span class="hoenzb">Qiming</span></span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-left:.5in"><br>
<br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in">-- <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Kind Regards<br>
Valeriy Ponomaryov<br>
<a href="http://www.mirantis.com" target="_blank">www.mirantis.com</a><br>
<a href="mailto:vponomaryov@mirantis.com" target="_blank">vponomaryov@mirantis.com</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>