<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>Hello Georg,</div>
<div><br>
</div>
<div>We had tried to make sure that tests test only one capability and not several. We had a spec that describes what we are looking for in test:</div>
<div><br>
</div>
<div><a href="https://github.com/openstack/interop/blob/master/working_materials/interop_test_spec.rst">https://github.com/openstack/interop/blob/master/working_materials/interop_test_spec.rst</a></div>
<div><br>
</div>
<div>Test guidelines:</div>
<div><dt style="box-sizing: border-box; padding: 0px; margin-top: 16px; font-size: 16px; font-style: italic; font-weight: 600; color: rgb(36, 41, 46); font-family: -apple-system, system-ui, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; orphans: 2; widows: 2; background-color: rgb(255, 255, 255);">
Guidelines</dt><dd style="box-sizing: border-box; margin-left: 0px; padding: 0px 16px; margin-bottom: 16px; color: rgb(36, 41, 46); font-family: -apple-system, system-ui, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; orphans: 2; widows: 2; background-color: rgb(255, 255, 255);">
<ol style="box-sizing: border-box; padding-left: 2em; margin-top: 0px; margin-bottom: 16px;">
<li style="box-sizing: border-box;">Interop API functional tests should target and validate one capability. Some capabilities may require discovery calls and these should be included also in the test. Each capability should have at least one interop API functional
test.</li><li style="box-sizing: border-box; margin-top: 0.25em;">Interop API functional tests should not use non-required capabilities.</li><li style="box-sizing: border-box; margin-top: 0.25em;">Interop scenario tests may use several capabilities to test end to end functionality and none of the capabilities used may be non-required capabilities.</li><li style="box-sizing: border-box; margin-top: 0.25em;">Tests must be reliable; race conditions must be avoided; tests should not be time dependent. If any of these problems occur the test will be flagged and either fixed or removed from the guidelines.</li><li style="box-sizing: border-box; margin-top: 0.25em;">Tests must be able to run consistently across all the releases covered by the Guidelines which are considered valid by the OpenStack Foundation for the OpenStack Powered program.</li><li style="box-sizing: border-box; margin-top: 0.25em;">Tests should not require more credentials than required to test the capability itself.</li><li style="box-sizing: border-box; margin-top: 0.25em;">Tests must not be harmful to the end user's environment, that is no changing credentials nor destroying user's data nor altering any other user resource other than those specifically created as part of
the test.</li><li style="box-sizing: border-box; margin-top: 0.25em;">Test cases must setup any data they need to run and clean up after themselves. It is not acceptable to corrupt the user's environment or leave any test data or test configuration behind.</li></ol>
</dd></div>
<div>Let us know if you have any questions!</div>
<div><br>
</div>
<div>-Egle</div>
<div><br>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Georg Kunz <<a href="mailto:georg.kunz@ericsson.com">georg.kunz@ericsson.com</a>><br>
<span style="font-weight:bold">Date: </span>Monday, June 26, 2017 at 2:54 AM<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:interop-wg@lists.openstack.org">interop-wg@lists.openstack.org</a>" <<a href="mailto:interop-wg@lists.openstack.org">interop-wg@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>[Interop-wg] Tempest scenario tests<br>
</div>
<div><br>
</div>
<div xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi Egle, Rocky,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you again for joining last week’s OPNFV Dovetail call. Your input was very valuable and I believe we agreed across the board that a closer collaboration between OPNFV Dovatail and the OpenStack Interop WG is fundamental for both communities.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have one follow-up question: the test suites of the three OpenStack certification programs currently include only tempest API tests (located in tempest.api.X) and none of the tempest scenarios (located in tempest.scenario.X). Is there
a specific reason why those more complex end-to-end tests are currently not included?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks<o:p></o:p></p>
<p class="MsoNormal">Georg<o:p></o:p></p>
</div>
</div>
</div>
</span>
</body>
</html>