<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(easier to insert my questions at top of discussion as they are more general)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">How would test deprecations work in a branchless Tempest? Right now, there is the discussion on removing the XML tests from Tempest, yet they are still valid
for Havana and Icehouse. If they get “removed”, will they still be accessible and runnable for Havana version tests? I can see running from a tagged version for Havana, but if you are *<b>not</b>* running from the tag, then the files would be “gone”. So,
I’m wondering how this would work for Refstack, testing backported bugfixes, etc.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Another related question arises from the discussion of Nova API versions. Tempest tests are being enhanced to do validation, and the newer API versions (2.1,
3.n, etc. when the approach is decided) will do validation, etc. How will these “backward incompatible” tests be handled if the test that works for Havana gets modified to work for Juno and starts failing Havana code base?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">With the discussion of project functional tests that could be maintained in one place, but run in two (maintenance location undecided, run locale local and
Tempest/Integrated), how would this “cross project” effort be affected by a branchless Tempest?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Maybe we need some use cases to ferret out the corner cases of a branchless Tempest implementation? I think we need to get more into some of the details to
understand what would be needed to be added/modified/ removed to make this design proposal work.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">--Rocky
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> David Kranz [mailto:dkranz@redhat.com]
<br>
<b>Sent:</b> Friday, April 04, 2014 6:10 AM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [RFC] Tempest without branches<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 04/04/2014 07:37 AM, Sean Dague wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>An interesting conversation has cropped up over the last few days in -qa<o:p></o:p></pre>
<pre>and -infra which I want to bring to the wider OpenStack community. When<o:p></o:p></pre>
<pre>discussing the use of Tempest as part of the Defcore validation we came<o:p></o:p></pre>
<pre>to an interesting question:<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Why does Tempest have stable/* branches? Does it need them?<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Historically the Tempest project has created a stable/foo tag the week<o:p></o:p></pre>
<pre>of release to lock the version of Tempest that will be tested against<o:p></o:p></pre>
<pre>stable branches. The reason we did that is until this cycle we had<o:p></o:p></pre>
<pre>really limited nobs in tempest to control which features were tested.<o:p></o:p></pre>
<pre>stable/havana means - test everything we know how to test in havana. So<o:p></o:p></pre>
<pre>when, for instance, a new API extension landed upstream in icehouse,<o:p></o:p></pre>
<pre>we'd just add the tests to Tempest. It wouldn't impact stable/havana,<o:p></o:p></pre>
<pre>because we wouldn't backport changes.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>But is this really required?<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>For instance, we don't branch openstack clients. They are supposed to<o:p></o:p></pre>
<pre>work against multiple server versions. Tempest, at some level, is<o:p></o:p></pre>
<pre>another client. So there is some sense there.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Tempest now also have flags on features, and tests are skippable if<o:p></o:p></pre>
<pre>services, or even extensions aren't enabled (all explicitly setable in<o:p></o:p></pre>
<pre>the tempest.conf). This is a much better control mechanism than the<o:p></o:p></pre>
<pre>course grained selection of stable/foo.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>If we decided not to set a stable/icehouse branch in 2 weeks, the gate<o:p></o:p></pre>
<pre>would change as follows:<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Project masters: no change<o:p></o:p></pre>
<pre>Project stable/icehouse: would be gated against Tempest master<o:p></o:p></pre>
<pre>Tempest master: would double the gate jobs, gate on project master and<o:p></o:p></pre>
<pre>project stable/icehouse on every commit.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>(That last one needs infra changes to work right, those are all in<o:p></o:p></pre>
<pre>flight right now to assess doability.)<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Some interesting effects this would have:<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre> * Tempest test enhancements would immediately apply on stable/icehouse *<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>... giving us more confidence. A large amount of tests added to master<o:p></o:p></pre>
<pre>in every release are enhanced checking for existing function.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre> * Tempest test changes would need server changes in master and<o:p></o:p></pre>
<pre>stable/icehouse *<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>In trying tempest master against stable/havana we found a number of<o:p></o:p></pre>
<pre>behavior changes in projects that there had been a 2 step change in the<o:p></o:p></pre>
<pre>Tempest tests to support. But this actually means that stable/havana and<o:p></o:p></pre>
<pre>stable/icehouse for the same API version are different. Going forward<o:p></o:p></pre>
<pre>this would require master + stable changes on the projects + Tempest<o:p></o:p></pre>
<pre>changes. Which would provide much more friction in changing these sorts<o:p></o:p></pre>
<pre>of things by accident.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre> * Much more stable testing *<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>If every Tempest change is gating on stable/icehouse, the week long<o:p></o:p></pre>
<pre>stable/havana can't pass tests won't happen. There will be much more<o:p></o:p></pre>
<pre>urgency to keep stable branches functioning.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>If we got rid of branches in Tempest the path would be:<o:p></o:p></pre>
<pre> * infrastructure to support this in infra - in process, probably<o:p></o:p></pre>
<pre>landing today<o:p></o:p></pre>
<pre> * don't set stable/icehouse - decision needed by Apr 17th<o:p></o:p></pre>
<pre> * changes to d-g/devstack to be extra explicit about what features<o:p></o:p></pre>
<pre>stable/icehouse should support in tempest.conf<o:p></o:p></pre>
<pre> * see if we can make master work with stable/havana to remove the<o:p></o:p></pre>
<pre>stable/havana Tempest branch (if this is doable in a month, great, if<o:p></o:p></pre>
<pre>not just wait for havana to age out).<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>I think we would still want to declare Tempest versions from time to<o:p></o:p></pre>
<pre>time. I'd honestly suggest a quarterly timebox. The events that are<o:p></o:p></pre>
<pre>actually important to Tempest are less the release itself, but the eol<o:p></o:p></pre>
<pre>of branches, as that would mean features which removed completely from<o:p></o:p></pre>
<pre>any supported tree could be removed.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>My current leaning is that this approach would be a good thing, and<o:p></o:p></pre>
<pre>provide a better experience for both the community and the defcore<o:p></o:p></pre>
<pre>process. However it's a big enough change that we're still collecting<o:p></o:p></pre>
<pre>data, and it would be interesting to hear other thoughts from the<o:p></o:p></pre>
<pre>community at large on this approach.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre> -Sean<o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><br>
With regard to havana, the problems with DefCore using stable/havana are the same as many of us have felt with testing real deployments of havana.<br>
Master (now icehouse) has many more tests, is more robust to individual test failures, and is more configurable. But the work to backport improvements is difficult or impossible due to many refactorings on master, api changes, and the tempest backport policy
that we don't want to spend our review time looking backwards. The reality is that almost nothing has been backported to stable/havana tempest, and we don't want to start doing that now. As defcore/refstack becomes a reality, more bugs and desired features
in tempest will be found and it would be good if issues could be addressed on master.
<br>
<br>
The approach advocated here would prevent this from happening again with icehouse and going forward. That still leaves havana as an important case for many folks. I did an initial run of master tempest against havana using nova-network but no heat/ceilo/swift).
148 out of 2009 tests failed. The failures seemed to be in these categories:<br>
<br>
1. An api change occurred such as change in response code, added fields in return dicts, and others I have not yet categorized.<br>
2. A new feature was added in icehouse and the tempest test was not behind a config option to see if it was enabled.<br>
3. A bug was fixed in icehouse which is still there in havana and the tempest test was changed or unskipped.<br>
4. A new test was added that never ran against havana, but could have, and it fails.<br>
<br>
Even if we adopt this approach going forward, case (3,4) will continue to exist in future iterations. That implies the need to have a config option and associated test tag to say which release is targeted so that the test will not run against an older release.
This will create some ugliness around such cases but seems to me less ugly than what we have now, which is one giant switch (new branch) that controls everything.<br>
<br>
The easiest way to get master running against havana would be to add such a config and then simply skip all of the failing tests when running against havana. There could also be conditionals in tempest saying what behaviour is expected but I'm not sure we want
to go there. <br>
<br>
This approach will add clarity to the notion that OpenStack releases are decoupled from API versions. As Sean said, doing a "tempest-two-step" to implement an api change would now need to also be done on a stable branch.<br>
<br>
-David<br>
<br>
<o:p></o:p></p>
<pre><o:p> </o:p></pre>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>OpenStack-dev mailing list<o:p></o:p></pre>
<pre><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><o:p></o:p></pre>
<pre><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></pre>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>