<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Top posting one note and direct comments inline, I’m proposing<div class="">this as a member of the DefCore working group, but this</div><div class="">proposal itself has not been accepted as the forward course of</div><div class="">action by the working group. These are my own views as the</div><div class="">administrator of the program and not that of the working group</div><div class="">itself, which may independently reject the idea outside of the</div><div class="">response from the upstream devs.</div><div class=""><br class=""></div><div class="">I posted a link to this thread to the DefCore mailing list to make</div><div class="">that working group aware of the outstanding issues.</div><div class=""><div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><div class="">On Jun 14, 2016, at 3:50 PM, Matthew Treinish <<a href="mailto:mtreinish@kortar.org" class="">mtreinish@kortar.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote:</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400:<br class=""><blockquote type="cite" class="">On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote:<br class=""><blockquote type="cite" class="">Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:<br class=""><blockquote type="cite" class="">On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:<br class=""><blockquote type="cite" class="">Last year, in response to Nova micro-versioning and extension updates[1],<br class="">the QA team added strict API schema checking to Tempest to ensure that<br class="">no additional properties were added to Nova API responses[2][3]. In the<br class="">last year, at least three vendors participating the the OpenStack Powered<br class="">Trademark program have been impacted by this change, two of which<br class="">reported this to the DefCore Working Group mailing list earlier this year[4].<br class=""><br class="">The DefCore Working Group determines guidelines for the OpenStack Powered<br class="">program, which includes capabilities with associated functional tests<br class="">from Tempest that must be passed, and designated sections with associated<br class="">upstream code [5][6]. In determining these guidelines, the working group<br class="">attempts to balance the future direction of development with lagging<br class="">indicators of deployments and user adoption.<br class=""><br class="">After a tremendous amount of consideration, I believe that the DefCore<br class="">Working Group needs to implement a temporary waiver for the strict API<br class="">checking requirements that were introduced last year, to give downstream<br class="">deployers more time to catch up with the strict micro-versioning<br class="">requirements determined by the Nova/Compute team and enforced by the<br class="">Tempest/QA team.<br class=""></blockquote><br class="">I'm very much opposed to this being done. If we're actually concerned with<br class="">interoperability and verify that things behave in the same manner between multiple<br class="">clouds then doing this would be a big step backwards. The fundamental disconnect<br class="">here is that the vendors who have implemented out of band extensions or were<br class="">taking advantage of previously available places to inject extra attributes<br class="">believe that doing so means they're interoperable, which is quite far from<br class="">reality. **The API is not a place for vendor differentiation.**<br class=""></blockquote><br class="">This is a temporary measure to address the fact that a large number<br class="">of existing tests changed their behavior, rather than having new<br class="">tests added to enforce this new requirement. The result is deployments<br class="">that previously passed these tests may no longer pass, and in fact<br class="">we have several cases where that's true with deployers who are<br class="">trying to maintain their own standard of backwards-compatibility<br class="">for their end users.<br class=""></blockquote><br class="">That's not what happened though. The API hasn't changed and the tests haven't<br class="">really changed either. We made our enforcement on Nova's APIs a bit stricter to<br class="">ensure nothing unexpected appeared. For the most these tests work on any version<br class="">of OpenStack. (we only test it in the gate on supported stable releases, but I<br class="">don't expect things to have drastically shifted on older releases) It also<br class="">doesn't matter which version of the API you run, v2.0 or v2.1. Literally, the<br class="">only case it ever fails is when you run something extra, not from the community,<br class="">either as an extension (which themselves are going away [1]) or another service<br class="">that wraps nova or imitates nova. I'm personally not comfortable saying those<br class="">extras are ever part of the OpenStack APIs.<br class=""><br class=""><blockquote type="cite" class="">We have basically three options.<br class=""><br class="">1. Tell deployers who are trying to do the right for their immediate<br class=""> users that they can't use the trademark.<br class=""><br class="">2. Flag the related tests or remove them from the DefCore enforcement<br class=""> suite entirely.<br class=""><br class="">3. Be flexible about giving consumers of Tempest time to meet the<br class=""> new requirement by providing a way to disable the checks.<br class=""><br class="">Option 1 goes against our own backwards compatibility policies.<br class=""></blockquote><br class="">I don't think backwards compatibility policies really apply to what what define<br class="">as the set of tests that as a community we are saying a vendor has to pass to<br class="">say they're OpenStack. From my perspective as a community we either take a hard<br class="">stance on this and say to be considered an interoperable cloud (and to get the<br class="">trademark) you have to actually have an interoperable product. We slowly ratchet<br class="">up the requirements every 6 months, there isn't any implied backwards<br class="">compatibility in doing that. You passed in the past but not in the newer stricter<br class="">guidelines.<br class=""><br class="">Also, even if I did think it applied, we're not talking about a change which<br class="">would fall into breaking that. The change was introduced a year and half ago<br class="">during kilo and landed a year ago during liberty:<br class=""><br class=""><a href="https://review.openstack.org/#/c/156130/" class="">https://review.openstack.org/#/c/156130/</a><br class=""><br class="">That's way longer than our normal deprecation period of 3 months and a release<br class="">boundary.<br class=""><br class=""><blockquote type="cite" class=""><br class="">Option 2 gives us no winners and actually reduces the interoperability<br class="">guarantees we already have in place.<br class=""><br class="">Option 3 applies our usual community standard of slowly rolling<br class="">forward while maintaining compatibility as broadly as possible.<br class=""></blockquote><br class="">Except in this case there isn't actually any compatibility being maintained.<br class="">We're saying that we can't make the requirements for interoperability testing<br class="">stricter until all the vendors who were passing in the past are able to pass<br class="">the stricter version.<br class=""><br class=""><blockquote type="cite" class=""><br class="">No one is suggesting that a permanent, or even open-ended, exception<br class="">be granted.<br class=""></blockquote><br class="">Sure, I agree an permanent or open-ended exception would be even worse. But, I<br class="">still think as a community we need to draw a hard line in the sand here. Just<br class="">because this measure is temporary doesn't make it any more palatable.<br class=""><br class="">By doing this, even as a temporary measure, we're saying it's ok to call things<br class="">an OpenStack API when you add random gorp to the responses. Which is something we've<br class="">very clearly said as a community is the exact opposite of the case, which the<br class="">testing reflects. I still contend just because some vendors were running old<br class="">versions of tempest and old versions of openstack where their incompatible API<br class="">changes weren't caught doesn't mean they should be given pass now.<br class=""></blockquote><br class="">Nobody is saying random gorp is OK, and I'm not sure "line in the<br class="">sand" rhetoric is really constructive. The issue is not with the<br class="">nature of the API policies, it's with the implementation of those<br class="">policies and how they were rolled out.<br class=""><br class="">DefCore defines its rules using named tests in Tempest. If these<br class="">new enforcement policies had been applied by adding new tests to<br class="">Tempest, then DefCore could have added them using its processes<br class="">over a period of time and we wouldn't have had any issues. That's<br class="">not what happened. Instead, the behavior of a bunch of *existing*<br class="">tests changed. As a result, deployments that have not changed fail<br class="">tests that they used to pass, without any action being taken on the<br class="">deployer's part. We've moved the goal posts on our users in a way<br class="">that was not easily discoverable, because it couldn't be tracked<br class="">through the (admittedly limited) process we have in place for doing<br class="">that tracking.<br class=""><br class="">So, we want a way to get the test results back to their existing<br class="">status, which will then let us roll adoption forward smoothly instead<br class="">of lurching from "pass" to "fail" to "pass".<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">It doesn't have to be a bright line pass or fail. My primary concern here is</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">that making this change is basically saying we're going to let things "pass"</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">when running out of tree stuff that's adding arbitrary fields to the response. This</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">isn't really interoperable and isn't being honest with what the vendor clouds are</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">actually doing. It would hide the truth from the people who rely on these results</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">to determine interoperability. The proposal as I read it (and maybe it's my</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">misconception) was to mask this and vendor clouds "pass" until they can fix it,</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">which essentially hides the issue. Especially given there are a lot of clouds and</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">products that don't have any issue here.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""></div></blockquote><div><br class=""></div>The opposite is the intention of this proposal. It’s a compromise that admits</div><div>that since the introduction of the OpenStack Powered program, and the release</div><div>of this strict checking on additional properties, vendors that once passed</div><div>now fail, and the incentives to force that change didn’t start being felt until</div><div>they hit their product renewal cycle.</div><div><br class=""></div><div>It’s not trying to mask anything, to the contrary by bringing it up here and</div><div>stating their public test results would indicate which APIs send additional</div><div>properties back, it’s shining a light on the issue and publicly stating that it’s</div><div>not an acceptable long-term solution.<br class=""><div><br class=""></div><blockquote type="cite" class=""><div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">But, if we add another possible state on the defcore side like conditional pass,</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">warning, yellow, etc. (the name doesn't matter) which is used to indicate that</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">things on product X could only pass when strict validation was disabled (and</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">be clear about where and why) then my concerns would be alleviated. I just do</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">not want this to end up not being visible to end users trying to evaluate</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">interoperability of different clouds using the test results.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""></div></blockquote><div><br class=""></div><div>The OpenStack Marketplace is where these comparisons would happen,</div><div>and the APIs with additional response data would be stated.</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class="">We should, separately, address the process issues and the limitations<br class="">this situation has exposed. That may mean changing the way DefCore<br class="">defines its policies, or tracks things, or uses Tempest. For<br class="">example, in the future, we may want tie versions of Tempest to<br class="">versions of the trademark more closely, so that it's possible for<br class="">someone running the Mitaka version of OpenStack to continue to use<br class="">the Mitaka version of Tempest and not have to upgrade Tempest in<br class="">order to retain their trademark (maybe that's how it already works?).<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Tempest master supports all currently supported stable branches. So right now</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">any commit to master is tested against a master cloud, a mitaka cloud, and a</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">liberty cloud in the gate. We tag/push a release whenever we add or drop support</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">for a release, the most recent being dropping kilo. [1][2] That being said the</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">openstack apis **should** be backwards compatible so ideally master tempest would</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">work fine on older clouds. (although this might not be reality) The primary</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">wrinkle here are the tests which would depend on feature flags to indicate it's</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">availability on newer versions. We eventually remove flags after all supported</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">releases have a given feature. But, this can be worked around with test</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">selection. (ie don't even try to run tests that require a feature juno didn’t</span></div></blockquote><blockquote type="cite" class=""><div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">have)</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""></div></blockquote><div><br class=""></div>The current active guidelines cover icehouse through mitaka. The release</div><div>of 2016.08 will change that to cover juno through mitaka (with newton</div><div>as an add-on to 2016.08 when it’s released). There’s overlap between</div><div>the guidelines, so 2016.01 covers juno through mitaka while 2016.08</div><div>will cover kilo through newton. Essentially two years of releases.</div><div><br class=""><br class=""><blockquote type="cite" class=""><div class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">We may also need to consider that test implementation details may<br class="">change, and have a review process within DefCore to help expose<br class="">those changes to make them clearer to deployers.<br class=""><br class="">Fixing the process issue may also mean changing the way we implement<br class="">things in Tempest. In this case, adding a flag helps move ahead<br class="">more smoothly. Perhaps we adopt that as a general policy in the<br class="">future when we make underlying behavioral changes like this to<br class="">existing tests. Perhaps instead we have a policy that we do not<br class="">change the behavior of existing tests in such significant ways, at<br class="">least if they're tagged as being used by DefCore. I don't know --<br class="">those are things we need to discuss.<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Sure I agree, this thread raises larger issues which need to be figured out.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">But, that is probably an independent discussion.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""></div></blockquote><div><br class=""></div>I’m beginning to wonder if we need to make DefCore use release</div><div>branches then back-port bug-fixes and relevant features additions</div><div>as necessary.</div><div><br class=""></div><div><blockquote type="cite" class=""><div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">-Matt Treinish</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">[1]<span class="Apple-converted-space"> </span></span><a href="http://docs.openstack.org/releasenotes/tempest/v12.0.0.html" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">http://docs.openstack.org/releasenotes/tempest/v12.0.0.html</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">[2]<span class="Apple-converted-space"> </span></span><a href="http://git.openstack.org/cgit/openstack/tempest/tag/?h=12.0.0" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">http://git.openstack.org/cgit/openstack/tempest/tag/?h=12.0.0</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class="">Doug<br class=""><br class=""><blockquote type="cite" class=""><br class="">-Matt Treinish<br class=""><br class="">[1] <a href="http://lists.openstack.org/pipermail/openstack-dev/2016-June/097285.html" class="">http://lists.openstack.org/pipermail/openstack-dev/2016-June/097285.html</a><br class=""><blockquote type="cite" class=""><br class="">Doug<br class=""><br class=""><blockquote type="cite" class=""><br class="">As a user of several clouds myself I can say that having random gorp in a<br class="">response makes it much more difficult to use my code against multiple clouds. I<br class="">have to determine which properties being returned are specific to that vendor's<br class="">cloud and if I actually need to depend on them for anything it makes whatever<br class="">code I'm writing incompatible for using against any other cloud. (unless I<br class="">special case that block for each cloud) Sean Dague wrote a good post where a lot<br class="">of this was covered a year ago when microversions was starting to pick up steam:<br class=""><br class=""><a href="https://dague.net/2015/06/05/the-nova-api-in-kilo-and-beyond-2" class="">https://dague.net/2015/06/05/the-nova-api-in-kilo-and-beyond-2</a><br class=""><br class="">I'd recommend giving it a read, he explains the user first perspective more<br class="">clearly there.<br class=""><br class="">I believe Tempest in this case is doing the right thing from an interoperability<br class="">perspective and ensuring that the API is actually the API. Not an API with extra<br class="">bits a vendor decided to add. I don't think a cloud or product that does this<br class="">to the api should be considered an interoperable OpenStack cloud and failing the<br class="">tests is the correct behavior.<br class=""><br class="">-Matt Treinish<br class=""><br class=""><blockquote type="cite" class=""><br class="">My reasoning behind this is that while the change that enabled strict<br class="">checking was discussed publicly in the developer community and took<br class="">some time to be implemented, it still landed quickly and broke several<br class="">existing deployments overnight. As Tempest has moved forward with<br class="">bug and UX fixes (some in part to support the interoperability testing<br class="">efforts of the DefCore Working Group), using an older versions of Tempest<br class="">where this strict checking is not enforced is no longer a viable solution<br class="">for downstream deployers. The TC has passed a resolution to advise<br class="">DefCore to use Tempest as the single source of capability testing[7],<br class="">but this naturally introduces tension between the competing goals of<br class="">maintaining upstream functional testing and also tracking lagging<br class="">indicators.<br class=""><br class="">My proposal for addressing this problem approaches it at two levels:<br class=""><br class="">* For the short term, I will submit a blueprint and patch to tempest that<br class=""> allows configuration of a grey-list of Nova APIs where strict response<br class=""> checking on additional properties will be disabled. So, for example,<br class=""> if the 'create servers' API call returned extra properties on that call,<br class=""> the strict checking on this line[8] would be disabled at runtime.<br class=""> Use of this code path will emit a deprecation warning, and the<br class=""> code will be scheduled for removal in 2017 directly after the release<br class=""> of the 2017.01 guideline. Vendors would be required so submit the<br class=""> grey-list of APIs with additional response data that would be<br class=""> published to their marketplace entry.<br class=""><br class="">* Longer term, vendors will be expected to work with upstream to update<br class=""> the API for returning additional data that is compatible with<br class=""> API micro-versioning as defined by the Nova team, and the waiver would<br class=""> no longer be allowed after the release of the 2017.01 guideline.<br class=""><br class="">For the next half-year, I feel that this approach strengthens interoperability<br class="">by accurately capturing the current state of OpenStack deployments and<br class="">client tools. Before this change, additional properties on responses<br class="">weren't explicitly disallowed, and vendors and deployers took advantage<br class="">of this in production. While this is behavior that the Nova and QA teams<br class="">want to stop, it will take a bit more time to reach downstream. Also, as<br class="">of right now, as far as I know the only client that does strict response<br class="">checking for Nova responses is the Tempest client. Currently, additional<br class="">properties in responses are ignored and do not break existing client<br class="">functionality. There is currently little to no harm done to downstream<br class="">users by temporarily allowing additional data to be returned in responses.<br class=""><br class="">Thanks,<br class=""><br class="">Chris Hoge<br class="">Interop Engineer<br class="">OpenStack Foundation<br class=""><br class="">[1] https://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/api-microversions.html<br class="">[2] http://lists.openstack.org/pipermail/openstack-dev/2015-February/057613.html<br class="">[3] https://review.openstack.org/#/c/156130<br class="">[4] http://lists.openstack.org/pipermail/defcore-committee/2016-January/000986.html<br class="">[5] http://git.openstack.org/cgit/openstack/defcore/tree/2015.07.json<br class="">[6] http://git.openstack.org/cgit/openstack/defcore/tree/2016.01.json<br class="">[7] http://git.openstack.org/cgit/openstack/governance/tree/resolutions/20160504-defcore-test-location.rst<br class="">[8] http://git.openstack.org/cgit/openstack/tempest-lib/tree/tempest_lib/api_schema/response/compute/v2_1/servers.py#n39<br class=""></blockquote></blockquote></blockquote></blockquote><br class="">__________________________________________________________________________<br class="">OpenStack Development Mailing List (not for usage questions)<br class="">Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org" class="">OpenStack-dev-request@lists.openstack.org</a>?subject:unsubscribe<br class=""><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class=""></blockquote><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">__________________________________________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">OpenStack Development Mailing List (not for usage questions)</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Unsubscribe:<span class="Apple-converted-space"> </span></span><a href="mailto:OpenStack-dev-request@lists.openstack.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">OpenStack-dev-request@lists.openstack.org</a><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">?subject:unsubscribe</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></div></blockquote></div><br class=""></div></div></body></html>