[openstack-dev] [tempest][nova][defcore] Add option to disable some strict response checking for interop testing

Sean Dague sean at dague.net
Wed Jun 22 19:33:21 UTC 2016

On 06/22/2016 03:13 PM, Jay Faulkner wrote:
> On 6/22/16 12:01 PM, Sean Dague wrote:
>> On 06/22/2016 02:37 PM, Chris Hoge wrote:
>>>> On Jun 22, 2016, at 11:24 AM, Sean Dague <sean at dague.net> wrote:
>>>> On 06/22/2016 01:59 PM, Chris Hoge wrote:
>>>>>> On Jun 20, 2016, at 5:10 AM, Sean Dague <sean at dague.net
>>>>>> <mailto:sean at dague.net>> wrote:
>>>>>> On 06/14/2016 07:19 PM, Chris Hoge wrote:
>>>>>>>> On Jun 14, 2016, at 3:59 PM, Edward Leafe <ed at leafe.com
>>>>>>>> <mailto:ed at leafe.com>> wrote:
>>>>>>>> On Jun 14, 2016, at 5:50 PM, Matthew Treinish <mtreinish at kortar.org
>>>>>>>> <mailto:mtreinish at kortar.org>> wrote:
>>>>>>>>> But, if we add another possible state on the defcore side like
>>>>>>>>> conditional pass,
>>>>>>>>> warning, yellow, etc. (the name doesn't matter) which is used to
>>>>>>>>> indicate that
>>>>>>>>> things on product X could only pass when strict validation was
>>>>>>>>> disabled (and
>>>>>>>>> be clear about where and why) then my concerns would be alleviated.
>>>>>>>>> I just do
>>>>>>>>> not want this to end up not being visible to end users trying to
>>>>>>>>> evaluate
>>>>>>>>> interoperability of different clouds using the test results.
>>>>>>>> +1
>>>>>>>> Don't fail them, but don't cover up their incompatibility, either.
>>>>>>>> -- Ed Leafe
>>>>>>> That’s not my proposal. My requirement is that vendors who want to do
>>>>>>> this
>>>>>>> state exactly which APIs are sending back additional data, and that this
>>>>>>> information be published.
>>>>>>> There are different levels of incompatibility. A response with
>>>>>>> additional data
>>>>>>> that can be safely ignored is different from a changed response that
>>>>>>> would
>>>>>>> cause a client to fail.
>>>>>> It's actually not different. It's really not.
>>>>>> This idea that it's safe to add response data is based on an assumption
>>>>>> that software versions only move forward. If you have a single deploy of
>>>>>> software, that's fine.
>>>>>> However as noted, we've got production clouds on Juno <-> Mitaka in the
>>>>>> wild. Which means if we want to support horizontal transfer between
>>>>>> clouds, the user experienced timeline might be start on a Mitaka cloud,
>>>>>> then try to move to Juno. So anything added from Juno -> Mitaka without
>>>>>> signaling has exactly the same client breaking behavior as removing
>>>>>> attributes.
>>>>>> Which is why microversions are needed for attribute adds.
>>>>> I’d like to note that Nova v2.0 is still a supported API, which
>>>>> as far as I understand allows for additional attributes and
>>>>> extensions. That Tempest doesn’t allow for disabling strict
>>>>> checking when using a v2.0 endpoint is a problem.
>>>>> The reporting of v2.0 in the Marketplace (which is what we do
>>>>> right now) is also a signal to a user that there may be vendor
>>>>> additions to the API.
>>>>> DefCore doesn’t disallow the use of a 2.0 endpoint as part
>>>>> of the interoperability standard.
>>>> This is a point of confusion.
>>>> The API definition did not allow that. The implementation of the API
>>>> stack did.
>>> And downstream vendors took advantage of that. We may
>>> not like it, but it’s a reality in the current ecosystem.
>> And we started saying "stop it" 2 years ago. And we've consistently been
>> saying stop it all along. And now it's gone.
>> And yes, for people that did not get ahead of this issue and engage the
>> community, it now hurts. But this has been a quite long process.
> I don't wanna wade fully into this discussion, but a question about this
> here as there seems to be somewhat of a double standard. I know
> upstream, we generally "pay the price" for bad API decisions almost
> indefinitely, because we don't want to break users. 

Upstream pays the upstream API cost. Because that's what's in upstream
and because that's what the API is defined as.

> Is it reasonable to
> expecting a public/vendor cloud, who will typically has even more
> change-averse users, to change that API out from under users without a
> version bump?

What you are asking is that upstream pays the fork cost of every public
cloud forever because they didn't collaborate upstream for their needs.

That can't be the right answer in Open Source, ever. It immediately
makes upstream actually impossible to fix root issues. Like security
issues, which is a huge reason that we put the strict inbound validation
into the API stack, as it means the data we're acting on has a
sanitization layer.

If your feature is not in tree, and part of the upstream testing, it is
at risk of break at any time. Upstream is not just a salt mine you
extract free stuff from, it's a garden you need to participate in and
sow and harvest together.

Which, again, is why we've been having this conversation for *2 years*
in the open. There was a 2 year coasting window to get there, to propose
the required elements into upstream, and have a completely soft landing.
People didn't do that. Why, I don't know.


Sean Dague

More information about the OpenStack-dev mailing list