Open Stack

Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400:
> On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
> > Last year, in response to Nova micro-versioning and extension updates[1],
> > the QA team added strict API schema checking to Tempest to ensure that
> > no additional properties were added to Nova API responses[2][3]. In the
> > last year, at least three vendors participating the the OpenStack Powered
> > Trademark program have been impacted by this change, two of which
> > reported this to the DefCore Working Group mailing list earlier this year[4].
> > 
> > The DefCore Working Group determines guidelines for the OpenStack Powered
> > program, which includes capabilities with associated functional tests
> > from Tempest that must be passed, and designated sections with associated
> > upstream code [5][6]. In determining these guidelines, the working group
> > attempts to balance the future direction of development with lagging
> > indicators of deployments and user adoption.
> > 
> > After a tremendous amount of consideration, I believe that the DefCore
> > Working Group needs to implement a temporary waiver for the strict API
> > checking requirements that were introduced last year, to give downstream
> > deployers more time to catch up with the strict micro-versioning
> > requirements determined by the Nova/Compute team and enforced by the
> > Tempest/QA team.
> 
> I'm very much opposed to this being done. If we're actually concerned with
> interoperability and verify that things behave in the same manner between multiple
> clouds then doing this would be a big step backwards. The fundamental disconnect
> here is that the vendors who have implemented out of band extensions or were
> taking advantage of previously available places to inject extra attributes
> believe that doing so means they're interoperable, which is quite far from
> reality. **The API is not a place for vendor differentiation.**

This is a temporary measure to address the fact that a large number
of existing tests changed their behavior, rather than having new
tests added to enforce this new requirement. The result is deployments
that previously passed these tests may no longer pass, and in fact
we have several cases where that's true with deployers who are
trying to maintain their own standard of backwards-compatibility
for their end users.

We have basically three options.

1. Tell deployers who are trying to do the right for their immediate
   users that they can't use the trademark.

2. Flag the related tests or remove them from the DefCore enforcement
   suite entirely.

3. Be flexible about giving consumers of Tempest time to meet the
   new requirement by providing a way to disable the checks.

Option 1 goes against our own backwards compatibility policies.

Option 2 gives us no winners and actually reduces the interoperability
guarantees we already have in place.

Option 3 applies our usual community standard of slowly rolling
forward while maintaining compatibility as broadly as possible.

No one is suggesting that a permanent, or even open-ended, exception
be granted.

Doug

> 
> As a user of several clouds myself I can say that having random gorp in a
> response makes it much more difficult to use my code against multiple clouds. I
> have to determine which properties being returned are specific to that vendor's
> cloud and if I actually need to depend on them for anything it makes whatever
> code I'm writing incompatible for using against any other cloud. (unless I
> special case that block for each cloud) Sean Dague wrote a good post where a lot
> of this was covered a year ago when microversions was starting to pick up steam:
> 
> https://dague.net/2015/06/05/the-nova-api-in-kilo-and-beyond-2
> 
> I'd recommend giving it a read, he explains the user first perspective more
> clearly there.
> 
> I believe Tempest in this case is doing the right thing from an interoperability
> perspective and ensuring that the API is actually the API. Not an API with extra
> bits a vendor decided to add. I don't think a cloud or product that does this
> to the api should be considered an interoperable OpenStack cloud and failing the
> tests is the correct behavior.
> 
> -Matt Treinish
> 
> > 
> > My reasoning behind this is that while the change that enabled strict
> > checking was discussed publicly in the developer community and took
> > some time to be implemented, it still landed quickly and broke several
> > existing deployments overnight. As Tempest has moved forward with
> > bug and UX fixes (some in part to support the interoperability testing
> > efforts of the DefCore Working Group), using an older versions of Tempest
> > where this strict checking is not enforced is no longer a viable solution
> > for downstream deployers. The TC has passed a resolution to advise
> > DefCore to use Tempest as the single source of capability testing[7],
> > but this naturally introduces tension between the competing goals of
> > maintaining upstream functional testing and also tracking lagging
> > indicators.
> > 
> > My proposal for addressing this problem approaches it at two levels:
> > 
> > * For the short term, I will submit a blueprint and patch to tempest that
> >   allows configuration of a grey-list of Nova APIs where strict response
> >   checking on additional properties will be disabled. So, for example,
> >   if the 'create  servers' API call returned extra properties on that call,
> >   the strict checking on this line[8] would be disabled at runtime.
> >   Use of this code path will emit a deprecation warning, and the
> >   code will be scheduled for removal in 2017 directly after the release
> >   of the 2017.01 guideline. Vendors would be required so submit the
> >   grey-list of APIs with additional response data that would be
> >   published to their marketplace entry.
> > 
> > * Longer term, vendors will be expected to work with upstream to update
> >   the API for returning additional data that is compatible with
> >   API micro-versioning as defined by the Nova team, and the waiver would
> >   no longer be allowed after the release of the 2017.01 guideline.
> > 
> > For the next half-year, I feel that this approach strengthens interoperability
> > by accurately capturing the current state of OpenStack deployments and
> > client tools. Before this change, additional properties on responses
> > weren't explicitly disallowed, and vendors and deployers took advantage
> > of this in production. While this is behavior that the Nova and QA teams
> > want to stop, it will take a bit more time to reach downstream. Also, as
> > of right now, as far as I know the only client that does strict response
> > checking for Nova responses is the Tempest client. Currently, additional
> > properties in responses are ignored and do not break existing client
> > functionality. There is currently little to no harm done to downstream
> > users by temporarily allowing additional data to be returned in responses.
> > 
> > Thanks,
> > 
> > Chris Hoge
> > Interop Engineer
> > OpenStack Foundation
> > 
> > [1] https://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/api-microversions.html
> > [2] http://lists.openstack.org/pipermail/openstack-dev/2015-February/057613.html
> > [3] https://review.openstack.org/#/c/156130
> > [4] http://lists.openstack.org/pipermail/defcore-committee/2016-January/000986.html
> > [5] http://git.openstack.org/cgit/openstack/defcore/tree/2015.07.json
> > [6] http://git.openstack.org/cgit/openstack/defcore/tree/2016.01.json
> > [7] http://git.openstack.org/cgit/openstack/governance/tree/resolutions/20160504-defcore-test-location.rst
> > [8] http://git.openstack.org/cgit/openstack/tempest-lib/tree/tempest_lib/api_schema/response/compute/v2_1/servers.py#n39