Open Stack

Last year, in response to Nova micro-versioning and extension updates[1],
the QA team added strict API schema checking to Tempest to ensure that
no additional properties were added to Nova API responses[2][3]. In the
last year, at least three vendors participating the the OpenStack Powered
Trademark program have been impacted by this change, two of which
reported this to the DefCore Working Group mailing list earlier this year[4].

The DefCore Working Group determines guidelines for the OpenStack Powered
program, which includes capabilities with associated functional tests
from Tempest that must be passed, and designated sections with associated
upstream code [5][6]. In determining these guidelines, the working group
attempts to balance the future direction of development with lagging
indicators of deployments and user adoption.

After a tremendous amount of consideration, I believe that the DefCore
Working Group needs to implement a temporary waiver for the strict API
checking requirements that were introduced last year, to give downstream
deployers more time to catch up with the strict micro-versioning
requirements determined by the Nova/Compute team and enforced by the
Tempest/QA team.

My reasoning behind this is that while the change that enabled strict
checking was discussed publicly in the developer community and took
some time to be implemented, it still landed quickly and broke several
existing deployments overnight. As Tempest has moved forward with
bug and UX fixes (some in part to support the interoperability testing
efforts of the DefCore Working Group), using an older versions of Tempest
where this strict checking is not enforced is no longer a viable solution
for downstream deployers. The TC has passed a resolution to advise
DefCore to use Tempest as the single source of capability testing[7],
but this naturally introduces tension between the competing goals of
maintaining upstream functional testing and also tracking lagging
indicators.

My proposal for addressing this problem approaches it at two levels:

* For the short term, I will submit a blueprint and patch to tempest that
  allows configuration of a grey-list of Nova APIs where strict response
  checking on additional properties will be disabled. So, for example,
  if the 'create  servers' API call returned extra properties on that call,
  the strict checking on this line[8] would be disabled at runtime.
  Use of this code path will emit a deprecation warning, and the
  code will be scheduled for removal in 2017 directly after the release
  of the 2017.01 guideline. Vendors would be required so submit the
  grey-list of APIs with additional response data that would be
  published to their marketplace entry.

* Longer term, vendors will be expected to work with upstream to update
  the API for returning additional data that is compatible with
  API micro-versioning as defined by the Nova team, and the waiver would
  no longer be allowed after the release of the 2017.01 guideline.

For the next half-year, I feel that this approach strengthens interoperability
by accurately capturing the current state of OpenStack deployments and
client tools. Before this change, additional properties on responses
weren't explicitly disallowed, and vendors and deployers took advantage
of this in production. While this is behavior that the Nova and QA teams
want to stop, it will take a bit more time to reach downstream. Also, as
of right now, as far as I know the only client that does strict response
checking for Nova responses is the Tempest client. Currently, additional
properties in responses are ignored and do not break existing client
functionality. There is currently little to no harm done to downstream
users by temporarily allowing additional data to be returned in responses.

Thanks,

Chris Hoge
Interop Engineer
OpenStack Foundation

[1] https://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/api-microversions.html
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-February/057613.html
[3] https://review.openstack.org/#/c/156130
[4] http://lists.openstack.org/pipermail/defcore-committee/2016-January/000986.html
[5] http://git.openstack.org/cgit/openstack/defcore/tree/2015.07.json
[6] http://git.openstack.org/cgit/openstack/defcore/tree/2016.01.json
[7] http://git.openstack.org/cgit/openstack/governance/tree/resolutions/20160504-defcore-test-location.rst
[8] http://git.openstack.org/cgit/openstack/tempest-lib/tree/tempest_lib/api_schema/response/compute/v2_1/servers.py#n39