Open Stack

On Mon, Apr 28, 2014 at 01:01:00AM +0000, Kenichi Oomichi wrote:
> 
> Hi,
> 
> Sorry for my late response, but I'd like to discuss this again.
> 
> Now we are working for adding Nova API responses checks to Tempest[1] to
> block backward incompatible changes.
> With this work, Tempest checks each response(status code, response body)
> and raises a test failure exception if detecting something unexpected.
> For example if some API parameter, which is defined as 'required' Tempest
> side, does not exist in response body, Tempest test fails.
> 
> We are defining API parameters as 'required' if they are not API extensions
> or they are not depended on Nova configuration. In addition now Tempest
> allows additional API parameters, that means Tempest does not fail even if
> Nova response includes unexpected API parameters. Because I think the removal
> of API parameter causes backward incompatible issue but the addition does not
> cause it.

So, AIUI we can only add parameters to an API with a new extension. The API
change guidelines also say that adding new properties must be conditional:

https://wiki.openstack.org/wiki/APIChangeGuidelines#Generally_Considered_OK

Adding or removing a parameter to an API is a backwards incompatible change IMO
for the exact reasons you mentioned here. If we have to worry about it in
tempest then end users do as well.

This is also why there are a bunch of nova v2 extensions that just add
properties to an existing API. I think in v3 the proposal was to do this with
microversioning of the plugins. (we don't have a way to configure
microversioned v3 api plugins in tempest yet, but we can cross that bridge when
the time comes) Either way it will allow tempest to have in config which
behavior to expect.

> 
> In this situation, there is a problem related to branchless Tempest.
> When we define new API parameter as 'required', Tempest against old release
> would fail.

So I feel that if we are marking something in the API as required in tempest
makes the test fail in a previous release than that should be considered a bug
in the old release (assuming it was correct to mark it as required), and that
should be backportable fix.

> 
> I think we need to define new parameters, which do not depended on the
> configuration, as 'required' in Tempest when we have added them in the
> development cycle because of blocking backward incompatible changes in
> the future. However these parameters are new and old releases don't contain
> them, so the Tempest change causes failures against old releases tests.

I agree we should mark the required parameters as those that are used without
any extensions. If we can also conditionally check those not marked as required
based on the enabled extensions or features in the tempest config that would
provide the best coverage.

So for master branch development on tempest I think we should only concern
ourselves with getting these things to work against Juno and Icehouse. I think
Havana support using master is a lost cause at this point so we'll keep the
stable branch around until it's EOL. So hopefully we can lock down things in
tempest with the new api attribute tests quickly so we can block the Juno
additions that would violate the stability guidelines. It would be a shame if
we managed to allow a breaking API change into an API since the release. (but
hopefully it would be an easy backport or revert)

> 
> Case: add new parameter 'A' in Juno cycle
> 
>            Icehouse            Juno                K                   L
>          --*-------------------*-------------------*-------------------*--
>  Nova:        new parameter 'A'
>  Tempest:         define 'A' as 'required'
>                                                    block 'A' removal   block ..
>            test fails due to non-existent 'A'

So in this example I feel that parameter 'A' can only be added as an extension.
(or some other condition) If it's not then it's a breaking api change which will
be caught which is the desired behavior from tempest here.

Thanks,

Matt Treinish

> 
> 
> I have not found enough idea for this yet.
> 
> Thanks
> Ken'ichi Ohmichi
> 
> ---
> [1]: https://blueprints.launchpad.net/tempest/+spec/nova-api-attribute-test
> 
> > -----Original Message-----
> > From: Sean Dague [mailto:sean at dague.net]
> > Sent: Monday, April 14, 2014 10:22 PM
> > To: OpenStack Development Mailing List
> > Subject: [openstack-dev] [all] Branchless Tempest QA Spec - final draft
> > 
> > As we're coming up on the stable/icehouse release the QA team is looking
> > pretty positive at no longer branching Tempest. The QA Spec draft for
> > this is here -
> > http://docs-draft.openstack.org/77/86577/2/check/gate-qa-specs-docs/3f84796/doc/build/html/specs/branchless-tempest.
> > html
> > and hopefully address a lot of the questions we've seen so far.
> > 
> > Additional comments are welcome on the review -
> > https://review.openstack.org/#/c/86577/
> > or as responses on this ML thread.
> > 
> > 	-Sean
> > 
> > --
> > Sean Dague
> > Samsung Research America
> > sean at dague.net / sean.dague at samsung.com
> > http://dague.net
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev