Open Stack

Hi,

Sorry for my late response, but I'd like to discuss this again.

Now we are working for adding Nova API responses checks to Tempest[1] to
block backward incompatible changes.
With this work, Tempest checks each response(status code, response body)
and raises a test failure exception if detecting something unexpected.
For example if some API parameter, which is defined as 'required' Tempest
side, does not exist in response body, Tempest test fails.

We are defining API parameters as 'required' if they are not API extensions
or they are not depended on Nova configuration. In addition now Tempest
allows additional API parameters, that means Tempest does not fail even if
Nova response includes unexpected API parameters. Because I think the removal
of API parameter causes backward incompatible issue but the addition does not
cause it.

In this situation, there is a problem related to branchless Tempest.
When we define new API parameter as 'required', Tempest against old release
would fail.

I think we need to define new parameters, which do not depended on the
configuration, as 'required' in Tempest when we have added them in the
development cycle because of blocking backward incompatible changes in
the future. However these parameters are new and old releases don't contain
them, so the Tempest change causes failures against old releases tests.

Case: add new parameter 'A' in Juno cycle

           Icehouse            Juno                K                   L
         --*-------------------*-------------------*-------------------*--
 Nova:        new parameter 'A'
 Tempest:         define 'A' as 'required'
                                                   block 'A' removal   block ..
           test fails due to non-existent 'A'


I have not found enough idea for this yet.

Thanks
Ken'ichi Ohmichi

---
[1]: https://blueprints.launchpad.net/tempest/+spec/nova-api-attribute-test

> -----Original Message-----
> From: Sean Dague [mailto:sean at dague.net]
> Sent: Monday, April 14, 2014 10:22 PM
> To: OpenStack Development Mailing List
> Subject: [openstack-dev] [all] Branchless Tempest QA Spec - final draft
> 
> As we're coming up on the stable/icehouse release the QA team is looking
> pretty positive at no longer branching Tempest. The QA Spec draft for
> this is here -
> http://docs-draft.openstack.org/77/86577/2/check/gate-qa-specs-docs/3f84796/doc/build/html/specs/branchless-tempest.
> html
> and hopefully address a lot of the questions we've seen so far.
> 
> Additional comments are welcome on the review -
> https://review.openstack.org/#/c/86577/
> or as responses on this ML thread.
> 
> 	-Sean
> 
> --
> Sean Dague
> Samsung Research America
> sean at dague.net / sean.dague at samsung.com
> http://dague.net