Open Stack

On 04/27/2014 10:02 PM, Matthew Treinish wrote:
> On Mon, Apr 28, 2014 at 01:01:00AM +0000, Kenichi Oomichi wrote:
>> Hi,
>>
>> Sorry for my late response, but I'd like to discuss this again.
>>
>> Now we are working for adding Nova API responses checks to Tempest[1] to
>> block backward incompatible changes.
>> With this work, Tempest checks each response(status code, response body)
>> and raises a test failure exception if detecting something unexpected.
>> For example if some API parameter, which is defined as 'required' Tempest
>> side, does not exist in response body, Tempest test fails.
>>
>> We are defining API parameters as 'required' if they are not API extensions
>> or they are not depended on Nova configuration. In addition now Tempest
>> allows additional API parameters, that means Tempest does not fail even if
>> Nova response includes unexpected API parameters. Because I think the removal
>> of API parameter causes backward incompatible issue but the addition does not
>> cause it.
> So, AIUI we can only add parameters to an API with a new extension. The API
> change guidelines also say that adding new properties must be conditional:
>
> https://wiki.openstack.org/wiki/APIChangeGuidelines#Generally_Considered_OK
I just wanted to note that the original referenced wiki page, assembled 
by markmc and myself, did not specify the need for an extension in order 
to add a value to a return dict or add a value to a dict argument if the 
existing api would ignore it. This was changed (and honestly I did not 
notice this change at the time) last August to require an extension: 
https://wiki.openstack.org/w/index.php?title=APIChangeGuidelines&diff=prev&oldid=28593. 

Is there any trace left of discussions around that change?

The original wording allowed the api to evolve as long as a "reasonable" 
application would not be broken. Essentially the extra value becomes 
optional and new client or server code can check for it. The new 
definition is quite strict and similar to what leads to many Windows 
APIs having names like CreateWindowEx. Not saying being strict is bad, 
but it will require a lot of changes to the client libraries as well as 
tempest because there are a lot of extensions that are not checked by 
either.


  -David
>
> Adding or removing a parameter to an API is a backwards incompatible change IMO
> for the exact reasons you mentioned here. If we have to worry about it in
> tempest then end users do as well.
>
> This is also why there are a bunch of nova v2 extensions that just add
> properties to an existing API. I think in v3 the proposal was to do this with
> microversioning of the plugins. (we don't have a way to configure
> microversioned v3 api plugins in tempest yet, but we can cross that bridge when
> the time comes) Either way it will allow tempest to have in config which
> behavior to expect.
>
>> In this situation, there is a problem related to branchless Tempest.
>> When we define new API parameter as 'required', Tempest against old release
>> would fail.
> So I feel that if we are marking something in the API as required in tempest
> makes the test fail in a previous release than that should be considered a bug
> in the old release (assuming it was correct to mark it as required), and that
> should be backportable fix.
>
>> I think we need to define new parameters, which do not depended on the
>> configuration, as 'required' in Tempest when we have added them in the
>> development cycle because of blocking backward incompatible changes in
>> the future. However these parameters are new and old releases don't contain
>> them, so the Tempest change causes failures against old releases tests.
> I agree we should mark the required parameters as those that are used without
> any extensions. If we can also conditionally check those not marked as required
> based on the enabled extensions or features in the tempest config that would
> provide the best coverage.
>
> So for master branch development on tempest I think we should only concern
> ourselves with getting these things to work against Juno and Icehouse. I think
> Havana support using master is a lost cause at this point so we'll keep the
> stable branch around until it's EOL. So hopefully we can lock down things in
> tempest with the new api attribute tests quickly so we can block the Juno
> additions that would violate the stability guidelines. It would be a shame if
> we managed to allow a breaking API change into an API since the release. (but
> hopefully it would be an easy backport or revert)
>
>> Case: add new parameter 'A' in Juno cycle
>>
>>             Icehouse            Juno                K                   L
>>           --*-------------------*-------------------*-------------------*--
>>   Nova:        new parameter 'A'
>>   Tempest:         define 'A' as 'required'
>>                                                     block 'A' removal   block ..
>>             test fails due to non-existent 'A'
> So in this example I feel that parameter 'A' can only be added as an extension.
> (or some other condition) If it's not then it's a breaking api change which will
> be caught which is the desired behavior from tempest here.
>
> Thanks,
>
> Matt Treinish
>
>>
>> I have not found enough idea for this yet.
>>
>> Thanks
>> Ken'ichi Ohmichi
>>
>> ---
>> [1]: https://blueprints.launchpad.net/tempest/+spec/nova-api-attribute-test
>>
>>> -----Original Message-----
>>> From: Sean Dague [mailto:sean at dague.net]
>>> Sent: Monday, April 14, 2014 10:22 PM
>>> To: OpenStack Development Mailing List
>>> Subject: [openstack-dev] [all] Branchless Tempest QA Spec - final draft
>>>
>>> As we're coming up on the stable/icehouse release the QA team is looking
>>> pretty positive at no longer branching Tempest. The QA Spec draft for
>>> this is here -
>>> http://docs-draft.openstack.org/77/86577/2/check/gate-qa-specs-docs/3f84796/doc/build/html/specs/branchless-tempest.
>>> html
>>> and hopefully address a lot of the questions we've seen so far.
>>>
>>> Additional comments are welcome on the review -
>>> https://review.openstack.org/#/c/86577/
>>> or as responses on this ML thread.
>>>
>>> 	-Sean
>>>
>>> --
>>> Sean Dague
>>> Samsung Research America
>>> sean at dague.net / sean.dague at samsung.com
>>> http://dague.net
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev