[openstack-dev] [all] Branchless Tempest QA Spec - final draft

Ken'ichi Ohmichi ken1ohmichi at gmail.com
Thu May 1 09:07:46 UTC 2014


Hi Matthew,

2014-04-28 11:54 GMT+09:00 Ken'ichi Ohmichi <ken1ohmichi at gmail.com>:
> 2014-04-28 11:02 GMT+09:00 Matthew Treinish <mtreinish at kortar.org>:
>> On Mon, Apr 28, 2014 at 01:01:00AM +0000, Kenichi Oomichi wrote:
>>>
>>> Now we are working for adding Nova API responses checks to Tempest[1] to
>>> block backward incompatible changes.
>>> With this work, Tempest checks each response(status code, response body)
>>> and raises a test failure exception if detecting something unexpected.
>>> For example if some API parameter, which is defined as 'required' Tempest
>>> side, does not exist in response body, Tempest test fails.
>>>
>>> We are defining API parameters as 'required' if they are not API extensions
>>> or they are not depended on Nova configuration. In addition now Tempest
>>> allows additional API parameters, that means Tempest does not fail even if
>>> Nova response includes unexpected API parameters. Because I think the removal
>>> of API parameter causes backward incompatible issue but the addition does not
>>> cause it.
>>
>> So, AIUI we can only add parameters to an API with a new extension. The API
>> change guidelines also say that adding new properties must be conditional:
>>
>> https://wiki.openstack.org/wiki/APIChangeGuidelines#Generally_Considered_OK
>>
>> Adding or removing a parameter to an API is a backwards incompatible change IMO
>> for the exact reasons you mentioned here. If we have to worry about it in
>> tempest then end users do as well.
>>
>> This is also why there are a bunch of nova v2 extensions that just add
>> properties to an existing API. I think in v3 the proposal was to do this with
>> microversioning of the plugins. (we don't have a way to configure
>> microversioned v3 api plugins in tempest yet, but we can cross that bridge when
>> the time comes) Either way it will allow tempest to have in config which
>> behavior to expect.
>
> Good point, my current understanding is:
> When adding new API parameters to the existing APIs, these parameters should
> be API extensions according to the above guidelines. So we have three options
> for handling API extensions in Tempest:
>
> 1. Consider them as optional, and cannot block the incompatible
> changes of them. (Current)
> 2. Consider them as required based on tempest.conf, and can block the
> incompatible changes.
> 3. Consider them as required automatically with microversioning, and
> can block the incompatible changes.

I investigated the way of the above option 3, then have one question
about current
Tempest implementation.

Now verify_tempest_config tool gets API extension list from each
service including
Nova and verifies API extension config of tempest.conf based on the list.
Can we use the list for selecting what extension tests run instead of
the verification?
As you said In the previous IRC meeting, current API tests will be
skipped if the
test which is decorated with requires_ext() and the extension is not
specified in
tempest.conf. I feel it would be nice that Tempest gets API extension
list and selects
API tests automatically based on the list.
In addition, The methods which are decorated with requires_ext() are
test methods now,
but I think it would be better to decorate client
methods(get_hypervisor_list, etc.) because
each extension loading condition affects available APIs.

Thanks
Ken'ichi Ohmichi



More information about the OpenStack-dev mailing list