Open Stack

On 7/13/2016 10:18 PM, Ken'ichi Ohmichi wrote:
> Hi Matt,
>
>
> 2016-07-14 11:55 GMT+09:00 Matt Riedemann <mriedem at linux.vnet.ibm.com>:
>> There are several changes in Tempest right now trying to add response schema
>> validation for the 2.26 microversion which added server tags to the server
>> GET response. This is needed for anything testing a microversion >=2.26,
>> which several people are trying to add.
>>
>> We have a similar issue with the 2.3 microversion which is really a bug, but
>> only exposed in jobs that have run_validation=True which is only in a
>> non-voting job right now.
>>
>> I've mostly been debating this in this change:
>>
>> https://review.openstack.org/#/c/233176/
>>
>> I've added an item to the nova midcycle meetup agenda to talk about the plan
>> for handling microversion testing in tempest for nova changes, specifically
>> around API response validation.
>>
>> I agree that nova doesn't test response schema validation in tree, so doing
>> it in tempest is good.
>>
>> But I'm not sure that we need a new set of tempest tests for every
>> microversion change in nova, e.g. if it's only touching the API and
>> database, like server tags, we can test that in nova.
>
> I think it is nice to implement every microversion tests in Tempest
> for preparing the base/minimum microversion bump in the future.
> On API microversions mechanism, we have defined the minimum
> microversion(now version 2.1) can be increased.
> Every microversion test will provide an option when we want to bump it
> to verify a new minimum microversion is stable by Tempest as an
> integrated test.

We're pretty far out from raising the minimum required microversion in 
Nova I think, like several releases at least.

The issue I have really is a few releases ago there was a push to pair 
down how much is tested in Tempest in the integrated gate, and if tests 
only hit the API for a single service, those tests should live in the 
project/service that's being tested. This is why the CLI tests moved out 
and a bunch of things that only hit the API and database.

We'd be going back on that now if we need to implement a test for every 
microversion that shows up.

Maybe Nova needs to think about a Tempest plugin for stuff like that?

>
>> It's also not great having several changes in flight at the same time to
>> tempest trying to add the same 2.26 response schema because it wasn't added
>> the at the same time the 2.26 API merged.
>>
>> I also wonder what it means if someone configures max_microversion in
>> tempest.conf to something we don't test, like say 2.11, what blows up? For
>> example, we know that we don't have response validation for 2.3 so some
>> tests are broken when you run with ssh validation and microversion>=2.3.
>
> Ideally, the response schema also should be implemented on Tempest.
> but it is difficult to catch up because the microversion bumping is
> fast on Nova side.
> The max_microversion option is useful to operate the same Tempest to
> master/stable branches which are different microversions.
>
> Thanks
> Ken Ohmichi
>
> ---
>
>> So I'm thinking we should:
>>
>> 1. Always add a schema change to Tempest if a microversion changes a
>> response.
>>
>> 2. Only add tests to Tempest for a microversion change if it can't be tested
>> within nova, e.g. you actually need to create a server, or use other
>> services like glance/cinder/neutron.
>>
>> mtreinish and sdague will be at the nova midcycle so hopefully they can
>> represent for the QA team.
>>
>> --
>>
>> Thanks,
>>
>> Matt Riedemann
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


-- 

Thanks,

Matt Riedemann