[openstack-dev] [nova][qa] When do we need tests for microversions in Tempest?
mriedem at linux.vnet.ibm.com
Fri Jul 15 00:13:01 UTC 2016
On 7/13/2016 10:18 PM, Ken'ichi Ohmichi wrote:
> Hi Matt,
> 2016-07-14 11:55 GMT+09:00 Matt Riedemann <mriedem at linux.vnet.ibm.com>:
>> There are several changes in Tempest right now trying to add response schema
>> validation for the 2.26 microversion which added server tags to the server
>> GET response. This is needed for anything testing a microversion >=2.26,
>> which several people are trying to add.
>> We have a similar issue with the 2.3 microversion which is really a bug, but
>> only exposed in jobs that have run_validation=True which is only in a
>> non-voting job right now.
>> I've mostly been debating this in this change:
>> I've added an item to the nova midcycle meetup agenda to talk about the plan
>> for handling microversion testing in tempest for nova changes, specifically
>> around API response validation.
>> I agree that nova doesn't test response schema validation in tree, so doing
>> it in tempest is good.
>> But I'm not sure that we need a new set of tempest tests for every
>> microversion change in nova, e.g. if it's only touching the API and
>> database, like server tags, we can test that in nova.
> I think it is nice to implement every microversion tests in Tempest
> for preparing the base/minimum microversion bump in the future.
> On API microversions mechanism, we have defined the minimum
> microversion(now version 2.1) can be increased.
> Every microversion test will provide an option when we want to bump it
> to verify a new minimum microversion is stable by Tempest as an
> integrated test.
We're pretty far out from raising the minimum required microversion in
Nova I think, like several releases at least.
The issue I have really is a few releases ago there was a push to pair
down how much is tested in Tempest in the integrated gate, and if tests
only hit the API for a single service, those tests should live in the
project/service that's being tested. This is why the CLI tests moved out
and a bunch of things that only hit the API and database.
We'd be going back on that now if we need to implement a test for every
microversion that shows up.
Maybe Nova needs to think about a Tempest plugin for stuff like that?
>> It's also not great having several changes in flight at the same time to
>> tempest trying to add the same 2.26 response schema because it wasn't added
>> the at the same time the 2.26 API merged.
>> I also wonder what it means if someone configures max_microversion in
>> tempest.conf to something we don't test, like say 2.11, what blows up? For
>> example, we know that we don't have response validation for 2.3 so some
>> tests are broken when you run with ssh validation and microversion>=2.3.
> Ideally, the response schema also should be implemented on Tempest.
> but it is difficult to catch up because the microversion bumping is
> fast on Nova side.
> The max_microversion option is useful to operate the same Tempest to
> master/stable branches which are different microversions.
> Ken Ohmichi
>> So I'm thinking we should:
>> 1. Always add a schema change to Tempest if a microversion changes a
>> 2. Only add tests to Tempest for a microversion change if it can't be tested
>> within nova, e.g. you actually need to create a server, or use other
>> services like glance/cinder/neutron.
>> mtreinish and sdague will be at the nova midcycle so hopefully they can
>> represent for the QA team.
>> Matt Riedemann
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
More information about the OpenStack-dev