Open Stack

On Thu, Jul 14, 2016 at 12:08 PM, Matthew Treinish <mtreinish at kortar.org> wrote:
> On Wed, Jul 13, 2016 at 09:55:33PM -0500, Matt Riedemann wrote:
>> There are several changes in Tempest right now trying to add response schema
>> validation for the 2.26 microversion which added server tags to the server
>> GET response. This is needed for anything testing a microversion >=2.26,
>> which several people are trying to add.
>>
>> We have a similar issue with the 2.3 microversion which is really a bug, but
>> only exposed in jobs that have run_validation=True which is only in a
>> non-voting job right now.
>>
>> I've mostly been debating this in this change:
>>
>> https://review.openstack.org/#/c/233176/
>>
>> I've added an item to the nova midcycle meetup agenda to talk about the plan
>> for handling microversion testing in tempest for nova changes, specifically
>> around API response validation.
>>
>> I agree that nova doesn't test response schema validation in tree, so doing
>> it in tempest is good.
>>
>> But I'm not sure that we need a new set of tempest tests for every
>> microversion change in nova, e.g. if it's only touching the API and
>> database, like server tags, we can test that in nova.
>
> I largely agree with this, we don't need 100% coverage of every microversion.
> Especially if it's just an API that's just extra metadata in the DB.
>
>>
>> It's also not great having several changes in flight at the same time to
>> tempest trying to add the same 2.26 response schema because it wasn't added
>> the at the same time the 2.26 API merged.
>
> I agree it's not ideal, but it's not like there is a huge burden on rebasing,
> no more than for developers having to bump their microversions because another
> bp landed and took the microversion they were using.
>
>
>>
>> I also wonder what it means if someone configures max_microversion in
>> tempest.conf to something we don't test, like say 2.11, what blows up? For
>> example, we know that we don't have response validation for 2.3 so some
>> tests are broken when you run with ssh validation and microversion>=2.3.
>
> We can easily add a job that changes the min microversion config flag in tempest
> to something higher than 2.1. This will ensure we send a higher microversion
> everywhere and will catch these issues sooner. But, I'm not sure we want to do
> that on a normal check/gate job.

Yes, there will be lot of tests failure not with schema thing only but
we do tests some bit
in tests which might got changed in versions.

But we have plan for that. to detect those tests and modify/cap with
appropriate boundary. But not sure when :)


>
>>
>> So I'm thinking we should:
>>
>> 1. Always add a schema change to Tempest if a microversion changes a
>> response.
>
> The problem with this is we shouldn't land a schema change by itself in tempest.
> Until we have something using the schema we have no verification that they
> actually work. We can and will land incorrect schemas if we did this. That's why
> there is a pretty strong policy of only landing code that is run in CI somewhere
> for Tempest.

+1, yes we should not add those without testing.

>
>>
>> 2. Only add tests to Tempest for a microversion change if it can't be tested
>> within nova, e.g. you actually need to create a server, or use other
>> services like glance/cinder/neutron.
>
> +1
>
> -Matt Treinish
>
>>
>> mtreinish and sdague will be at the nova midcycle so hopefully they can
>> represent for the QA team.
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>