Open Stack

Thanks Matt for bringing this up.


On Thu, Jul 14, 2016 at 11:55 AM, Matt Riedemann
<mriedem at linux.vnet.ibm.com> wrote:
> There are several changes in Tempest right now trying to add response schema
> validation for the 2.26 microversion which added server tags to the server
> GET response. This is needed for anything testing a microversion >=2.26,
> which several people are trying to add.
>
> We have a similar issue with the 2.3 microversion which is really a bug, but
> only exposed in jobs that have run_validation=True which is only in a
> non-voting job right now.
>
> I've mostly been debating this in this change:
>
> https://review.openstack.org/#/c/233176/
>
> I've added an item to the nova midcycle meetup agenda to talk about the plan
> for handling microversion testing in tempest for nova changes, specifically
> around API response validation.
>
> I agree that nova doesn't test response schema validation in tree, so doing
> it in tempest is good.
>
> But I'm not sure that we need a new set of tempest tests for every
> microversion change in nova, e.g. if it's only touching the API and
> database, like server tags, we can test that in nova.
>
> It's also not great having several changes in flight at the same time to
> tempest trying to add the same 2.26 response schema because it wasn't added
> the at the same time the 2.26 API merged.
>
> I also wonder what it means if someone configures max_microversion in
> tempest.conf to something we don't test, like say 2.11, what blows up? For
> example, we know that we don't have response validation for 2.3 so some
> tests are broken when you run with ssh validation and microversion>=2.3.

you mean if we increase min_microversion? For max_microversion
capping, It should just skip the tests which are going
out of configured boundary.

But yes there will be failure if we increase the min_microversion to
some higher version which schema is missing (as tests is not added).

2.3 schema was missed as this tests was not added yet in tempest. So
we might have same case in future also.

I am trying to add all version schema needed till Mitaka max supported
version which is 2.25 in https://review.openstack.org/#/c/287605/
After that, we should have all lower version schema available for
other version tests(<2.25). This should make things easy for other new
tests.

>
> So I'm thinking we should:
>
> 1. Always add a schema change to Tempest if a microversion changes a
> response.

We should not actually. Schema are more error prone if we add
manually. We have to be very careful on all attributes, their type and
range etc
which can be verified with tests only. Adding schema in Tempest may
end up adding wrong schema in Tempest.

>
> 2. Only add tests to Tempest for a microversion change if it can't be tested
> within nova, e.g. you actually need to create a server, or use other
> services like glance/cinder/neutron.

Almost +1.
Actually This is fine till we do not increase the min_microversion.
Any new version tests coming in Tempest should add lower version
schema also if needed. This is mentioned very clearly in doc [1] also.
But if anytime we increase the minimum version (which is almost
impossible) then yes, we should have tests for all microversion so
that no failure with schema things. Honestly saying, I would like to
ignore this case as of now as very rare to rare that nova will bump
minimum version.

I am agree on this But also want to tests each microversion changes
through schema which does not mean we need to add tests for each
microversion. Higher microversion tests(which are valid for Tempest
like interacting with other services etc) can verify same API's lower
version changes through schema automatically.

Tempest testing which does schema validation are very useful to find
API bugs. We have found many in past and recently also [2].  How about
this:
- Do not add tests for changes which can be tested in Nova as
mentioned by you in 2nd point with below exception:
    - Exception- If that version need schema change and those schema
changes are not covered by any higher version tests then add tests in
Tempest.

And this we can done at end of each cycle to make sure each version
changes in this cycle are tested thorough schema validation (is change
in schema) in Tempest.



>
> mtreinish and sdague will be at the nova midcycle so hopefully they can
> represent for the QA team.

..[1] http://docs.openstack.org/developer/tempest/microversion_testing.html#notes-about-compute-microversion-tests
..[2]  https://bugs.launchpad.net/nova/+bug/1602044
>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev