[openstack-dev] [api][qa][tc][nova][cinder] Testing of a microversioned world

Andrea Frittoli andrea.frittoli at gmail.com
Fri Mar 10 21:02:55 UTC 2017


On Fri, Mar 10, 2017 at 8:39 PM Ken'ichi Ohmichi <ken1ohmichi at gmail.com>
wrote:

> Hi John,
>
> Now Tempest is testing microversions only for Nova and contains some
> testing framework for re-using for another projects.
> On this framework, we can implement necessary microversions tests as
> we want and actually many microversions of Nova are not tested by
> Tempest.
> We can see the tested microversion of Nova on
>
> https://github.com/openstack/tempest/blob/master/doc/source/microversion_testing.rst#microversion-tests-implemented-in-tempest
>
> Before implementing microversion testing for Cinder, we will implement
> JSON-Schema validation for API responses for Cinder.
> The validation will be helpful for testing base microversion of Cinder
> API and we will be able to implement the microversion tests based on
> that.
> This implementation is marked as 7th priority in this Pike cycle as
> https://etherpad.openstack.org/p/pike-qa-priorities
>
> In addition, now Cinder V3 API is not tested. So we are going to
> enable v3 tests with some restructure of Tempest in this cycle.
> The detail is described on the part of "Volume API" of
> https://etherpad.openstack.org/p/tempest-api-versions-in-pike
>
> Thanks
> Ken Ohmichi
>
> ---
>
> 2017-03-10 11:37 GMT-08:00 John Griffith <john.griffith8 at gmail.com>:
> > Hey Everyone,
> >
> > So along the lines of an earlier thread that went out regarding testing
> of
> > deprecated API's and Tempest etc [1].
> >
> > Now that micro-versions are *the API versioning scheme to rule them all*
> one
> > question I've not been able to find an answer for is what we're going to
> > promise here for support and testing.  My understanding thus far is that
> the
> > "community" approach here is "nothing is ever deprecated, and everything
> is
> > supported forever".
> >
> > That's sort of a tall order IMO, but ok.  I've already had some questions
> > from folks about implementing an explicit Tempest test for every
> > micro-versioned implementation of an API call also.  My response has been
> > "nahh, just always test latest available".  This kinda means that we're
> not
> > testing/supporting the previous versions as promised though.
>

We had a couple of sessions related to this topic at the PTG [0][1].

We agreed that we want to still maintain integration tests only in Tempest,
which means that API micro versions that have no integration impact can be
tested via functional tests.

Since we do schema validation, when someone wants to develop an integration
tests for a specific micro version, there may be a gap in the schemas
implemented on Tempest side and the current one - we had this case already
in nova.
We decided we shall monitor this gap, and fill it with schema definition at
least at the end of each cycle, or more often if required.

Tests can define which range or micro versions they are applicable to.
Initial tests for v3 in cinder will have no min_version, and they may get a
max version if they're behaviour is not valid anymore beyond a certain
microversion. Tests developed specifically for a micro versions will have
min_version set accordingly.

In terms of which versions we test in the gate, for nova we always run with
min_microversion = None and max_microversion = latest, which means that all
tests will be executed.
Since micro versions are incremental, and each micro version usually
involves no or one test on Tempest side, I think it will be a while before
this becomes an issue for the common gate.

andrea

[0] https://etherpad.openstack.org/p/qa-ptg-pike-microversion
[1] https://etherpad.openstack.org/p/qa-ptg-pike-schema-validation
[2] https://docs.openstack.org/developer/tempest/microversion_testing.html

> >
> > Anyway; I'm certain that between Nova and the API-WG this has come up
> and is
> > probably addressed, just wondering if somebody can point me to some
> > documentation or policies in this respect.
> >
> > Thanks,
> > John
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170310/050e1483/attachment.html>


More information about the OpenStack-dev mailing list