[openstack-qa] Multi API version testing

Matthew Treinish mtreinish at kortar.org
Thu Jun 20 15:54:33 UTC 2013

On Wed, Jun 19, 2013 at 07:38:17AM +0000, Frittoli, Andrea (Cloud Services) wrote:
> What is our strategy about testing combination of API versions?
> If we start testing all possible combinations of API the gate time will grow
> exponentially - e.g. all tests would be run using v2 and v3 auth, all tests
> using a VM would create it via nova v2 and v3, all tests using an image
> would fetch it via glance v1 and v2.
> My idea would be to:
> -          Test all available versions only in the dedicated tests. For
> instance identity tests shall verify keystone v2 and v3, compute tests shall
> verify nova v2 and v3.

So in the api tests this is done for the most part I believe. I wrote the glance
tests to be a framework for doing multiple api version tests. The glance tests
auto discover which version of the api are available and then test them
appropriately. The underlying assumption is that we have to assume nothing will
remain the same between API versions (because there isn't a guarantee of that)
so we have to test everything in the new version (even if it remains unchanged)
this is what I started to do for the glance testing, but there are probably still
gaps in what is being tested. I'm sure that there are other gaps in which versions
we test currently for the other apis too, so feel free to contribute to get us
closer to full coverage.

> -          In the gate each tests shall use only the latest stable API
> version from other components.  This means that all tests shall authenticate
> via keystone v2 as long v3 is completed and stable. All tests which require
> a VM will set it up via nova v2 as long as nova v3 is completed and stable.
> -          We could also have a non-gating job which uses the latest
> available API for each component in each tests, to prepare for the version
> switch in the gate.
> Whichever approach we go for, we need our tests and base classes to be
> designed to allow for easily switching between API versions for one of the
> APIs.
> Any given test should be executable with any chosen combination of API
> versions, so that we can select which combination we want to use in the
> gate.

So I don't think I follow this completely. With keystone being the exception
(because it's used everywhere) do we have any tests that make straight api
calls to another service? I know, for example, all the compute api tests for
volumes and images are just using the nova api and api extensions. (which
then use official python clients inside of nova) Then for things like the
scenario tests we also use the python clients, and for most of the clients
I don't think there is multiple api version support. So unless that gets
implemented I think those are fine. But, if multiple version support becomes
available for the clients we probably should revisit this for scenario and
non-api testing.

I do agree though if we use a straight api call from another service for any
part of any test(not through an official client) it has to be able to support
multiple versions(if there are multiple versions). I don't think that for keystone
we aren't there yet, but I haven't looked at that code too closely so I'm not
sure. So we definitely have a todo there, but I just don't think that this
applies anywhere else yet.

I also agree that once we get to a point where multiple version support is
configurable for tests that we should have a separate gate job that runs with it.
But, I feel it should also be a gating job especially if the API is supported.

-Matt Treinish

More information about the openstack-qa mailing list