[openstack-qa] Multi API version testing

Frittoli, Andrea (Cloud Services) frittoli at hp.com
Fri Jun 21 08:39:28 UTC 2013


> -----Original Message-----
> From: Matthew Treinish [mailto:mtreinish at kortar.org] 
> Sent: 20 June 2013 16:55
> To: All Things QA.
> Subject: Re: [openstack-qa] Multi API version testing
> 
> On Wed, Jun 19, 2013 at 07:38:17AM +0000, Frittoli, Andrea (Cloud
Services) wrote:
> > What is our strategy about testing combination of API versions?
> > 
> >  
> > 
> > If we start testing all possible combinations of API the gate time 
> > will grow exponentially - e.g. all tests would be run using v2 and v3 
> > auth, all tests using a VM would create it via nova v2 and v3, all 
> > tests using an image would fetch it via glance v1 and v2.
> > 
> >  
> > 
> > My idea would be to:
> > 
> > -          Test all available versions only in the dedicated tests. For
> > instance identity tests shall verify keystone v2 and v3, compute tests 
> > shall verify nova v2 and v3.
> 
> So in the api tests this is done for the most part I believe. I wrote the
glance tests to be a framework for doing multiple api version tests. The
glance tests auto discover which version of the api are available and then
test them appropriately. The underlying assumption is that we have to assume
nothing will remain the same between API versions (because there isn't a
guarantee of that) so we have to test everything in the new version (even if
it remains unchanged) this is what I started to do for the glance testing,
but there are probably still gaps in what is being tested. I'm sure that
there are other gaps in which versions we test currently for the other apis
too, so feel free to contribute to get us closer to full coverage.

+1 we need to retest the entire API set in each version

> 
> > 
> > -          In the gate each tests shall use only the latest stable API
> > version from other components.  This means that all tests shall 
> > authenticate via keystone v2 as long v3 is completed and stable. All 
> > tests which require a VM will set it up via nova v2 as long as nova v3
is completed and stable.
> > 
> > -          We could also have a non-gating job which uses the latest
> > available API for each component in each tests, to prepare for the 
> > version switch in the gate.
> > 
> >  
> > 
> > Whichever approach we go for, we need our tests and base classes to be 
> > designed to allow for easily switching between API versions for one of 
> > the APIs.
> > 
> > Any given test should be executable with any chosen combination of API 
> > versions, so that we can select which combination we want to use in 
> > the gate.
> 
> So I don't think I follow this completely. With keystone being the
exception (because it's used everywhere) do we have any tests that make
straight api calls to another service? I know, for example, all the compute
api tests for volumes and images are just using the nova api and api
extensions. (which then use official python clients inside of nova) Then for
things like the scenario tests we also use the python clients, and for most
of the clients I don't think there is multiple api version support. So
unless that gets implemented I think those are fine. But, if multiple
version support becomes available for the clients we probably should revisit
this for scenario and non-api testing.
> 
> I do agree though if we use a straight api call from another service for
any part of any test(not through an official client) it has to be able to
support multiple versions(if there are multiple versions). I don't think
that for keystone we aren't there yet, but I haven't looked at that code too
closely so I'm not sure. So we definitely have a todo there, but I just
don't think that this applies anywhere else yet.

Fair point, really the major use case is keystone. If we can implement that,
we should be able to reuse the same concept for other APIs as well if
needed. 
For instance if a volume tests requires a virtual machine, it probably does
not matter much through which API version the VM is created, but still it
would be good to be able to run the test with support from any compute api
version.

The current approach is the code is two have additional client instances
added to the manager using the keystone v3 api, e.g. from
https://github.com/openstack/tempest/blob/master/tempest/clients.py 

            if client_args_v3_auth:
                self.servers_client_v3_auth = SERVERS_CLIENTS[interface](
                    *client_args_v3_auth)
            else:
                self.servers_client_v3_auth = None

The client itself support working with both keystone versions, as
authentication is ultimately delegated to the rest_client.py.

I can  file a blueprint to provide support for switching auth version used
by all tests. It should quite small, most of the logic is in place already.

> 
> I also agree that once we get to a point where multiple version support is
configurable for tests that we should have a separate gate job that runs
with it.
> But, I feel it should also be a gating job especially if the API is
supported.

Perhaps we can run multiple jobs in parallel, each using a different version
of the API?

>  
> 
> -Matt Treinish
> 
> _______________________________________________
> openstack-qa mailing list
> openstack-qa at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6202 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-qa/attachments/20130621/83aff2d8/attachment.bin>


More information about the openstack-qa mailing list