[openstack-qa] scenario vs api overlap

Tal Kammer tkammer at redhat.com
Mon Aug 19 15:40:59 UTC 2013


Thanks David, Matthew, I will send future e-mails to openstack-dev with the [QA] tag in the subject.

Thanks,
Tal

----- Original Message -----
From: "Matthew Treinish" <mtreinish at kortar.org>
To: "All Things QA." <openstack-qa at lists.openstack.org>
Sent: Monday, August 19, 2013 5:47:40 PM
Subject: Re: [openstack-qa] scenario vs api overlap

Hi Tal,

One quick thing is we are trying to move away from using the dedicated QA list.
So for emails can you send this to the openstack-dev list with a [QA] tag. 

On Mon, Aug 19, 2013 at 09:49:57AM -0400, Tal Kammer wrote:
> Thanks giulio, though I'm not sure the scenarios are the right place to check the python clients as it acts more as a mean to an end rather than the subject of testing, so it might not get the necessary coverage.

The scenario tests are not designed to provide full coverage for the official
clients. It is just a side effect of the tests being written using the official
clients. The reason that the scenario tests use the official python clients is
because we are trying to test common usage scenarios, which in most cases will
use the python libraries. They are not dedicated client tests.

 
> ----- Original Message -----
> From: "Giulio Fidente" <gfidente at redhat.com>
> To: openstack-qa at lists.openstack.org
> Sent: Monday, August 19, 2013 1:08:13 PM
> Subject: Re: [openstack-qa] scenario vs api overlap
> 
> On 08/19/2013 10:28 AM, Tal Kammer wrote:
> > I wanted to start building some scenario based testing and noticed that there are some overlap of code where some scenarios re-implement code that is also present in the API.
> > For example, some of the scenarios (test_server_basic_ops.py for example) re-implement methods that were written in the corresponding REST client in the API.
> > I was wondering if there is some guidelines of what is a resalable overlap of code (though I'm not sure it really is an overlap when scenario uses the python clients and the API the REST client), for example, say I want to write a scenario for nova that tests something with the quota, I can see from the code of the API (tempest/tempest/api/compute/admin/test_quotas.py) that it has a lot of the quota related code already implemented with the REST client, so when I'm building a scenario, how much "freedom" do I have to write methods that uses the python clients instead of the API methods?

Because we are using the python clients the code pretty much has to be
duplicated, there are too many differences between the tempest client and the
official clients. Take a look at tempest.common.isolated_creds that was my
attempt at common code between the tempest keystone client and
python-keystoneclient. You can see all the differences between the 2 clients.
I will recommend that if you are writing methods that can be common that you
do that and put it in the base test class instead of in the individual test
file. The scenario tests have too much duplication between tests right now
in my opinion.

> 
> there is an attempt to describe the scope of the scenario tests in the 
> doc[1]
> 
> my understanding is that scenario tests, by using the official API 
> clients (and not the tempest clients), will be a tool for testing the 
> official clients too and not only the target services, so some code 
> duplication in the test cases is acceptable
> 
> in addition to that, scenario tests should check for integration between 
> projects and try to reproduce some sequence of steps possibly performed 
> by end users so, in contrast with an API suite which tries to test 
> extensively all the quota calls (eventually with negative tests too), a 
> scenario test should probably just create some resource until a quota 
> enforcement is hit and then adjust the quota setting and maybe try again 
> the initial operation to ensure it succeeds
> 
> 1. http://docs.openstack.org/developer/tempest/field_guide/scenario.html
> -- 
> Giulio Fidente
> GPG KEY: 08D733BA | IRC: giulivo
> 
> _______________________________________________
> openstack-qa mailing list
> openstack-qa at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
> 
> _______________________________________________
> openstack-qa mailing list
> openstack-qa at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa

_______________________________________________
openstack-qa mailing list
openstack-qa at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa



More information about the openstack-qa mailing list