Open Stack

This is a bit of a funny case, where the methods themselves are uniquely identified,
but code is reused to test different functionality based on parameters passed
to the tests.

In this case, perhaps the choice of the name idempotent_id was a wise one, since
we have multiple code paths. My suggestion would be for in the short term to
use a combination of test name and id in the case where api tests result in a 
collision. Another option is to admit capabilities against ids and allow for passing
of a subset of the tests.

This is good knowledge to have, and we might need to consider the addition of
automatically generated secondary ids for the collision case that uses the
primary id as seed we can reproduce against.

-Chris 


> On Mar 4, 2015, at 10:50 AM, Matthew Treinish <mtreinish at kortar.org> wrote:
> 
> On Wed, Mar 04, 2015 at 10:41:45AM -0800, Catherine Cuong Diep wrote:
>> 
>> 
>> Examining the newly implemented test UUIDs, I found that there are 496 test
>> cases that have multiple test cases associated with same UUIDs.   These can
>> be categorized into 2 groups:
>> 
>>   A single UUID is associated to multiple test cases for different API
>>   versions.  Since we can only test one API version at a time, this will
>>   not be an issue for Refstack.  Example:
>>      tempest.api.volume.admin.test_volume_types_negative.VolumeTypesNegativeV2Test.test_create_with_nonexistent_volume_type
>> [gate,id-b48c98f2-e662-4885-9b71-032256906314]
>>      tempest.api.volume.admin.test_volume_types_negative.VolumeTypesNegativeV1Test.test_create_with_nonexistent_volume_type
>> [gate,id-b48c98f2-e662-4885-9b71-032256906314]
>> 
>>   A single UUID associates to multiple test cases for different features,
>>   protocols (IPv4 vs IPv6) ... In this case, it is very possible that we
>>   could invoke all tests on the same test run. Should these tests have
>>   different UUIDs? The issue is DefCore will define must-pass tests using
>>   test UUID.   In the example below since all 3 test cases have the same
>>   UUID, which would be the test cases that must pass? all 3? or just one
>>   of the 3?
>>      tempest.api.network.test_networks.NetworksTestJSON.test_list_networks_fields
>> [gate,id-6ae6d24f-9194-4869-9c85-c313cb20e080,smoke]
>>      tempest.api.network.test_networks.NetworksIpV6TestJSON.test_list_networks_fields
>> [gate,id-6ae6d24f-9194-4869-9c85-c313cb20e080,smoke]
>>      tempest.api.network.test_networks.NetworksIpV6TestAttrs.test_list_networks_fields
>> [gate,id-6ae6d24f-9194-4869-9c85-c313cb20e080,smoke]
>> 
>> A complete list of the 496 test cases can be found in
>> https://docs.google.com/spreadsheets/d/1mXeDNsJJJl93juzlNG9ccxSdO9ywqIeM-E2EJdvNImk/edit#gid=1273407686
> 
> This was a concious design decision for adding the ids to the tests. If the test
> method is the same and the class is different we treat it as a single test for
> tracking purposes. (a slight abuse of test class inheritance) Since it's testing
> the same functionality just either using a different api version, or I guess ipv4
> vs v6. (I'm not exactly sure what IPv6Attrs is vs IPv6, I'll have to check the
> code)
> 
> If you're planning to specify which tests get run by the ids, then you can use
> the tempest config file to determine which versions get run. For example, there's
> a config flag for whether ipv6 is available or not. The same is true for the api
> versions.
> 
> Test classes in tempest should be split on these feature boundaries which should
> have skip checks from the config file in place so things aren't executed if the
> feature isn't available. If there are cases in tempest where the split isn't as
> clear cut it might be a bug.
> 
> -Matt Treinish

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/fits/attachments/20150304/8894dd91/attachment-0001.html>