Open Stack

I agree, preference should be to the function level skip vs. the class
exception. Especially as I have some sample fixture code from Robert
that will remove setUpClass in the future (we do have a long term goal
of getting rid of it in favor of proper fixtures). It also gives us an
actual count of the number of tests skipped in the configuration, which
is nice.

	-Sean

On 12/04/2013 06:46 AM, Daisuke Morita wrote:
> Hi, everyone.
> 
> Which do you think is the best way of coding test skipping, "writing
> cls.skipException statement in setUpClass method" or "skipIf annotation
> for each test method" ?
> 
> This question comes to me in reviewing
> https://review.openstack.org/#/c/59759/ . I think that work itself is
> great and I hope this patch is merged to Tempest. I just want to focus
> on coding styles and explicitness of test outputs.
> 
> If skipIf annotation is used, test output of Swift is as follows.
> 
> ---
> tempest.api.object_storage.test_account_quotas.AccountQuotasTest
>     test_admin_modify_quota[gate,smoke]
> SKIP  1.15
>     test_upload_large_object[gate,negative,smoke]
> SKIP  0.03
>     test_upload_valid_object[gate,smoke]
> SKIP  0.03
>     test_user_modify_quota[gate,negative,smoke]
> SKIP  0.03
> tempest.api.object_storage.test_account_services.AccountTest
>     test_create_and_delete_account_metadata[gate,smoke]               OK
>  0.32
>     test_list_account_metadata[gate,smoke]                            OK
>  0.02
>     test_list_containers[gate,smoke]                                  OK
>  0.02
> 
> ...(SKIP)...
> 
> Ran 54 tests in 85.977s
> 
> OK
> ---
> 
> 
> On the other hand, if cls.skipException is used, an output is changed as
> follows.
> 
> ---
> setUpClass (tempest.api.object_storage.test_account_quotas
>     AccountQuotasTest)
> SKIP  0.00
> tempest.api.object_storage.test_account_services.AccountTest
>     test_create_and_delete_account_metadata[gate,smoke]               OK
>  0.48
>     test_list_account_metadata[gate,smoke]                            OK
>  0.02
>     test_list_containers[gate,smoke]                                  OK
>  0.02
> 
> ...(SKIP)...
> 
> Ran 49 tests in 81.475s
> 
> OK
> ---
> 
> 
> I believe the output of the code using skipIf annotation is better.
> Since the coverage of tests is displayed more definitely, it is easier
> to find out what tests are really skipped.
> 
> I scanned the whole code of Tempest. The count of cls.skipException
> statements is 63, and the count of skipIf annotations is 24. Replacing
> them is not trivial task, but I think the most impportant for testing is
> to output consistent and accurate log.
> 
> 
> Am I missing something? Or, this kind of discussion has been done
> already in the past? If so, could you let me know?
> 
> 
> Best Regards,
> 


-- 
Sean Dague
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131204/e6942c6a/attachment.pgp>