Open Stack

Hi Robert,

https://review.openstack.org/#/c/64733 is updated and is using testscenarios
now. I was wondering if it would make sense to use a decorator instead of
a class attribute to mark a test case for a certain test scenario, like:

    @TestCaseWithScenario(_scenarios1)
    def test_demo(self):
    ...
    @TestCaseWithScenario(_scenarios2)
    def test_demo2(self):


I already implement something that seems to work with subunit:

http://paste.openstack.org/show/61175/

I think this would ease up the usage of testscenarios for the negative test
framework in tempest and looks quite cool.

Feedback is welcome!

Regards
Marc

________________________________________
From: Robert Collins [robertc at robertcollins.net]
Sent: Tuesday, January 07, 2014 3:59 AM
To: David Kranz
Cc: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [qa] negative test generation and testscenarios

On 7 January 2014 07:41, David Kranz <dkranz at redhat.com> wrote:
> Thanks to all who looked at https://review.openstack.org/#/c/64733/. There
> were a few minor issues I will address but the biggest one was the
> suggestion to run each variation as a separate test case using
> testscenarios. After looking into that I see a problem with this use case.
> Many of these negative tests need to allocate resources such as servers that
> are then referenced in the test cases. Our tests currently do stuff like
> that in setUpClass. But testscenarios, unless I misunderstand the code,
> replicates the whole class which means the resource setup would also be
> duplicated for each test. I think we need to share allocated resources
> across the potentially large number of negative variations, and see two
> options for that:

You don't understand the code :). testscenarios doesn't affect the
class of tests. As long as the resulting suite is wrapped in a
setUpClass compatible TestSuite object, setUpClass should keep
working.

> 1. Some magic similar to testscenarios but that creates test methods with
> varying parameters instead of test classes with varying attributes.
>     Leave allocation in setUpClass.

-1 No different to testscenarios, just different to be different.

> 2. Using testscenarios as-is but move the allocation/cleanup to the module
> level (fixtures?)
>     This would be a little complicated because a bunch of the setup stuff is
> defined as methods on the test base classes. Also, the classes would
>     run in parallel.

Long term perhaps, not needed for this work.

-Rob

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev