[openstack-dev] [qa] negative test generation and testscenarios
robertc at robertcollins.net
Tue Jan 7 02:59:46 UTC 2014
On 7 January 2014 07:41, David Kranz <dkranz at redhat.com> wrote:
> Thanks to all who looked at https://review.openstack.org/#/c/64733/. There
> were a few minor issues I will address but the biggest one was the
> suggestion to run each variation as a separate test case using
> testscenarios. After looking into that I see a problem with this use case.
> Many of these negative tests need to allocate resources such as servers that
> are then referenced in the test cases. Our tests currently do stuff like
> that in setUpClass. But testscenarios, unless I misunderstand the code,
> replicates the whole class which means the resource setup would also be
> duplicated for each test. I think we need to share allocated resources
> across the potentially large number of negative variations, and see two
> options for that:
You don't understand the code :). testscenarios doesn't affect the
class of tests. As long as the resulting suite is wrapped in a
setUpClass compatible TestSuite object, setUpClass should keep
> 1. Some magic similar to testscenarios but that creates test methods with
> varying parameters instead of test classes with varying attributes.
> Leave allocation in setUpClass.
-1 No different to testscenarios, just different to be different.
> 2. Using testscenarios as-is but move the allocation/cleanup to the module
> level (fixtures?)
> This would be a little complicated because a bunch of the setup stuff is
> defined as methods on the test base classes. Also, the classes would
> run in parallel.
Long term perhaps, not needed for this work.
More information about the OpenStack-dev