Open Stack

On 02/27/2013 06:18 AM, Christopher Yeoh wrote:
> Hi,
>
> For a few days now the tempest gating has been fairly quietly skipping
> some EC2 tests. This has happened because of a change in devstack which
> changed the version of the cirros image used. As a result ami/aki/ari
> files defined in the generated tempest.conf did not match and the
> setupClass for the tests noticed the missing files and automatically
> skipped the tests as they could not run successfully. There's output in
> the log files, but hey no one looks at the logs when the tests pass :-)
>
> Now I'm all for an easy to configure and setup test suite so I
> understand why it works this way but I'm wondering now that if we can
> so easily get into the state of accidentally disabling tests that its
> in fact not such a good feature to have.
>
> I'm wondering instead if we should have some fields in the tempest
> configuration file to allow people to skip tests - eg they have to
> explicitly say that they do not want them to run. If the skip setting
> for a specific test (or class of tests) is not set then we should fail
> the test instead, perhaps with a warning message saying that it can be
> set to skip the tempest config file. That way we won't ever get into the
> situation of ever accidentally disabling tests.
>
> I don't think we can fail a test or class of tests in setUpClass -
> raising an exception at that point just results in both nose or testr
> treating them as skips. Does anyone know of a way to do this?

Really, I thought throwing an exception in setUpClass makes it an error, 
not a skip. Skip is a very specific type of exception.

> Otherwise we'd have to implement it using a setUp or maybe a decorator
> on each of the tests (a bit of a pain but not that intrusive).
>
> Any suggestions/comments?

If we didn't quietly skip, it would have blocked the devstack change, 
which is what we want. If a tempest config file variable is there, but 
not right, it should be a fail, not a skip. Would that have fixed this?

	-Sean

-- 
Sean Dague
IBM Linux Technology Center
email: sdague at linux.vnet.ibm.com
alt-email: sldague at us.ibm.com