Open Stack

Hi,

For a few days now the tempest gating has been fairly quietly skipping
some EC2 tests. This has happened because of a change in devstack which
changed the version of the cirros image used. As a result ami/aki/ari
files defined in the generated tempest.conf did not match and the
setupClass for the tests noticed the missing files and automatically
skipped the tests as they could not run successfully. There's output in
the log files, but hey no one looks at the logs when the tests pass :-)

Now I'm all for an easy to configure and setup test suite so I
understand why it works this way but I'm wondering now that if we can
so easily get into the state of accidentally disabling tests that its
in fact not such a good feature to have.

I'm wondering instead if we should have some fields in the tempest
configuration file to allow people to skip tests - eg they have to
explicitly say that they do not want them to run. If the skip setting
for a specific test (or class of tests) is not set then we should fail
the test instead, perhaps with a warning message saying that it can be
set to skip the tempest config file. That way we won't ever get into the
situation of ever accidentally disabling tests.

I don't think we can fail a test or class of tests in setUpClass -
raising an exception at that point just results in both nose or testr
treating them as skips. Does anyone know of a way to do this?

Otherwise we'd have to implement it using a setUp or maybe a decorator
on each of the tests (a bit of a pain but not that intrusive). 

Any suggestions/comments?

Regards,

Chris
-- 
yeohc at au1.ibm.com