[openstack-qa] automatically skipping tests based on config

Attila Fazekas afazekas at redhat.com
Wed Feb 27 11:57:22 UTC 2013


Hi,

I agree we must stop skipping tests based on weather conditions or something like that.
(But, We can keep and/or extend the early failure logic.)

The other option would be using attributes or coding patterns to the testcase's name.
(nosetests -x ec2 # skips the ec2 tests for example)
We could skip tests based on path as well.

Any not skip exception in setUpClass is considered as an ERROR or FAIL (if the class has non zero test_ method).

We should do the skipping based on service endpoint expected to work or not, if we doing it on a configuration level.

For example if in a test run we do not expect a working glance,
 it leads to we cannot boot servers and cannot test ssh connection with newtwork tests,
 but we can create a keypair if the corresponding nova service is up.
 
Best Regards,
Attila


$ cat tests.py
import testtools as unittest
class test(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        raise Exception
    def test_foo():
        pass

$ nosetests tests.py || echo Gate test FAILED
E
======================================================================
ERROR: test suite for <class 'tests.test'>
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nose/suite.py", line 208, in run
    self.setUp()
  File "/usr/lib64/python2.7/site-packages/nose/suite.py", line 291, in setUp
    self.setupContext(ancestor)
  File "/usr/lib64/python2.7/site-packages/nose/suite.py", line 314, in setupContext
    try_run(context, names)
  File "/usr/lib64/python2.7/site-packages/nose/util.py", line 478, in try_run
    return func()
  File "/tmp/tests.py", line 5, in setUpClass
    raise Exception
Exception

----------------------------------------------------------------------
Ran 0 tests in 0.001s

FAILED (errors=1)
Gate test FAILED



----- Original Message -----
From: "Christopher Yeoh" <cyeoh at au1.ibm.com>
To: "All Things QA." <openstack-qa at lists.openstack.org>
Sent: Wednesday, February 27, 2013 12:18:37 PM
Subject: [openstack-qa] automatically skipping tests based on config

Hi,

For a few days now the tempest gating has been fairly quietly skipping
some EC2 tests. This has happened because of a change in devstack which
changed the version of the cirros image used. As a result ami/aki/ari
files defined in the generated tempest.conf did not match and the
setupClass for the tests noticed the missing files and automatically
skipped the tests as they could not run successfully. There's output in
the log files, but hey no one looks at the logs when the tests pass :-)

Now I'm all for an easy to configure and setup test suite so I
understand why it works this way but I'm wondering now that if we can
so easily get into the state of accidentally disabling tests that its
in fact not such a good feature to have.

I'm wondering instead if we should have some fields in the tempest
configuration file to allow people to skip tests - eg they have to
explicitly say that they do not want them to run. If the skip setting
for a specific test (or class of tests) is not set then we should fail
the test instead, perhaps with a warning message saying that it can be
set to skip the tempest config file. That way we won't ever get into the
situation of ever accidentally disabling tests.

I don't think we can fail a test or class of tests in setUpClass -
raising an exception at that point just results in both nose or testr
treating them as skips. Does anyone know of a way to do this?

Otherwise we'd have to implement it using a setUp or maybe a decorator
on each of the tests (a bit of a pain but not that intrusive). 

Any suggestions/comments?

Regards,

Chris
-- 
yeohc at au1.ibm.com


_______________________________________________
openstack-qa mailing list
openstack-qa at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa



More information about the openstack-qa mailing list