Open Stack

So we've current got a problem in Tempest that's due to the 
incompatibility of 2 test tools that we want to use (flake8 & testr). 
This can best be exemplified in the following review - 
https://review.openstack.org/#/c/29293/ - which tries to fix the flake8 
issue, but will cause an issue in switching to testr.


Crux of the problem - Tempest needs to have a config file, and we need 
it parsed early
========

Unlike unit tests, Tempest is designed as a tool that you run against a 
living cloud. In the devstack-gate that's a cloud of our own creation, 
but in the general case tempest should be runnable against any cloud. 
That means you have to configure it to match your OpenStack cloud 
configuration, including credentials, images, flavors, and services that 
you may or may not have in your environment.

This is done in a tempest.conf file.

Under nose we had a setup_package hook, which let us do the config file 
parsing early, and once for the entire run. As we made tempest testr 
compatible, we had to stop using that hook, as it doesn't exist for 
testr. Instead this all went into __init__.py files.

This causes issues with flake8, because flake8 evals python code as part 
of it's scan (largely to check the import rules IIRC). So for anyone 
that's seen a tempest pep8 run since the flake8 transition will notice 
the following 
(http://logs.openstack.org/29395/4/check/gate-tempest-pep8/3408/console.html)

Config file /etc/tempest/tempest.conf not found
2013-05-21 17:03:43.195 | Traceback (most recent call last):
2013-05-21 17:03:43.196 |   File 
"/home/jenkins/workspace/gate-tempest-pep8/.tox/pep8/local/lib/python2.7/site-packages/hacking/core.py", 
line 184, in is_module_for_sure
2013-05-21 17:03:43.196 |     __import__(mod)
2013-05-21 17:03:43.196 |   File 
"/home/jenkins/workspace/gate-tempest-pep8/.tox/pep8/local/lib/python2.7/site-packages/tempest/api/compute/base.py", 
line 31, in <module>
2013-05-21 17:03:43.196 |     class 
BaseComputeTest(tempest.test.BaseTestCase):
2013-05-21 17:03:43.196 |   File 
"/home/jenkins/workspace/gate-tempest-pep8/.tox/pep8/local/lib/python2.7/site-packages/tempest/api/compute/base.py", 
line 34, in BaseComputeTest
2013-05-21 17:03:43.197 |     conclusion = compute.generic_setup_package()
2013-05-21 17:03:43.197 |   File 
"/home/jenkins/workspace/gate-tempest-pep8/.tox/pep8/local/lib/python2.7/site-packages/tempest/api/compute/__init__.py", 
line 42, in generic_setup_package
2013-05-21 17:03:43.197 |     os = clients.Manager()
2013-05-21 17:03:43.197 |   File 
"/home/jenkins/workspace/gate-tempest-pep8/.tox/pep8/local/lib/python2.7/site-packages/tempest/clients.py", 
line 279, in __init__
2013-05-21 17:03:43.197 |     self.token_client = 
TOKEN_CLIENT[interface](self.config)
2013-05-21 17:03:43.198 |   File 
"/home/jenkins/workspace/gate-tempest-pep8/.tox/pep8/local/lib/python2.7/site-packages/tempest/services/identity/json/identity_client.py", 
line 225, in __init__
2013-05-21 17:03:43.198 |     if 'tokens' not in auth_url:
2013-05-21 17:03:43.199 | TypeError: argument of type 'NoneType' is not 
iterable


We had to disable exiting if we couldn't find the config file to get 
flake8 in, which was the previous behavior.


Why do we want tempest.conf parsed early?
========
2 reasons.

1 - it's really a read only data parse, and moving it to class 
initialization ends up meaning we reparse the same config file many many 
times. Which is inefficient. It's still in the CI noise level though, so 
if we had to be inefficient, so be it.

2 - That's because we'd like the config early to be able to use config 
variables in skipIf conditions:

i.e. 
https://github.com/openstack/tempest/blob/master/tempest/api/compute/servers/test_server_actions.py#L150

     @attr(type='smoke')
     @testtools.skipIf(not resize_available, 'Resize not available.')
     def test_resize_server_confirm(self):
         # The server's RAM and disk space should be modified to that of
         # the provided flavor

         previous_flavor_ref, new_flavor_ref = \
             self._detect_server_image_flavor(self.server_id)

         resp, server = self.client.resize(self.server_id, new_flavor_ref)
         self.assertEqual(202, resp.status)
         self.client.wait_for_server_status(self.server_id, 'VERIFY_RESIZE')

         self.client.confirm_resize(self.server_id)
         self.client.wait_for_server_status(self.server_id, 'ACTIVE')

         resp, server = self.client.get_server(self.server_id)
         self.assertEqual(new_flavor_ref, int(server['flavor']['id']))

Which would be a bigger rewrite to work around a testr limitation.

So right now we're kind of caught between a rock and a hard place, as in 
the current state we can't satisfy both flake8 and testr without another 
chunk of rewrite. I'd really appreciate thoughts, especially from folks 
that have been on various parts of this as to how we get to a point of 
not stack tracking on flake8, and still being able to move forward on testr.

	-Sean

-- 
Sean Dague
http://dague.net