Open Stack

On Mon, 2012-12-24 at 09:47 -0800, James E. Blair wrote:

> I looked into this, and I believe the problem is that in this file:
> 
> https://github.com/openstack/keystone/blob/stable/folsom/tests/test_keystoneclient.py
> 
> It checks out python-keystoneclient git repos from gerrit into the
> keystone workspace and attempts tests with different branches.  The
> problem is that different versions of python-keystoneclient will have
> different dependencies, and since they are not being installed properly
> with setuptools, those dependencies will not be in the environment.  In
> this case, 'requests', which is a current dependency of keystoneclient,
> is not available.
> 
> In general, I don't think _inside_ the test cases is a good place to be
> installing software.  This might be more appropriate as a set of jenkins
> jobs and tox configs that install the needed dependencies before running
> tests.

Well now, that's pretty interesting ... it's actually fairly amusing
this hasn't reared its head before :)

Honestly, the checking out of different versions of keystoneclient,
reloading the module and testing against the keystone server is a pretty
insane thing to do in unit tests ... but I have to applaud the effort at
good test coverage. I'm guessing a big part of the rationale here was to
make sure that the pre-keystonelight client worked with newer keystone.

Why insane? Well, for one thing, as a packager I would expect to be able
to run the unit tests during my package build process without any
network access being involved. I'd also find it pretty odd that the
keystone tests are testing keystoneclient, nevermind that they're
testing different versions of keystoneclient.

I guess there are a bunch of different ways to approach this:

  - As James suggests, keep the tests in keystone but only use the 
    currently installed version of keystoneclient. Add different tox 
    configs for the different versions of keystoneclient we want to 
    test and set up jenkins jobs for each. This would mean the tests 
    would need to be able to detect which version of keystoneclient is 
    installed because not all tests are applicable to each version. It 
    would also be nice if the jobs to test older versions just ran the 
    keystoneclient tests, not all the rest of the unit tests.

  - Think of these as API compatibility tests, record the API requests 
    produced by older versions of keystoneclient, check them into git 
    and have the tests check that these older API requests still work 
    correctly.

  - Think of these as integration tests for keystoneclient, move them 
    out of keystone and have different jenkins jobs for each version of 
    keystoneclient we want to test.

None of these are quick fixes, but IMHO we need an alternative to the
current approach.

To fix the stable/folsom issues, how about we say that the stable/folsom
tests only need to test the 0.2.0 version of keystoneclient, not master?

That's as simple as this change:

  @@ -789,7 +789,7 @@ class KeystoneClientTests(object):
 
   class KcMasterTestCase(CompatTestCase, KeystoneClientTests):
       def get_checkout(self):
  -        return KEYSTONECLIENT_REPO, 'master'
  +        return KEYSTONECLIENT_REPO, '0.2.0'
 
       def test_tenant_add_and_remove_user(self):
           client = self.get_client(admin=True)

Cheers,
Mark.