Open Stack

Excerpts from Sascha Peilicke's message of 2013-06-18 00:48:23 -0700:
> On 06/18/2013 01:01 AM, Joe Gordon wrote:
> > Hi All,
> >
> > A patch to move the run_tests.sh script into oslo-incubator
> > (https://review.openstack.org/#/c/32736/), has brought up the bigger
> > question of what is the future of './run_tests.sh.'
> >
> > This seems like a topic that directly affects all developers, so it
> > sounded like it should be brought to the Mailing list.
> >
> > Reasons to keep run_tests.sh:
> >
> > * there is no possibility to run tests with testtools instead of testr.
> > This feature allows us to use the debugger.
> > * in some projects tox doesn't use a testr wrapper to report progress
> > while tests are running
> > * building the sdist can be slow (slow == noticeable)
> >
> >
> > Monty's Rant:
> >
> > "So, I'm SUPER torn on this.
> >
> > On the one hand, we use tox in the gate, and run_tests.sh hides that
> > from people and then they get confused about why some change they made
> > to run_tests.sh doesn't get reflected in the reality of the project.
> >
> > On the other hand, what Victor says is true. We need to land a couple of
> > changes to upstream tox that would allow it to not run develop each time
> > - but we seem to have a hard time doing that.
> >
> > On the third hand, we'd doing an INSANE amount of work to make working
> > with testing in a venv "easy" and what we've wound up with is a
> > situation where we have multiple competing incompatible ways of doing
> > things.
> >
> > Here's what I do: source .tox/py27/bin/activate testr run --parallel
> > testr run --parallel testr run some-test testr run --failing deactivate
> >
> > Sometimes, if I'm being fancy, I'll do: virtualenv foo foo/bin/pip
> > install -r requirements.txt -r test-requirements.txt
> >
> > I believe vish just installs the depends directly on his box and skips
> > venvs altogether.
> 
> It's not only vish but basically how distros test their packages, 
> actually :-)
> 
> > It turns out that testr is a powerful tool with a nice UI. Instead of
> > putting wrappers around it, perhaps we should add support for our output
> > stream thing we like to it upstream. We should also add support to it
> > for dropping into a debugger directly, so that the testtools issue goes
> > away. Then we should land a patch to upstream tox to get it to do
> > setup.py develop instead of sdist+ install.
> >
> > If we did that, we could have "run tox" be the simple answer for anyone
> > who just wants to run whatever the gate runs, because that will always
> > work. And if they want to do fancier things, they should learn about
> > venvs and testr.
> 
> "run tox" would work for distros too, they only have to patch tox.ini with:
> 
> [testenv]
> sitepackages = True

Sounds to me like tox needs a switch that runs the tests without the
virtualenv. This can be done right now by running tox --showconfig and
parsing that to get the commands, but something like tox --no-venv should
be easy to do and useful to all.

> 
> to avoid
> 
> >
> > However, if we ARE going to keep a run_tests.sh file in our trees, we
> > should certainly have a single copy and have it behave consistently.
> >
> > Please note that some projects actually do just have run_tests.sh as a
> > thin wrapper around tox."
> >
> >
> > What do you think? Should we keep run_tests as is and port to oslo, or
> > should we revisit the role of run_tests.sh?
> 
> I don't think the demands for testing on the gate are necessarily the 
> same for developers and/or packagers. For gating, we want fast, i.e. 
> parallel testing. Developers want PDB and running individual tests. 
> That's why nosetests is and remains so popular. However, packagers hate 
> everything that messes up their build environment. This includes stuff 
> that downloads from the interwebs or ignores the underlying systems. 
> Thus virtualenvs, setuptools_git, sphinx.ext.intersphinx & co. are a no 
> go and usually patched away. "./run_tests.sh -N -P" does a good job but 
> "nostests" is equally suitable for package testing. By contrast, tox 
> still creates a venv with "sitepackages = True", which is overhead for 
> no gain. When you test distro packages, you don't need several testenvs, 
> test against the Python interpreter your packages where build against 
> and don't care for PEP-8.
> 

To be clear, tox runs individual tests just fine, simply pass in a module
to run. Agreed that it takes quite a bit of digging to get to a point
where you can use pdb.

Trying to be everything to everyone is almost never a good idea. However,
what is a good idea is loosely integrating and not repeating ourselves.
.testr.conf and tox.ini have all of the info necessary to run tests. If
run_tests.sh does need to stay, I would propose that it parse these
files as much as possible so that when the gate changes, so does the way
developers run tests.