Open Stack

On 06/18/2013 01:01 AM, Joe Gordon wrote:
> Hi All,
>
> A patch to move the run_tests.sh script into oslo-incubator
> (https://review.openstack.org/#/c/32736/), has brought up the bigger
> question of what is the future of './run_tests.sh.'
>
> This seems like a topic that directly affects all developers, so it
> sounded like it should be brought to the Mailing list.
>
> Reasons to keep run_tests.sh:
>
> * there is no possibility to run tests with testtools instead of testr.
> This feature allows us to use the debugger.
> * in some projects tox doesn't use a testr wrapper to report progress
> while tests are running
> * building the sdist can be slow (slow == noticeable)
>
>
> Monty's Rant:
>
> "So, I'm SUPER torn on this.
>
> On the one hand, we use tox in the gate, and run_tests.sh hides that
> from people and then they get confused about why some change they made
> to run_tests.sh doesn't get reflected in the reality of the project.
>
> On the other hand, what Victor says is true. We need to land a couple of
> changes to upstream tox that would allow it to not run develop each time
> - but we seem to have a hard time doing that.
>
> On the third hand, we'd doing an INSANE amount of work to make working
> with testing in a venv "easy" and what we've wound up with is a
> situation where we have multiple competing incompatible ways of doing
> things.
>
> Here's what I do: source .tox/py27/bin/activate testr run --parallel
> testr run --parallel testr run some-test testr run --failing deactivate
>
> Sometimes, if I'm being fancy, I'll do: virtualenv foo foo/bin/pip
> install -r requirements.txt -r test-requirements.txt
>
> I believe vish just installs the depends directly on his box and skips
> venvs altogether.

It's not only vish but basically how distros test their packages, 
actually :-)

> It turns out that testr is a powerful tool with a nice UI. Instead of
> putting wrappers around it, perhaps we should add support for our output
> stream thing we like to it upstream. We should also add support to it
> for dropping into a debugger directly, so that the testtools issue goes
> away. Then we should land a patch to upstream tox to get it to do
> setup.py develop instead of sdist+ install.
>
> If we did that, we could have "run tox" be the simple answer for anyone
> who just wants to run whatever the gate runs, because that will always
> work. And if they want to do fancier things, they should learn about
> venvs and testr.

"run tox" would work for distros too, they only have to patch tox.ini with:

[testenv]
sitepackages = True

to avoid

>
> However, if we ARE going to keep a run_tests.sh file in our trees, we
> should certainly have a single copy and have it behave consistently.
>
> Please note that some projects actually do just have run_tests.sh as a
> thin wrapper around tox."
>
>
> What do you think? Should we keep run_tests as is and port to oslo, or
> should we revisit the role of run_tests.sh?

I don't think the demands for testing on the gate are necessarily the 
same for developers and/or packagers. For gating, we want fast, i.e. 
parallel testing. Developers want PDB and running individual tests. 
That's why nosetests is and remains so popular. However, packagers hate 
everything that messes up their build environment. This includes stuff 
that downloads from the interwebs or ignores the underlying systems. 
Thus virtualenvs, setuptools_git, sphinx.ext.intersphinx & co. are a no 
go and usually patched away. "./run_tests.sh -N -P" does a good job but 
"nostests" is equally suitable for package testing. By contrast, tox 
still creates a venv with "sitepackages = True", which is overhead for 
no gain. When you test distro packages, you don't need several testenvs, 
test against the Python interpreter your packages where build against 
and don't care for PEP-8.

So, as long as we make sure all popular test runners can be used, I 
don't see any issue in changing defaults.
-- 
Sascha Peilicke
SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer HRB 16746 (AG Nürnberg)