Open Stack

Please keep in mind that not everyone uses tox also ;)

Some of us use VM's instead of tox+venv and throwaway the VM's when we are done testing whatever change.

VM's seem more cloudy to me ;)

I personally like 'run_tests.sh' since its a easy one-stop to running tests, making me not worry about tox/testr/nose syntax (and how the backing test impl seems to change about once a year).

My 2 cents.

From: Joe Gordon <joe.gordon0 at gmail.com<mailto:joe.gordon0 at gmail.com>>
Reply-To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Monday, June 17, 2013 4:01 PM
To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Subject: [openstack-dev] The future of run_tests.sh

Hi All,

A patch to move the run_tests.sh script into oslo-incubator (https://review.openstack.org/#/c/32736/), has brought up the bigger question of what is the future of './run_tests.sh.'

This seems like a topic that directly affects all developers, so it sounded like it should be brought to the Mailing list.

Reasons to keep run_tests.sh:

* there is no possibility to run tests with testtools instead of testr. This feature allows us to use the debugger.
* in some projects tox doesn't use a testr wrapper to report progress while tests are running
* building the sdist can be slow (slow == noticeable)


Monty's Rant:

"So, I'm SUPER torn on this.

On the one hand, we use tox in the gate, and run_tests.sh hides that from people and then they get confused about why some change they made to run_tests.sh doesn't get reflected in the reality of the project.

On the other hand, what Victor says is true. We need to land a couple of changes to upstream tox that would allow it to not run develop each time - but we seem to have a hard time doing that.

On the third hand, we'd doing an INSANE amount of work to make working with testing in a venv "easy" and what we've wound up with is a situation where we have multiple competing incompatible ways of doing things.

Here's what I do: source .tox/py27/bin/activate testr run --parallel testr run --parallel testr run some-test testr run --failing deactivate

Sometimes, if I'm being fancy, I'll do: virtualenv foo foo/bin/pip install -r requirements.txt -r test-requirements.txt

I believe vish just installs the depends directly on his box and skips venvs altogether.

It turns out that testr is a powerful tool with a nice UI. Instead of putting wrappers around it, perhaps we should add support for our output stream thing we like to it upstream. We should also add support to it for dropping into a debugger directly, so that the testtools issue goes away. Then we should land a patch to upstream tox to get it to do setup.py develop instead of sdist+ install.

If we did that, we could have "run tox" be the simple answer for anyone who just wants to run whatever the gate runs, because that will always work. And if they want to do fancier things, they should learn about venvs and testr.

However, if we ARE going to keep a run_tests.sh file in our trees, we should certainly have a single copy and have it behave consistently.

Please note that some projects actually do just have run_tests.sh as a thin wrapper around tox."


What do you think? Should we keep run_tests as is and port to oslo, or should we revisit the role of run_tests.sh?


best,

Joe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130618/9b431216/attachment.html>