Open Stack

On Mon, Jun 17, 2013 at 6:02 PM, Henry Gessau <gessau at cisco.com> wrote:

> On Mon, Jun 17, at 7:01 pm, Joe Gordon <joe.gordon0 at gmail.com> wrote:
> > Hi All,
> >
> > A patch to move the run_tests.sh script into oslo-incubator
> > (https://review.openstack.org/#/c/32736/), has brought up the bigger
> > question of what is the future of './run_tests.sh.'
> >
> > This seems like a topic that directly affects all developers, so it
> > sounded like it should be brought to the Mailing list.
> >
> > Reasons to keep run_tests.sh:
> >
> > * there is no possibility to run tests with testtools instead of testr.
> > This feature allows us to use the debugger.
> > * in some projects tox doesn't use a testr wrapper to report progress
> > while tests are running
> > * building the sdist can be slow (slow == noticeable)
> >
> >
> > Monty's Rant:
> >
> > "So, I'm SUPER torn on this.
> >
> > On the one hand, we use tox in the gate, and run_tests.sh hides that from
> > people and then they get confused about why some change they made to
> > run_tests.sh doesn't get reflected in the reality of the project.
> >
> > On the other hand, what Victor says is true. We need to land a couple of
> > changes to upstream tox that would allow it to not run develop each time
> -
> > but we seem to have a hard time doing that.
> >
> > On the third hand, we'd doing an INSANE amount of work to make working
> > with testing in a venv "easy" and what we've wound up with is a situation
> > where we have multiple competing incompatible ways of doing things.
> >
> > Here's what I do: source .tox/py27/bin/activate testr run --parallel
> testr
> > run --parallel testr run some-test testr run --failing deactivate
> >
> > Sometimes, if I'm being fancy, I'll do: virtualenv foo foo/bin/pip
> install
> > -r requirements.txt -r test-requirements.txt
> >
> > I believe vish just installs the depends directly on his box and skips
> > venvs altogether.
> >
> > It turns out that testr is a powerful tool with a nice UI. Instead of
> > putting wrappers around it, perhaps we should add support for our output
> > stream thing we like to it upstream. We should also add support to it for
> > dropping into a debugger directly, so that the testtools issue goes away.
> > Then we should land a patch to upstream tox to get it to do setup.py
> > develop instead of sdist+ install.
> >
> > If we did that, we could have "run tox" be the simple answer for anyone
> > who just wants to run whatever the gate runs, because that will always
> > work. And if they want to do fancier things, they should learn about
> venvs
> > and testr.
> >
> > However, if we ARE going to keep a run_tests.sh file in our trees, we
> > should certainly have a single copy and have it behave consistently.
> >
> > Please note that some projects actually do just have run_tests.sh as a
> > thin wrapper around tox."
> >
> >
> > What do you think? Should we keep run_tests as is and port to oslo, or
> > should we revisit the role of run_tests.sh?
> >
> >
> My vote is for run_tests.sh to contain just one line, "cat TESTING", and
> ensure the instructions are detailed and up to date. If course, it would be
> nice if all the projects gravitated towards the same TESTING.
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

Post conversation with Lifeless and Clarkb on IRC, I'll change my answer
slightly:

As long as I can do the following:

1. Run individual tests easily
2. Drop into PDB by setting a break-point
3. Run WITHOUT virtual env
4. Not have to download a bunch of stuff or run some monster init every
time I run tests on a fresh clone of a repo (see item 3)


I don't care if that's done via a wrapper script that calls a framework
that supports the option (well I do care but we're already there so I'd
prefer not to take it away), better documentation of how to run tests and
how the frameworks operate or whatever.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130617/d045a2c1/attachment.html>