Open Stack

On 05/01/2013 10:19 PM, Russell Bryant wrote:
> On 05/01/2013 09:32 PM, Thomas Goirand wrote:
>> On 05/01/2013 11:27 PM, Chuck Short wrote:
>>> Hi,
>>>
>>> Please don't we use it extensively in the packaging that we do.
>>>
>>> regards
>>> chuck
>>
>> I agree with Chuck here. The run_tests.sh are convenient for package
>> maintainers, and I would like to have it stay as well.
> 
> We're obviously not talking about making it so you can't run tests.
> We're talking about the fact that we have 2 different methods of doing
> so.  Further, we need to identify if there's anything you can do with
> run_tests.sh that you can not do with tox.
> 
> Can you be a bit more specific about what run_tests.sh does for you
> that's special?

I think the key thing about run_tests.sh for the packagers is the "I can
run tests without creating a venv" Also, I believe that run_tests.sh
does a better job of reusing existing venvs in a slightly more efficient
manner. I think we can fix tox to handle the second thing (I can point
someone in the right direction if they would like to get it sorted - I
have a half-done patch on bitbucket)

The other thing is that, while we have a defined testing API via tox, we
do not have a defined API for what gets run inside of the venv. It's
getting better with the move to flake8 (no need for specific different
command lines) and we've already added a setuptools command for testr so
once we get everyone on testr, the tox->project api will be:

flake8
python setup.py testr

I could easily add a mapping in pbr (once we're all testr all the time)
to map python setup.py testr to python setup.py test so that the debian
packaging guys don't have to override what debhelper wants to run.

At that point - I think it'll be easy to reason about what additional
helper interfaces we might need.

Now mind you - that's the conservative approach. Each project today has
a command that can run its tests outside of a venv (just look at the
commands portion of the testenv section of its tox.ini file) - so we
have everything we need to replace run_tests.sh with README.tests and
for the packagers to patch their packaging to run the tests without
run_tests.sh. Virtualenvs themselves are fairly easy to make and
maintain, and once you do that, you can just directly run testr itself,
which has tons of cool features. So I can also support the "rip the
bandaid off" approach.

There's also a door number three which would be to take the code that's
in run_tests.sh/install_venv.py and put it in as an installable command
in pbr and figure out how to integrate that sensibly with tox. However,
that feels more like adding yet-another method rather than coalescing
the current methods into a sensible set.

Monty