[openstack-dev] [testing] moving testrepository *outside* the tox venv

Robert Collins robertc at robertcollins.net
Sat Jul 11 10:04:54 UTC 2015

On 11 July 2015 at 12:38, Matthew Treinish <mtreinish at kortar.org> wrote:
> On Sat, Jul 11, 2015 at 11:35:16AM +1200, Robert Collins wrote:
>> On 9 July 2015 at 10:52, Jeremy Stanley <fungi at yuggoth.org> wrote:
>> > On 2015-07-09 10:37:17 +1200 (+1200), Robert Collins wrote:
>> >> So - I'm looking to:
>> >>
>> >> A) have a discussion and identify any issues with moving testr out of
>> >> the venvs. (Note: this doesn't mean stop using it, just removing it
>> >> from test-requirements.txt, in the same way that tox isn't in
>> >> test-requirements.txt).
>> >> B) Capture that in a spec if its non-trivial.
>> >> C) find volunteers to make it happen.
>> >
>> > D) keep reminding developers to install it on their systems when
>> > they ask why they can't run tests
>> We can make the os-testr which *is* installed in the venv look for
>> testr in the PATH and error with a good explanation when its not
>> there.
> I guess we could do this, it would be very simple to implement and I'm not
> necessarily opposed to it, if that's the direction everyone wants to go. Right
> now ostestr only interacts with testr through subprocess, which makes the
> implementation of something like this quite simple.
> But, I have 2 concerns right now: First was that my medium/long term my plan was
> to use testr's ui layer to implement ostestr's cli instead of the dirty
> subprocess hack I did to get things done faster. Doing this would require either
> breaking out of the venv to get the testr imports (which I wouldn't really want
> to do), require all the tox venv use system site-packages (which I don't think
> is really a good idea, and wouldn't solve the python version issue) or making
> sure testr was installed in the venv too.

Or moving os-testr out of the venv as well.

> The other is ostestr isn't universally used right now, only a few projects are
> using the ostestr CLI. os-testr is only really being installed in the venv
> everywhere because projects use subunit-trace, which is also part of os-testr,
> via a pretty_tox.sh script. So to make this a solution we would have to migrate
> everything over to ostestr first. Which wouldn't really be an issue and it's
> something I want to do eventually.
> TBH, I tend to agree with everyone else's opinion elsewhere on this thread.
> While it might be viewed as an anti-pattern for how testr was originally
> intended to be used it doesn't change that this is pretty much the model used
> and expected by everything and everyone using testr (or any test runner) in
> OpenStack. I would view doing something like this as more of a workaround than
> just fixing the storage engine to be compatible with multiple python versions.

Wearing my upstream hat, testr is *still* intended to be used
differently than OpenStack is doing. Running all the tests for all
python versions at once in parallel is the sort of thing testr is
aimed at, and thats fairly fundamentally incompatible with running
from inside a venv. As is distributed testing across multiple
machines. I haven't had time to help OpenStack mature its use of testr
until recently, and fixing these issues that are and will have seems
pretty important to me.

> The whole argument for making testr live outside of the venv and being an
> implicit dependency like tox is based around tracking the results between the
> tox venvs right? If we decoupled the database and other result storage from
> testr a bit and made that the external platform independent piece wouldn't we
> maintain everything this thread is trying to address without having the issues
> we're seeing now or requiring that everyone change their usage model and
> expectations?

Huh, no.

The argument is this: a single testr knows how to run heterogeneous
test backends which may be anywhere in the world, in any language, in
any context. Running it from within one of those contexts is a
seriously poor chicken-and-egg situation which doesn't make any sense.
And the current problem with database compat due to switching out the
python version between invocations is just the tip of the iceberg.


Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud

More information about the OpenStack-dev mailing list