[openstack-dev] How to turn tempest CLI tests into python-*client in-tree functional tests

Sean Dague sean at dague.net
Wed May 6 09:43:29 UTC 2015


On 05/06/2015 04:57 AM, Chris Dent wrote:
> On Wed, 6 May 2015, Robert Collins wrote:
> 
>> Its actually an antipattern. It tells testr that tests are appearing
>> and disappearing depending on what test entry point a user runs each
>> time.
>>
>> testr expects the set of tests to only change when code changes.
>>
>> So, I fully expect that this pattern is going to lead to wtf moments
>> now, and likely more in future.
>>
>> Whats the right forum for discussing the pressures that lead to this
>> hack, so we can do something that works better with the underlying
>> tooling, rather than in such a disruptive fashion?
> 
> I'd appreciate here (that is on this list) because from my
> perspective there are a lot of embedded assumptions in the way testr
> does things and wants the environment to be that aren't immediately
> obvious and would perhaps be made more clear if you could expand on
> the details of what's wrong with this particular hack.
> 
> I tried to come up with a specific question here to drive that
> illumination a bit more concretely but a) not enough coffee yet b)
> mostly I just want to know more detail about the first three
> paragraphs above.

There are 2 reasons that pattern exists.

1) testr discovery is quite slow on large trees, especially if your
intent is to run a small subset of tests by sending an argument

2) testr still doesn't have an exclude facility, so top level test
exclusion has to be done by quite complicated negative asserting regex,
which are very error prone (and have been done incorrectly many times).
Especially if you *still* want to support partial test passing.

	-Sean

-- 
Sean Dague
http://dague.net



More information about the OpenStack-dev mailing list