Open Stack

On Tue, Jun 18, 2013 at 4:26 AM, Alexander Gorodnev
<agorodnev at mirantis.com> wrote:
> Hi,
>
> I have the same problem with Quantum test. I don't even know what it depends
> on, but I get the different result every time I run unittests using "tox
> -epy27".
> My latest result is:
>
> Ran 4548 (-721) tests in 205.026s (-63.932s)
> PASSED (id=8, skips=201)
>
> As you can see amount of test decreased, but I haven't changed any files.
> The best result is:
>
> Ran 5279 (+14) tests in 290.402s (+4.897s)
>
> Could it be some perfomance issue? And how can I fix it or try to debug?
>
> Thanks,
> Alexander
>
>
> On Mon, Jun 17, 2013 at 8:42 PM, Roman Bogorodskiy
> <rbogorodskiy at mirantis.com> wrote:
>>
>> Hi,
>>
>> On a fresh checkout of quantum with new venv I have tox missing lots of
>> tests (exact command is 'tox -epy27').
>>
>> It reports:
>>
>> ....
>>
>> Ran 2368 tests in 470.017s (+2.755s)
>> PASSED (id=10, skips=131)
>>
>> ....
>>
>>   py27: commands succeeded
>>     congratulations :)
>>
>> I noticed that there should be 5000+ tests (e.g. in jenkins logs).
>>
>> What could be wrong here?
I finally got around to looking into this today. I ran `tox -epy27`
locally a couple times and noticed that test numbers did vary (but not
as wildly as reported earlier). My next step was to source the tox
py27 virtualenv then to `testr list-tests | wc -l` to get a rough
number for the total number of tests (if tests are dynamically created
at run time this number will be low). This gave me 5986 well above the
~5500-5600 that were actually running. At this point I wanted to
remove parallel testing from the equation to see if all tests run
properly in serial. `testr run` ran 2368 tests so removing parallel
testing did not fix the problem.

Running the tests in this way did produce some interesting output in
the subunit log [1]. Notice that the NEC agent is returning 0. This
kills the test runner prematurely and tells testr that tests passed.
This sys.exit(0) shouldn't be in the test path like this. I haven't
looked at what fixing this will entail but it is probably possible to
catch the SystemExit exception in the test and ignore it. I have a
hunch the proper fix will involve making the NEC agent more properly
unit testable. I will defer to those with more knowledge of OpenStack
Networking for the actual fix.

Clark

[1] http://paste.openstack.org/show/38858/