Open Stack

Victor,

To correct the math, Python3 runs 686 tests in 15.971s or 42.95 tests per second.  Python2 runs 1495 tests in 22.595s or 66.16 tests per second.  Python2 is 54% faster (but even that's just an estimate, since it assumes all tests run at the same speed, which isn't true - plus it runs them on different hardware and not all VMs are created equal ;) ).

We're aware that the 'dangling mock detector' makes the code run significantly slower (historically ~15 min instead of ~5 min) however it's helped us catch more than one bug with minimal effort.  The question is why does it run even slower in python3?  If it is just the infra issue of running on slower hardware, then we could probably address that.

Peter



-----Original Message-----
From: Victor Stinner [mailto:vstinner at redhat.com] 
Sent: May-16-16 10:57 AM
To: openstack-dev at lists.openstack.org
Subject: Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db

Le 16/05/2016 16:12, Peter Stachowski a écrit :
> Amrith is right though - python34 is *much* slower running this check 
> (and somewhat in general) than python27.  Maybe that doesn't translate 
> into real-world performance data, but it has made me a bit nervous 
> nonetheless.

IMHO we should look more closely to the output. See my comments on the bug report:

    https://bugs.launchpad.net/trove/+bug/1582257

Currently, running unit tests on Python 3 runs 686 tests in 15.971s, whereas Python 2 runs 1495 tests in 22.595s. Python 3 is "faster" :-)
(16 sec < 23 sec)

The whole Python 3 job takes 10 minutes, but in these 10 minutes, almost
8 minutes are taken just to build binary wheel packages. On Python 2, prebuilt packages are used and so creating the virtual environment is much faster (around one minute). I proposed a change to prebuild also wheel packages on Python 3:

    https://review.openstack.org/#/c/316890/

Ok, but sometimes we get timeout, right? Again, if you look closely to the output, between two runs on Python 2, the timing also changes a lot:

(a) Ran 1495 tests in 22.595s
(b) Ran 1495 tests in 393.364s => 17x slower!

The issue is not specific to Python 3. But it looks like the random slowdown is much larger on Python 3. Duration of Python 3 unit tests:

(a) "Ran 686 tests in 15.971s"
(b) "Ran 686 tests in 1581.090s" => 100x slower!

I guess that the difference is that Python 3 unit tests are currently run sequentially (1 process), whereas Python 2 unit tests are run in parallel (8 processes).

*Again* please check the code detecting dangling mocks. For example, "tox -e py34" takes 14 seconds without this code, 177 seconds with the
code: 12x slower with the code.

Victor

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev