[openstack-dev] [Neutron] tox run forever

Clark Boylan clark.boylan at gmail.com
Fri Feb 28 01:28:05 UTC 2014


On Thu, Feb 27, 2014 at 4:43 PM, Shixiong Shang
<sparkofwisdom.cloud at gmail.com> wrote:
> Hi, guys:
>
> I created a fresh local repository and pulled the most recent Neutron code. Before I put in my own code, I did TOX run. However, seems like it is stuck to the following condition for over a hour and didn't go any further. Yesterday, the TOX had been running with a fresh copy of Neutron, but didn't return SUCCESS after the entire night.
>
> I assume the copy from MASTER BRANCH should already be sanitized.....However, what I saw in the past 48 hours told me different story. Did I do anything wrong?
>
>
> shshang at net-ubuntu2:~/github/neutron$ tox -e py27
> py27 create: /home/shshang/github/neutron/.tox/py27
> py27 installdeps: -r/home/shshang/github/neutron/requirements.txt, -r/home/shshang/github/neutron/test-requirements.txt, setuptools_git>=0.4
> py27 develop-inst: /home/shshang/github/neutron
> py27 runtests: commands[0] | python -m neutron.openstack.common.lockutils python setup.py testr --slowest --testr-args=
> [pbr] Excluding argparse: Python 2.6 only dependency
> running testr
> running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit} --list
> running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit}  --load-list /tmp/tmpbZwLwg
> running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit}  --load-list /tmp/tmp39qJYM
> running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit}  --load-list /tmp/tmpppXiTc
> running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit}  --load-list /tmp/tmpPhJZDc
>
> Thanks!
>
> Shixiong
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

I think there are two potential problems here. Either a test is
deadlocking due to something it has done or
neutron.openstack.common.lockutils is deadlocking. In either case
OS_TEST_TIMEOUT is not set in .testr.conf so the test suite will not
timeout individual tests if necessary. I would start by setting that
in the .testr.conf next to OS_STDOUT_CAPTURE and you probably want a
value of like 300 (that is seconds).

The other thing you can do to debug this is grab the subunit log file
out of .testrepository. While tests are running it will have a tmp
random generated name. After tests have run it will be moved to a file
named after the most recent test run eg 1 for the first run. The
subunit log should offer clues to what was running at the time of
deadlocking.

Clark



More information about the OpenStack-dev mailing list