Open Stack

On 06/04/2012 04:00 PM, Kevin L. Mitchell wrote:
> Today I've noticed some significant problems with nova's test suite
> leaving literally hundreds of python processes out.  I'm guessing that
> this has to do with the unit tests for the multiprocess patch, which was
> just approved.  This could be causing problems with jenkins, too…
>
> Anybody have any other insights?

Yes, several Jenkins slaves have been taken out by running nova unit 
tests.  The one that I am able to log into seems to be continuously 
respawing python processes.  Other slaves are inaccessible due to having 
exhausted their RAM.

I note that all of the tests run after that change merged carry this 
warning notice from Jenkins:

   Process leaked file descriptors. See 
http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build 
for more information

So I think it's fair to say that Jenkins corroborates your suspicion 
that change introduced a problem with leaking processes.

This is affecting any Jenkins slave that the nova unit tests job runs 
on, which in turn affects jobs for unrelated projects that happen to 
later run on that slave.

In addition to correcting this problem, I believe we should add a build 
step to Jenkins to ensure that all of the test processes have terminated 
correctly.

-Jim