<font size=2 face="sans-serif">Dear all,</font>
<br>
<br><font size=2 face="sans-serif">I found that in following scenario jenkins
job may run on same node in CI environment.</font>
<br>
<br><img src=cid:_1_0A0723FC07276DDC00304CC348257C7C>
<br><font size=2 face="sans-serif">When nodepool is trying to delete the
node 1 on jenkins, but the node has been assigned to queued 2, so nodepool
will be failed to delete node 1.</font>
<br>
<br><font size=2 face="sans-serif"><b>2014-01-21 03:13:16,520</b> DEBUG
nodepool.NodeUpdateListener: Received: <b>onFinalized </b>{"name":"gate-ci-devstack-test","url":"job/gate-ci-devstack-test/","build":{"full_url":"</font><a href="https://172.16.2.115/job/gate-ci-devstack-test/3815/"><font size=2 face="sans-serif">https://172.16.2.115/job/gate-ci-devstack-test/3815/</font></a><font size=2 face="sans-serif">","number":3815,"phase":"FINISHED","status":"FAILURE","url":"job/gate-ci-devstack-test/3815/","parameters":{"BASE_LOG_PATH":"70/61470/3/check","LOG_PATH":"70/61470/3/check/gate-ci-devstack-test/739f893","ZUUL_BRANCH":"master","ZUUL_CHANGE":"61470","ZUUL_CHANGE_IDS":"61470,3","ZUUL_CHANGES":"openstack/nova:master:refs/changes/70/61470/3","ZUUL_COMMIT":"956132e8df4377e66d5b78b5b9864c7da37c6bde","ZUUL_PATCHSET":"3","ZUUL_PIPELINE":"check","ZUUL_PROJECT":"openstack/nova","ZUUL_REF":"refs/zuul/master/Z1473cd61831a445792d06152612ce7f9","ZUUL_URL":"</font><a href=http://172.16.2.118/p><font size=2 face="sans-serif">http://172.16.2.118/p</font></a><font size=2 face="sans-serif">","ZUUL_UUID":"739f893379b84a64a22ea4db1721f7e7"},"node_name":"<b>devstack-precise-check-v1-gemini-cdl-7323</b>"}}</font>
<br><font size=2 face="sans-serif"><b>2014-01-21 03:13:16,544</b> DEBUG
nodepool.NodeUpdateListener: Received: <b>onStarted </b>{"name":"gate-ci-devstack-test","url":"job/gate-ci-devstack-test/","build":{"full_url":"</font><a href="https://172.16.2.115/job/gate-ci-devstack-test/3823/"><font size=2 face="sans-serif">https://172.16.2.115/job/gate-ci-devstack-test/3823/</font></a><font size=2 face="sans-serif">","number":3823,"phase":"STARTED","url":"job/gate-ci-devstack-test/3823/","parameters":{"BASE_LOG_PATH":"61/43061/7/check","LOG_PATH":"61/43061/7/check/gate-ci-devstack-test/cf9f514","ZUUL_BRANCH":"master","ZUUL_CHANGE":"43061","ZUUL_CHANGE_IDS":"43061,7","ZUUL_CHANGES":"openstack/nova:master:refs/changes/61/43061/7","ZUUL_COMMIT":"a6cd36551b778be3903eb552c22338e16708ed6e","ZUUL_PATCHSET":"7","ZUUL_PIPELINE":"check","ZUUL_PROJECT":"openstack/nova","ZUUL_REF":"refs/zuul/master/Z8721bb3c40d743d882b4c18ff896a079","ZUUL_URL":"</font><a href=http://172.16.2.118/p><font size=2 face="sans-serif">http://172.16.2.118/p</font></a><font size=2 face="sans-serif">","ZUUL_UUID":"cf9f514c02a04da8a83ba2222dc4bebd"},"node_name":"<b>devstack-precise-check-v1-gemini-cdl-7323</b>"}}</font>
<br><font size=2 face="sans-serif">2014-01-21 03:13:16,551 INFO nodepool.NodeUpdateListener:
Setting node id: 7323 to USED</font>
<br><font size=2 face="sans-serif">2014-01-21 03:13:16,557 DEBUG nodepool.JenkinsManager:
Manager jenkins01 running task <nodepool.jenkins_manager.NodeExistsTask
object at 0x7faa14315090></font>
<br><font size=2 face="sans-serif">2014-01-21 03:13:17,576 DEBUG nodepool.JenkinsManager:
Manager jenkins01 running task <nodepool.jenkins_manager.DeleteNodeTask
object at 0x7faa10095e90></font>
<br><font size=2 face="sans-serif"><b>2014-01-21 03:13:17,883</b> <b>ERROR
nodepool.NodeCompleteThread: Exception handling event for devstack-precise-check-v1-gemini-cdl-7323:</b></font>
<br><font size=2 face="sans-serif">Traceback (most recent call last):</font>
<br><font size=2 face="sans-serif"> File "/usr/local/lib/python2.7/dist-packages/nodepool/nodepool.py",
line 65, in run</font>
<br><font size=2 face="sans-serif"> self.handleEvent(session)</font>
<br><font size=2 face="sans-serif"> File "/usr/local/lib/python2.7/dist-packages/nodepool/nodepool.py",
line 101, in handleEvent</font>
<br><font size=2 face="sans-serif"> self.nodepool.deleteNode(session,
node)</font>
<br><font size=2 face="sans-serif"> File "/usr/local/lib/python2.7/dist-packages/nodepool/nodepool.py",
line 1032, in deleteNode</font>
<br><font size=2 face="sans-serif"> jenkins.deleteNode(jenkins_name)</font>
<br><font size=2 face="sans-serif"> File "/usr/local/lib/python2.7/dist-packages/nodepool/jenkins_manager.py",
line 118, in deleteNode</font>
<br><font size=2 face="sans-serif"> return self.submitTask(DeleteNodeTask(name=name))</font>
<br><font size=2 face="sans-serif"> File "/usr/local/lib/python2.7/dist-packages/nodepool/task_manager.py",
line 90, in submitTask</font>
<br><font size=2 face="sans-serif"> return task.wait()</font>
<br><font size=2 face="sans-serif"> File "/usr/local/lib/python2.7/dist-packages/nodepool/task_manager.py",
line 51, in run</font>
<br><font size=2 face="sans-serif"> self.done(self.main(client))</font>
<br><font size=2 face="sans-serif"> File "/usr/local/lib/python2.7/dist-packages/nodepool/jenkins_manager.py",
line 64, in main</font>
<br><font size=2 face="sans-serif"> return jenkins.delete_node(self.args['name'])</font>
<br><font size=2 face="sans-serif"> File "/usr/local/lib/python2.7/dist-packages/python_jenkins-0.2.1-py2.7.egg/jenkins/__init__.py",
line 508, in delete_node</font>
<br><font size=2 face="sans-serif"> raise JenkinsException('delete[%s]
failed' % (name))</font>
<br><font size=2 face="sans-serif"><b>JenkinsException: delete[devstack-precise-check-v1-gemini-cdl-7323]
failed</b></font>
<br>
<br><font size=2 face="Tahoma">This error often occurs when all nodes are
busy for zuul queue.</font>
<br><font size=2 face="sans-serif">Changing following codes to no delay
in <i>nodepool.py</i> can reduce the probability:</font>
<br><font size=2 face="Consolas"><i> time.sleep(DELETE_DELAY)</i></font>
<br><font size=2 face="Consolas"><i> self.nodepool.deleteNode(session,
node)</i></font>
<br><font size=2 face="sans-serif">But as you see the above log, the probability
still exists, how to resolve it thoroughly ? Welcome your opinions!</font>
<br>
<br><font size=2 face="sans-serif">Thanks & Best Regards,</font>
<br>
<br><font size=2 face="sans-serif">Godwin Hu(؛ْ¹ْاه)<br>
<br>
Software Engineer<br>
IBM China System and Technology Lab(CSTL), Beijing<br>
Tel(Seat): 86-010-82451453</font>
<br><font size=2 face="sans-serif">Location : Ring Building, 1BW270<br>
E-mail Address: hguoqing@cn.ibm.com <br>
Address: IBM ZGC Campus. Ring Building, 28# ZhongGuanCun Software Park,
Shang Di, Beijing P.R.China 100193</font>