<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>Hello, Mathew.</div><div><br></div><div>I'm seeing same issues with the gate.</div><div>I also tried to found out why gate job is failing. First ran into issue related to cinder installation failure in devstack. But then I found same problem as you described. The best option is to increase job time range. </div><div>Thanks for such research. I hope gate will be fixed in the easiest way and for the shortest period of time.<br><br>Best regards</div><div>Denis Makogon.<br>Sent from an iPad</div><div><br>16 февр. 2014, в 00:46, "Lowery, Mathew" <<a href="mailto:mlowery@ebay.com">mlowery@ebay.com</a>> написал(а):<br><br></div><blockquote type="cite"><div>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
Hi all,</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<b>Issue #1: Jobs that need more than one hour</b></div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
Of the last 30 <a href="https://rdjenkins.dyndns.org/job/Trove-Gate/">Trove-Gate</a> builds (spanning three days), 7 have failed due to a Jenkins job-level timeout (not a proboscis timeout). These jobs had no failed tests when the timeout occurred.</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
Not having access to the job config to see what the job looks like, I used the console output to guess what was going on. It appears that a Jenkins plugin named
<a href="https://github.com/mrhoades/boot-hpcloud-vm/blob/2272770b0ce54752eabb84229dc8939d79b2be50/models/boot_vm_concurrent.rb#L181">
boot-hpcloud-vm</a> is booting a VM and running the commands given, including redstack int-tests. From the console output, it states that it was supplied with an ssh_shell_timeout="7200". This is passed down to another library called
<a href="https://github.com/busyloop/net-ssh-simple/blob/e3834f259a47606bfb06a487ca701fc20dbad8a5/lib/net/ssh/simple.rb#L632">
net-ssh-simple</a>. net-ssh-simple has two timeouts: an idle timeout and an operation timeout.</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
In the <a href="https://github.com/mrhoades/boot-hpcloud-vm/blob/2272770b0ce54752eabb84229dc8939d79b2be50/models/boot_vm_concurrent.rb#L182">
latest boot-hpcloud-vm</a>, ssh_shell_timeout is passed down to net-ssh-simple for both the idle timeout and the operation timeout. But in
<a href="https://github.com/mrhoades/boot-hpcloud-vm/blob/9260e957d6c54142c33dd9e9632b86e17fd5c02f/models/boot_vm_concurrent.rb#L141">
older versions of boot-hp-cloud-vm</a>, ssh_shell_timeout is passed down to net-ssh-simple for only the idle timeout, leaving a default operation timeout of 3600. This is why I believe these jobs are failing after exactly one hour.</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>FYI: Here are the jobs that failed due to the Jenkins job-level timeout (and had no test failures when the timeout occurred) along with their associated patch sets:</div>
<div><a href="https://rdjenkins.dyndns.org/job/Trove-Gate/2532/console">https://rdjenkins.dyndns.org/job/Trove-Gate/2532/console</a> (<a href="http://review.openstack.org/73786">http://review.openstack.org/73786</a>)</div>
<div><a href="https://rdjenkins.dyndns.org/job/Trove-Gate/2530/console">https://rdjenkins.dyndns.org/job/Trove-Gate/2530/console</a> (<a href="http://review.openstack.org/73736">http://review.openstack.org/73736</a>)</div>
<div><a href="https://rdjenkins.dyndns.org/job/Trove-Gate/2517/console">https://rdjenkins.dyndns.org/job/Trove-Gate/2517/console</a> (<a href="http://review.openstack.org/63789">http://review.openstack.org/63789</a>)</div>
<div><a href="https://rdjenkins.dyndns.org/job/Trove-Gate/2514/console">https://rdjenkins.dyndns.org/job/Trove-Gate/2514/console</a> (<a href="https://review.openstack.org/50944">https://review.openstack.org/50944</a>)</div>
<div><a href="https://rdjenkins.dyndns.org/job/Trove-Gate/2513/console">https://rdjenkins.dyndns.org/job/Trove-Gate/2513/console</a> (<a href="https://review.openstack.org/50944">https://review.openstack.org/50944</a>)</div>
<div><a href="https://rdjenkins.dyndns.org/job/Trove-Gate/2504/console">https://rdjenkins.dyndns.org/job/Trove-Gate/2504/console</a> (<a href="https://review.openstack.org/73147">https://review.openstack.org/73147</a>)</div>
<div><a href="https://rdjenkins.dyndns.org/job/Trove-Gate/2503/console">https://rdjenkins.dyndns.org/job/Trove-Gate/2503/console</a> (<a href="https://review.openstack.org/73147">https://review.openstack.org/73147</a>)</div>
<div><br>
</div>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<b>Suggested action items:</b></div>
<ul style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<li>If it is acceptable to have jobs that run over one hour, then install the latest boot-hpcloud-vm plugin for Jenkins which will increase the make the operation timeout match the idle timeout.</li></ul>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<b>Issue #2: The running time of all jobs is 1 hr 1 min</b></div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
While the Jenkins job-level timeout will end the job after one hour, it also appears to keep every job running for a minimum of one hour. To be more precise, the timeout (or minimum running time) occurs on the part of the Jenkins job that runs commands on
the VM; the VM provision (which takes about one minute) is excluded from this timeout which is why the
<a href="https://rdjenkins.dyndns.org/job/Trove-Gate/buildTimeTrend">running time of all jobs is around 1 hr 1 min</a>. A sampling of console logs showing the time the int-tests completed and when the timeout kicks in:</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<a href="https://rdjenkins.dyndns.org/job/Trove-Gate/2531/console">https://rdjenkins.dyndns.org/job/Trove-Gate/2531/console</a> (00:01:03 wasted)</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<pre style="white-space: pre-wrap; word-wrap: break-word; margin-top: 0px; margin-bottom: 0px; font-size: 11px;"><pre style="background-color: rgb(255, 255, 255); white-space: pre-wrap; word-wrap: break-word; margin-top: 0px; margin-bottom: 0px;"><span class="timestamp"><b>04:51:12</b> COMMAND_0: echo refs/changes/36/73736/2</span></pre><pre style="white-space: pre-wrap; word-wrap: break-word; margin-top: 0px; margin-bottom: 0px;"><span class="timestamp" style="background-color: rgb(255, 254, 254);">...</span></pre><pre style="background-color: rgb(255, 255, 255); white-space: pre-wrap; word-wrap: break-word; margin-top: 0px; margin-bottom: 0px;"><span class="timestamp"><b>05:50:10</b> </span> 335.41 proboscis.case.MethodTest (test_instance_created)
<span class="timestamp"><b>05:50:10</b> </span> 194.05 proboscis.case.MethodTest (test_instance_returns_to_active_after_resize)
<span class="timestamp"><b>05:51:13</b> </span>**************************************
<span class="timestamp"><b>05:51:13</b> </span>****** STDERR-BEGIN ******</pre></pre>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<a href="https://rdjenkins.dyndns.org/job/Trove-Gate/2521/console">https://rdjenkins.dyndns.org/job/Trove-Gate/2521/console</a> (00:06:44 wasted)</div>
<div>
<pre style="color: rgb(0, 0, 0); white-space: pre-wrap; word-wrap: break-word; margin-top: 0px; margin-bottom: 0px;"><pre style="font-size: 11px; white-space: pre-wrap; word-wrap: break-word; margin-top: 0px; margin-bottom: 0px;"><pre style="background-color: rgb(255, 255, 255); white-space: pre-wrap; word-wrap: break-word; margin-top: 0px; margin-bottom: 0px;"><span class="timestamp"><b>21:11:44</b> </span>COMMAND_0: echo refs/changes/89/63789/13</pre><pre style="white-space: pre-wrap; word-wrap: break-word; margin-top: 0px; margin-bottom: 0px;"><span style="background-color: rgb(255, 254, 254);">...</span></pre></pre><pre style="background-color: rgb(255, 255, 255); font-size: 11px; white-space: pre-wrap; word-wrap: break-word; margin-top: 0px; margin-bottom: 0px;"><font face="Courier"><span class="timestamp"><b>22:05:00</b> </span> 195.11 proboscis.case.MethodTest (test_instance_returns_to_active_after_resize)
<span class="timestamp"><b>22:05:00</b> </span> 186.89 proboscis.case.MethodTest (test_resize_down)
<span class="timestamp"><b>22:11:44</b> </span>**************************************
<span class="timestamp"><b>22:11:44</b> </span>****** STDERR-BEGIN ******</font><span style="font-family: Calibri, sans-serif;">
</span></pre><div style="background-color: rgb(255, 255, 255); font-size: 11px; font-family: Calibri, sans-serif;"><br></div><div style="background-color: rgb(255, 255, 255);"><font face="Calibri"><a href="https://rdjenkins.dyndns.org/job/Trove-Gate/2518/consoleFull">https://rdjenkins.dyndns.org/job/Trove-Gate/2518/consoleFull</a> (00:06:01 wasted)</font></div><div style="font-size: 11px; font-family: Calibri, sans-serif;"><pre style="background-color: rgb(255, 255, 255); white-space: pre-wrap; word-wrap: break-word; margin-top: 0px; margin-bottom: 0px;"><span class="timestamp"><b>17:46:59</b> COMMAND_0: echo refs/changes/02/64302/20</span></pre><pre style="white-space: pre-wrap; word-wrap: break-word; margin-top: 0px; margin-bottom: 0px;"><span class="timestamp" style="background-color: rgb(255, 254, 254);">...</span></pre><pre style="background-color: rgb(255, 255, 255); white-space: pre-wrap; word-wrap: break-word; margin-top: 0px; margin-bottom: 0px;"><span class="timestamp"><b>18:40:57</b> </span> 210.03 proboscis.case.MethodTest (test_instance_returns_to_active_after_resize)
<span class="timestamp"><b>18:40:57</b> </span> 187.89 proboscis.case.MethodTest (test_resize_down)
<span class="timestamp"><b>18:46:58</b> </span>**************************************
<span class="timestamp"><b>18:46:58</b> </span>****** STDERR-BEGIN ******
</pre></div><div style="background-color: rgb(255, 255, 255); font-size: 11px; font-family: Calibri, sans-serif;"><br></div><font face="Calibri,sans-serif" style="background-color: rgb(255, 255, 255);"><b>Suggested action items:</b></font></pre>
<ul>
<li>
<pre style="word-wrap: break-word; margin-top: 0px; margin-bottom: 0px; background-color: rgb(255, 255, 255);"><font face="Calibri,sans-serif"><span style="white-space: pre-wrap;">Given that the minimum running time is one hour, I assume the problem is in the net-ssh-simple library. Needs more investigation.</span></font></pre>
</li></ul>
<pre style="color: rgb(0, 0, 0); white-space: pre-wrap; word-wrap: break-word; margin-top: 0px; margin-bottom: 0px; background-color: rgb(255, 255, 255);"><font face="Calibri,sans-serif" style="font-size: 11px;"><br></font></pre>
<pre style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; white-space: pre-wrap; word-wrap: break-word; margin-top: 0px; margin-bottom: 0px;"><span style="background-color: rgb(255, 254, 254);"><b>Issue #3: Jenkins console log line timestamps different between full and truncated views</b></span></pre>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
I assume this is due to <a href="https://issues.jenkins-ci.org/browse/JENKINS-17779">JENKINS-17779</a>.</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<b>Suggested action items:</b></div>
<ul style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<li>Upgrade the <a href="https://wiki.jenkins-ci.org/display/JENKINS/Timestamper">
timestamper plugin</a>.</li></ul>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>OpenStack-dev mailing list</span><br><span><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a></span><br><span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></span><br></div></blockquote></body></html>