Open Stack

Hello, Mathew.

I'm seeing same issues with the gate.
I also tried to found out why gate job is failing. First ran into issue related to cinder installation failure in devstack. But then I found same problem as you described. The best option is to increase job time range. 
Thanks for such research. I hope gate will be fixed in the easiest way and for the shortest period of time.

Best regards
Denis Makogon.
Sent from an iPad

> 16 февр. 2014, в 00:46, "Lowery, Mathew" <mlowery at ebay.com> написал(а):
> 
> Hi all,
> 
> Issue #1: Jobs that need more than one hour
> 
> Of the last 30 Trove-Gate builds (spanning three days), 7 have failed due to a Jenkins job-level timeout (not a proboscis timeout). These jobs had no failed tests when the timeout occurred.
> 
> Not having access to the job config to see what the job looks like, I used the console output to guess what was going on. It appears that a Jenkins plugin named boot-hpcloud-vm is booting a VM and running the commands given, including redstack int-tests. From the console output, it states that it was supplied with an ssh_shell_timeout="7200". This is passed down to another library called net-ssh-simple. net-ssh-simple has two timeouts: an idle timeout and an operation timeout.
> 
> In the latest boot-hpcloud-vm, ssh_shell_timeout is passed down to net-ssh-simple for both the idle timeout and the operation timeout. But in older versions of boot-hp-cloud-vm, ssh_shell_timeout is passed down to net-ssh-simple for only the idle timeout, leaving a default operation timeout of 3600. This is why I believe these jobs are failing after exactly one hour.
> 
> FYI: Here are the jobs that failed due to the Jenkins job-level timeout (and had no test failures when the timeout occurred) along with their associated patch sets:
> https://rdjenkins.dyndns.org/job/Trove-Gate/2532/console (http://review.openstack.org/73786)
> https://rdjenkins.dyndns.org/job/Trove-Gate/2530/console (http://review.openstack.org/73736)
> https://rdjenkins.dyndns.org/job/Trove-Gate/2517/console (http://review.openstack.org/63789)
> https://rdjenkins.dyndns.org/job/Trove-Gate/2514/console (https://review.openstack.org/50944)
> https://rdjenkins.dyndns.org/job/Trove-Gate/2513/console (https://review.openstack.org/50944)
> https://rdjenkins.dyndns.org/job/Trove-Gate/2504/console (https://review.openstack.org/73147)
> https://rdjenkins.dyndns.org/job/Trove-Gate/2503/console (https://review.openstack.org/73147)
> 
> Suggested action items:
> If it is acceptable to have jobs that run over one hour, then install the latest boot-hpcloud-vm plugin for Jenkins which will increase the make the operation timeout match the idle timeout.
> 
> Issue #2: The running time of all jobs is 1 hr 1 min
> 
> While the Jenkins job-level timeout will end the job after one hour, it also appears to keep every job running for a minimum of one hour.  To be more precise, the timeout (or minimum running time) occurs on the part of the Jenkins job that runs commands on the VM; the VM provision (which takes about one minute) is excluded from this timeout which is why the running time of all jobs is around 1 hr 1 min. A sampling of console logs showing the time the int-tests completed and when the timeout kicks in:
> 
> https://rdjenkins.dyndns.org/job/Trove-Gate/2531/console (00:01:03 wasted)
> 04:51:12 COMMAND_0: echo refs/changes/36/73736/2
> ...
> 05:50:10     335.41     proboscis.case.MethodTest (test_instance_created)
> 05:50:10     194.05     proboscis.case.MethodTest (test_instance_returns_to_active_after_resize)
> 05:51:13 **************************************
> 05:51:13 ****** STDERR-BEGIN ******
> 
> https://rdjenkins.dyndns.org/job/Trove-Gate/2521/console (00:06:44 wasted)
> 21:11:44 COMMAND_0: echo refs/changes/89/63789/13
> ...
> 22:05:00     195.11     proboscis.case.MethodTest (test_instance_returns_to_active_after_resize)
> 22:05:00     186.89     proboscis.case.MethodTest (test_resize_down)
> 22:11:44 **************************************
> 22:11:44 ****** STDERR-BEGIN ******
> 
> https://rdjenkins.dyndns.org/job/Trove-Gate/2518/consoleFull (00:06:01 wasted)
> 17:46:59 COMMAND_0: echo refs/changes/02/64302/20
> ...
> 18:40:57     210.03     proboscis.case.MethodTest (test_instance_returns_to_active_after_resize)
> 18:40:57     187.89     proboscis.case.MethodTest (test_resize_down)
> 18:46:58 **************************************
> 18:46:58 ****** STDERR-BEGIN ******
> 
> Suggested action items:
> Given that the minimum running time is one hour, I assume the problem is in the net-ssh-simple library. Needs more investigation.
> 
> Issue #3: Jenkins console log line timestamps different between full and truncated views
> 
> I assume this is due to JENKINS-17779.
> 
> Suggested action items:
> Upgrade the timestamper plugin.
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140216/39a31bc5/attachment.html>