Open Stack

Sorry for top-posting...

We are aware of the issues and have put in several patches to tempest
that are now merged into tempest trunk. Several unrelated issues were
uncovered, some of which were "covering up" other issues which made
diagnosing failures difficult (especially considering the runtime for
the full tempest suite -- which tempest itself is gated on -- takes
45-90 minutes to complete).

Anyway, we are monitoring things very closely and believe we've fixed
the source of the erratic failures now.

We have the QA IRC meeting today in about 1.75 hours if you'd like to
pop into #openstack-meeting to discuss further.

Apologies for the delays and distractions.

Best,
-jay

On 01/10/2013 08:10 AM, Daniel P. Berrange wrote:
> In the past 2 days the Jenkins failure rate for the Tempest testsuite
> seems to have become incredibly high - seemingly over 50% failure rate.
> While we can trigger a rerun using 'recheck', this causes delays when
> you have run recheck multiple times for many patches.
> 
> Does anyone know what's going on with Tempest right now & more importantly
> if there's a way to fix it ?
> 
> eg this patch
> 
>   https://review.openstack.org/#/c/19116/
> 
> And this result:
> 
>   http://logs.openstack.org/19116/5/check/gate-tempest-devstack-vm/25818/console.html.gz
> 
> 12:44:47 ======================================================================
> 12:44:47 ERROR: test_reboot_server_hard (tempest.tests.compute.servers.test_server_actions.ServerActionsTestJSON)
> 12:44:47 ----------------------------------------------------------------------
> 12:44:47 Traceback (most recent call last):
> 12:44:47   File "/opt/stack/new/tempest/tempest/tests/compute/servers/test_server_actions.py", line 234, in setUp
> 12:44:47     ServerActionsTestBase.setUp(self)
> 12:44:47   File "/opt/stack/new/tempest/tempest/tests/compute/servers/test_server_actions.py", line 44, in setUp
> 12:44:47     self.client.wait_for_server_status(self.server_id, 'ACTIVE')
> 12:44:47   File "/opt/stack/new/tempest/tempest/services/compute/json/servers_client.py", line 156, in wait_for_server_status
> 12:44:47     raise exceptions.BuildErrorException(server_id=server_id)
> 12:44:47 BuildErrorException: Server fc9009c1-dafe-49b1-b01f-c575cee838f9 failed to build and is in ERROR status
> 12:44:47 
> 12:44:47 ----------------------------------------------------------------------
> 12:44:47 XML: nosetests-smoke.xml
> 12:44:47 ----------------------------------------------------------------------
> 12:44:47 Ran 84 tests in 837.230s
> 12:44:47 
> 12:44:47 FAILED (SKIP=15, errors=1)
> 
> 
> Or any of these ones had the same errors
> 
> https://review.openstack.org/#/c/19381/
> https://review.openstack.org/#/c/19301/
> https://review.openstack.org/#/c/19304/
> ...
> 
> Regrds,
> Daniel
>