On Thu, Jan 21, 2016 at 08:18:14AM -0500, Davanum Srinivas wrote: > Hi, > > Failures for this job has been trending up and is causing the large > gate queue as well. I've logged a bug: > https://bugs.launchpad.net/openstack-gate/+bug/1536622 > > and am requesting switching the voting to off for this job: > https://review.openstack.org/#/c/270788/ I think this was premature, we were actually looking at the problem last night. If you look at: http://status.openstack.org/openstack-health/#/g/node_provider/internap-nyj01 and http://status.openstack.org/openstack-health/#/g/node_provider/bluebox-sjc1 grenade-multinode is 100% failure on both providers. The working hypothesis is that it's because tempest is trying to login to the guest over the "private" network which isn't setup as accessible outside. You can see the discussion on this starting here: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-01-20.log.html#t2016-01-20T22:44:24 > > We need to find and fix the underlying issue which can help us > determine when to switch this back on to voting or we cleanup this job > from all the gate queues and move them to check queues (i have a TODO > for this in this review) TBH, there is always this push to remove jobs or testing whenever there is release pressure and a gate backup. No one seems to notice whenever anything isn't working and recheck grinds patches through. (well maybe not you Dims, because you're more on top of it then almost everyone) I know that I get complacent when there isn't a gate backup. The problem is when things like our categorization rate on: http://status.openstack.org/elastic-recheck/data/uncategorized.html routinely has been at or below 50% this cycle it's not really a surprise we have gate backups like this. More people need to be actively debugging these problems as they come up, it can't just be the same handful of us. I don't think making things non-voting is the trend we want to set because then what's gonna be the motivation to get others to help on this. -Matt Treinish -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160121/2c111eaa/attachment.pgp>