Open Stack

On 2014-11-25 07:33:14 -0500 (-0500), Sean Dague wrote:
> Right now all icehouse jobs are showing up as NOT_REGISTERED. It's
> cross project and current.

This should be solved now for any dsvm icehouse jobs started since
15:00 UTC today.

Contributing factors:

1. We restarted Zuul and it doesn't keep a history of Gearman worker
registrations over restart, relying on the Jenkins masters to
reregister what jobs they know about.

2. The Gearman plug-in for Jenkins only registers jobs for a given
worker type when a worker of that type is added to that master.

3. Due to extremely low relative demand for devstack-precise workers
(as they only run for stable/icehouse branch and backward-compat
jobs), Nodepool was only trying to build one at a time.

4. For some reason the latest devstack-precise snapshot in one of
our providers' regions was causing nova boot to consistently fail
and discard the instance as being in an error state.

5. Due to the way the Nodepool scheduler is designed, when it is
retrying to boot one single node of a given label it retries it in
the same provider and region: in this case repeatedly trying to
build one single devstack-precise node from a snapshot which was not
working in that location.

Resolution was ultimately achieved by briefly shuffling the provider
order for the devstack-precise label in Nodepool's configuration,
causing it to retry in a different region where the boot ultimately
succeeded and got attached to a Jenkins master which then registered
all jobs requiring devstack-precise nodes into Zuul.
-- 
Jeremy Stanley