[openstack-dev] [CI] Unscheduled emergency Jenkins outage 2013-01-07
James E. Blair
corvus at inaugust.com
Tue Jan 8 01:34:32 UTC 2013
As announced on IRC, we took Jenkins offline today to address the
We do not believe we were the target of an attack related to this
vulnerability, but due to its severity, we did the following:
* Immediately stopped Jenkins
* Performed the recommended upgrade
* Re-keyed all stored jenkins credentials
* Treated all stored jenkins credentials as untrusted and replaced them
This last step in particular required coordinated changing of a number
of credentials across several systems.
One aspect of that is that we needed to replace all of the devstack
nodes (and cached images) with ones created with the new Jenkins SSH
key. Due to a recent problem related to the git-core PPA that we
install on Oneiric devstack slaves, we're unable to spin up an Oneiric
devstack node at the moment, which means we're unable to run devstack
tests on the stable/diablo branch. We'll continue to work on that and
correct it as soon as possible.
Between 2013-01-07 20:30 and 2013-01-08 01:00, Zuul and Jenkins were not
running, and will have missed any Gerrit events during that time. If
you uploaded changes, you may want to leave a "recheck" comment to get
an initial test result from Jenkins, and if you approved a change, you
may need to leave another approval vote or a "reverify" comment in order
for it to merge.
I'd like to thank Clark Boylan, Jeremy Stanley, and Monty Taylor all of
whom worked throughout the day to minimize the amount of downtime.
We're available in #openstack-infra if you observe any issues.
More information about the OpenStack-dev