[openstack-dev] [all][infra] Zuul v3 rollout, the sequel
Gary Kotton
gkotton at vmware.com
Tue Oct 10 12:05:05 UTC 2017
Hi,
At the moment the neutron, neutron-lib and many of the decomposed projects are still failing with the v3. Does this mean that we are broken from the 11th?
For the decomposed projects we have a work around to help address this in the short term – need to increase timeout and need a flag from Zuul3 that is not part of Jenkins - https://github.com/openstack/vmware-nsx/blob/master/tools/tox_install_project.sh#L37 (can we have ZUUL3_CLONER?)
Thanks
Gary
On 10/10/17, 2:59 AM, "Jeremy Stanley" <fungi at yuggoth.org> wrote:
The tl;dr is that we're planning to roll forward out of our partial
Zuul v3 rollback starting at 11:00 UTC on Wednesday October 11
(a little over 35 hours from now), so expect some CI downtime and
all of the benefits (though hopefully none of the drawbacks!) you
witnessed when we tried the first time.
It's been right at a week since we instituted a partial rollback of
our initial v3 roll-out. That week has been filled by diagnosing and
fixing all of the misbehaviors and performance degradation we
identified, including some new issues we discovered while running
under even heavier load and memory pressure artificially induced by
trying to fire check jobs for everything v2 was running but with
only a fraction of the node capacity. Numerous issues within the
translated legacy job configs were also fixed, and a bunch more of
them replaced by v3-native jobs.
We anticipate an outage for the CI system of somewhere between 30-60
minutes starting at 11:00 UTC on Wednesday October 11. Once
complete, we'll send a follow-up announcement along with a link to
where we're coordinating and triaging any newly observed issues. In
the meantime, if you've been digging into recent legacy job failures
for your projects you should consider trying to bring them to our
attention as soon as possible (in #openstack-infra on the Freenode
IRC network, or replying on-list to this announcement) and work on
fixing them if you're familiar enough with the failures to do so
yourself. We're assembling a sort of FAQ in the migration guide
here:
https://docs.openstack.org/infra/manual/zuulv3.html
...and we also have some more content in progress at:
https://etherpad.openstack.org/p/zuulv3-migration-faq
In summary, Zuul v3 is looking better than ever, and we hope you'll
be as pleased with it as we are!
--
Jeremy Stanley
More information about the OpenStack-dev
mailing list