Open Stack

Hey everybody,

As noted by fungi yesterday:

 
http://lists.openstack.org/pipermail/openstack-dev/2017-October/123337.html

We are planning to roll Zuul v3 out to take over the gate again tomorrow.

Since then it has become evident that what people should be doing about 
that right now is unclear. SOO ....

Things to Do Today to Prepare
=============================

* Please triage failures: As of right now we are aware of no SYSTEMIC 
issues that should cause v3 jobs to fail. If you have a job failing in 
the v3 check pipeline, you should at the very least triage it.

* Check the fixed issues and open issues etherpad when you triage:

   https://etherpad.openstack.org/p/zuulv3-fixed-issues

* Restart coming - only recheck things you're concerned about - Things 
are backed up on v3 due to capacity management, being down a cloud and 
running double jobs. We're about to restart v3 to pick up some changes. 
That will reset the v3 queues allowing you to recheck things to verify 
if they have been fixed and get a response more quickly.

* Read migration guide if you haven't already:

   https://docs.openstack.org/infra/manual/zuulv3.html

A Few Notes About Tomorrow
==========================

* The performance issues from before have been sorted out, so responding 
to issues should no longer take hours.

* The status page should also behave normally - it is currently in an 
exceptional state due to nodepool capacity management.

* We have a temporary high-priority check pipeline for project-config 
changes, so fixes to that repo will be able to merge quickly without 
blocking work in other repos.

* We'll be tracking known issues at:

   https://etherpad.openstack.org/p/zuulv3-issues

* Shifting jobs to being in-repo jobs is totally in-game so that you can 
iterate on things without us:

 
https://docs.openstack.org/infra/manual/zuulv3.html#moving-legacy-jobs-to-projects

* Reach out at the first sign off issues - and roll up your sleeves. 
We're in #openstack-infra and we'll be primed to jump immediately on any 
issues you're seeing, but we can't do that if we don't know about the 
issues. The teams that  have been the most successful so far have been 
the ones who have been talking to us and who have had someone dive in 
and migrate jobs away from legacy jobs. We know not everyone has the 
bandwidth to have a person dive in on reworking jobs tomorrow, but at 
least let us know if you're having issues.

The most important thing is to communicate with us.

We cannot say thank you enough for your patience over the last week and 
a half. We expect tomorrow's 'rollout, the sequel' to be MUCH smoother, 
but there will surely still be issues. The fixes from the last week 
should allow us to respond to those issues much more quickly.

Thanks,
Monty