Open Stack

Hey all!

tl;dr - PLEASE DO NOT APPROVE PATCHES BEFORE THE CHECK JOBS HAVE FINISHED

I'm sure everyone has noticed, it's Feature Freeze time again, which
means that everyone has eleventy-billion patches that need to land NOW.
As always, we're discovering new and exciting scaling opportunities!

The infra team has done a lot of work in prep for our favorite time of
year, and we've actually landed several upgrades to the gate without
which we'd be in particularly bad shape right now. (I'll let Jim write
about some of them later when he's not battling the current operational
issues - they're pretty spectacular) As with many scaling issues, some
of these upgrades have resulted in moving the point of pain further
along the stack. We're working on solutions to the current pain points.
(Or, I should say they are, because I'm on a plane headed to Burning Man
and not useful for much other than writing emails.)

Which brings me to - the gate queue is really long right now. As much
technology as we are bringing to bear on the problem, it's just a
reality. That means everyone gets antsy and wants their stuff to land
RIGHT NOW. I get it- I'm probably the most impatient person in the world.

However, as the tl;dr says above, we're seeing folks hitting Approve
before the upload check passes. That may seem expedient, but failures in
the gate itself are actually quite costly, as they cause everything
behind it to stack up. If the check queue can catch an error, we REALLY
need it to right now, because we need every little bit of help we can get.

So to summarize:

PLEASE DO NOT APPROVE PATCHES BEFORE THE CHECK JOBS HAVE FINISHED

At least not this week.

Thanks!
Monty