Open Stack

Hey everybody,

tl;dr - We have new AFS-based consistent per-region mirrors of PyPI and 
APT repos with additional wheel repos containing pre-built wheels for 
all the modules in global-requirements

We've just rolled out a new change that you should mostly never notice - 
except that jobs should be a bit faster and more reliable.

The underpinning of the new mirrors is AFS, which is a global 
distributed filesystem developed by Carnegie Mellon back in the 1980's. 
In a lovely fit of old-is-new-again, the challenges that software had to 
deal with in the 80s (flaky networks, frequent computer failures) mirror 
life in the cloud pretty nicely, and the engineering work to solve them 
winds up being quite relevant.

One of the nice things we get from AFS is the ability to do atomic 
consistent releases of new filesystem snapshots to read-only replicas. 
That means we can build a new version of our mirror content, check it 
for consistency, and then release it for consumption to all of the 
consumers at the same time. That's important for the gate, because our 
"package not found" errors are usually about the mirror state shifting 
during a test job run.

We've had per-region PyPI mirrors for quite some time (and indeed the 
gate would largely be dead in the water without them). The improvement 
from this work for them is that they're now AFS based, so we should 
never have a visible mirror state that's wonky or inconsistent between 
regions, and we can more easily expand into new cloud regions.

We've added per-region apt mirrors (with yum to come soon) to the mix 
based on the same concept - we build the new mirror state then release 
it. There is one additional way that apt can fail even with consistent 
mirror states, which is that apt repos purge old versions of packages 
that are no longer referenced. If a new mirror state rolls out between 
the time devstack runs apt-get update and the time it tries to do 
apt-get install of something, you can get a situation where apt is 
trying to install a version of a package that is no longer present in 
the archive. To mitigate this, we're purging our mirror on a delay ... 
in our mirror runs every 2 hours we add new packages and update the 
index, and then in the next mirror run we'll delete the packages the 
previous run made unreferenced. This should make apt errors about 
package not found go away.

Last but certainly not least, there are now also wheel repositories of 
wheels built for all of our python packages from global-requirements. 
This is a speed increase and shaves 1.8 tens of minutes off of  a normal 
devstack run.

With these changes, it means we're writing not only pip.conf but now 
sources.list files into the test nodes. If you happen to be doing extra 
special things with either of those in your jobs, you'll want to make 
sure you consume the config files we're laying down

Finally, although all Infra projects are a team effort - a big shout out 
to Michael Krotschek and Jim Blair for diving in and getting this 
finished over the past couple of weeks.

Monty