Open Stack

On Wed, Feb 10, 2016 at 06:45:25PM -0600, Monty Taylor wrote:
> Hey everybody,
> 
> tl;dr - We have new AFS-based consistent per-region mirrors of PyPI and APT
> repos with additional wheel repos containing pre-built wheels for all the
> modules in global-requirements
> 
> We've just rolled out a new change that you should mostly never notice -
> except that jobs should be a bit faster and more reliable.
> 
> The underpinning of the new mirrors is AFS, which is a global distributed
> filesystem developed by Carnegie Mellon back in the 1980's. In a lovely fit
> of old-is-new-again, the challenges that software had to deal with in the
> 80s (flaky networks, frequent computer failures) mirror life in the cloud
> pretty nicely, and the engineering work to solve them winds up being quite
> relevant.
> 
> One of the nice things we get from AFS is the ability to do atomic
> consistent releases of new filesystem snapshots to read-only replicas. That
> means we can build a new version of our mirror content, check it for
> consistency, and then release it for consumption to all of the consumers at
> the same time. That's important for the gate, because our "package not
> found" errors are usually about the mirror state shifting during a test job
> run.
> 
> We've had per-region PyPI mirrors for quite some time (and indeed the gate
> would largely be dead in the water without them). The improvement from this
> work for them is that they're now AFS based, so we should never have a
> visible mirror state that's wonky or inconsistent between regions, and we
> can more easily expand into new cloud regions.
> 
> We've added per-region apt mirrors (with yum to come soon) to the mix based
> on the same concept - we build the new mirror state then release it. There
> is one additional way that apt can fail even with consistent mirror states,
> which is that apt repos purge old versions of packages that are no longer
> referenced. If a new mirror state rolls out between the time devstack runs
> apt-get update and the time it tries to do apt-get install of something, you
> can get a situation where apt is trying to install a version of a package
> that is no longer present in the archive. To mitigate this, we're purging
> our mirror on a delay ... in our mirror runs every 2 hours we add new
> packages and update the index, and then in the next mirror run we'll delete
> the packages the previous run made unreferenced. This should make apt errors
> about package not found go away.
> 
> Last but certainly not least, there are now also wheel repositories of
> wheels built for all of our python packages from global-requirements. This
> is a speed increase and shaves 1.8 tens of minutes off of  a normal devstack
> run.

This is a big win for everyone. You can see the speed improvement:

http://status.openstack.org/openstack-health/#/test/devstack?end=2016-02-10T15:04:32.039Z&resolutionKey=hour&duration=P1M

It's quite obvious devstack started getting much faster when the wheel mirror
was enabled.

> 
> With these changes, it means we're writing not only pip.conf but now
> sources.list files into the test nodes. If you happen to be doing extra
> special things with either of those in your jobs, you'll want to make sure
> you consume the config files we're laying down
> 
> Finally, although all Infra projects are a team effort - a big shout out to
> Michael Krotschek and Jim Blair for diving in and getting this finished over
> the past couple of weeks.
> 
> Monty
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160211/9b0d1285/attachment.pgp>