Open Stack

w00t! thanks infra team

On Wed, Feb 10, 2016 at 7:45 PM, Monty Taylor <mordred at inaugust.com> wrote:
> Hey everybody,
>
> tl;dr - We have new AFS-based consistent per-region mirrors of PyPI and APT
> repos with additional wheel repos containing pre-built wheels for all the
> modules in global-requirements
>
> We've just rolled out a new change that you should mostly never notice -
> except that jobs should be a bit faster and more reliable.
>
> The underpinning of the new mirrors is AFS, which is a global distributed
> filesystem developed by Carnegie Mellon back in the 1980's. In a lovely fit
> of old-is-new-again, the challenges that software had to deal with in the
> 80s (flaky networks, frequent computer failures) mirror life in the cloud
> pretty nicely, and the engineering work to solve them winds up being quite
> relevant.
>
> One of the nice things we get from AFS is the ability to do atomic
> consistent releases of new filesystem snapshots to read-only replicas. That
> means we can build a new version of our mirror content, check it for
> consistency, and then release it for consumption to all of the consumers at
> the same time. That's important for the gate, because our "package not
> found" errors are usually about the mirror state shifting during a test job
> run.
>
> We've had per-region PyPI mirrors for quite some time (and indeed the gate
> would largely be dead in the water without them). The improvement from this
> work for them is that they're now AFS based, so we should never have a
> visible mirror state that's wonky or inconsistent between regions, and we
> can more easily expand into new cloud regions.
>
> We've added per-region apt mirrors (with yum to come soon) to the mix based
> on the same concept - we build the new mirror state then release it. There
> is one additional way that apt can fail even with consistent mirror states,
> which is that apt repos purge old versions of packages that are no longer
> referenced. If a new mirror state rolls out between the time devstack runs
> apt-get update and the time it tries to do apt-get install of something, you
> can get a situation where apt is trying to install a version of a package
> that is no longer present in the archive. To mitigate this, we're purging
> our mirror on a delay ... in our mirror runs every 2 hours we add new
> packages and update the index, and then in the next mirror run we'll delete
> the packages the previous run made unreferenced. This should make apt errors
> about package not found go away.
>
> Last but certainly not least, there are now also wheel repositories of
> wheels built for all of our python packages from global-requirements. This
> is a speed increase and shaves 1.8 tens of minutes off of  a normal devstack
> run.
>
> With these changes, it means we're writing not only pip.conf but now
> sources.list files into the test nodes. If you happen to be doing extra
> special things with either of those in your jobs, you'll want to make sure
> you consume the config files we're laying down
>
> Finally, although all Infra projects are a team effort - a big shout out to
> Michael Krotschek and Jim Blair for diving in and getting this finished over
> the past couple of weeks.
>
> Monty
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims