Open Stack

On 2015-05-28 23:19:36 +0200 (+0200), Thomas Goirand wrote:
[...]
> By the way, I was thinking about the sbuild package caching system, and
> thought: how about network mounting /var/cache/pbuilder/aptcache using
> something like Manila (or any other distributed filesystem)? Does infra
> have such tooling in place?

We pre-cache resources onto the local filesystems of the images used
to boot our job workers, and update those images daily.

> What would be the distributed filesystem of choice in such a case?

We have an AFS cell which we could consider using for this
eventually. We're working on using it as a distributed backend for
package mirrors, git repositories, documentation and potentially
lots of other things. However, as we've seen repeatedly, any actions
in a job which require network access (even locally to other servers
in the same cloud provider/region) have a variable percentage of
failures associated with network issues. The more you can avoid
depending on network resources, the better off your jobs will be.

> Also, could we setup approx somewhere? Or do we have Debian and Ubuntu
> mirrors available within infra?
[...]

There is a separate effort underway to maintain distributed
multi-distro package mirrors in all our providers/regions like we
already do for our PyPI mirrors. As mentioned above, it will likely
be updated in one place on a writeable AFS volume, automatically
consistency-checked, and then released to the read-only volumes
published from each of the mirror servers.
-- 
Jeremy Stanley