Open Stack

On 2016-02-24 15:47:12 +0000 (+0000), Steven Dake (stdake) wrote:
[...]
> Where we are stumped is should we push to registry.openstack.org
> which I would prefer, or push to the docker registry.

If we use a Jenkins publisher the files get pulled from the slave
where they were generated to the Jenkins master server and then
copied from there to the archive location. This might present some
significant disk utilization issues on our Jenkins masters given the
file sizes you've quoted, so I wouldn't recommend that as a
solution.

Another possible publication model is what we did with log uploads
to Swift containers, where the Jenkins slave is given temporary
constrained write credentials assigned by Zuul and directly uploads
the files to the container, though index generation becomes a
challenge if you need any sort of hierarchical organization (it's
not like a POSIX filesystem so emulating Apache autoindex is
challenging).

Another major challenge for either of these is bandwidth. It's
entirely possible your images are built on slaves in France and then
shipped across the Atlantic to Dallas for archiving. I would not
expect this to be a fast operation.

> Another issue is getting the authentication credentials in the
> gate securely - somehow it needs to be injected into our post jobs
> without us having to check our credentials into the repository.

If uploading to a third party service, the way we do that is to
upload the files from the single-use slave first to a location under
our control (by way of one of the mechanisms described above), and
then run another job dependent on a trusted long-lived slave to
retrieve the file and upload it to the final destination using
preinstalled credentials for that remote service.

The storage and bandwidth needs for this use case make me suspect
our CI system is a poor place to attempt to implement such a
solution, but hopefully others have ideas which simply aren't
occurring to me.
-- 
Jeremy Stanley