Open Stack

On 06/01/2015 07:16 PM, Jeremy Stanley wrote:
> On 2015-06-01 14:55:06 +0200 (+0200), Thomas Goirand wrote:
> [...]
>> So, should I start writing a script to build an image for package
>> building (ie: an image with sbuild, git-buildpackage, and so on...)?
> [...]
> 
> Probably what we'd want to do is something like debootstrap/rpmstrap

FYI, I was the Debian package maintainer for rpmstrap, and I just gave
up on it because it was not maintainable (ie: breaking constantly).
Bootstraping an RPM distribution can simply be done with yum (which is
why I maintain Yum in Debian).

> a chroot for each platform we want to build, then in each of them
> iterate through the packaging git repos and --download-only the
> build-deps listed therein. That will prime a local cache in each
> chroot and then it will get baked into that image.

Ok, got the idea. Though what should be done is fill
/var/cache/pbuilder/aptarchive rather than the host OS. I'll try to do
with something of that sort.

> Later when a
> worker is booted from that image, the package build job just chroots
> into the appropriate filesystem subtree and has a warm cache
> available to it so it only needs to hopefully update at most the
> package list and maybe a handful of packages before starting to
> build whatever new package was requested.

Ok. That's what sbuild will do if we fill /var/cache/pbuilder/aptarchive
with relevant content.

> The good thing about this approach is that it can be added later, it
> doesn't need to be implemented on day one.

Yeah. I'm currently working on a Jessie cloud image with an sbuild
configuration, so that when Paul is ready, we can use that.
>> How would a job get the latest version of a Git repository then? This
>> still needs network, right?
> [...]
> 
> The way our jobs work right now is that the workers start with a
> recent (generally no more than a day old) clone of all the Git
> repositories we maintain. It still has to hit the network to
> retrieve more recent Git refs, but this at least minimizes network
> interaction and significantly reduces the failure rate.

That will be the little bit more tricky part. Some libraries are very
small, and probably caching will not be useful (too much work when
building the VM image). However, for big projects (nova, neutron,
cinder...), then we'll have to do something for Git repository caching.

Thanks for your input Jeremy, this is very valuable.

Cheers,

Thomas Goirand (zigo)