Open Stack

On 04/28/2015 02:02 PM, Clark Boylan wrote:
> On Mon, Apr 27, 2015, at 11:13 PM, Steve Kowalik wrote:
>> Hi,
>>
>> 	Monty and I have been discussing Infra building and hosting
>> distribution packages that we may require -- things like third-party
>> libraries where the distros might lag behind pypi, or where the library
>> is not provided by the distro. Which means I'm writing a specification
>> for it, and at the moment, there are far too many unanswered questions
>> to push up a spec.

Thanks for kicking this off!

>> 	I thought I would bring them up here and see if we can reach
>> rough consensus, or if we want to defer until Vancouver.
>>
>>
>> * Decide where to host said git repositories (or indeed, if we even
>> want to), and who should have permissions to approve changes.
> Hopefully we can agree to use Gerrit. If we intend on these packages
> being reconsumable then using the openstack/ or is probably best. If
> they are only intended for our testing maybe openstack-infra? Though
> there is some recent thought that we should stop distinguishing orgs all
> together like that so maybe just openstack/.

I don't think there is a thing to agree to here. If it's source code
related to the openstack project, it goes into gerrit.

As far as which namespace - I think that gets informed a little bit by
the next thing ...

> As far as permissions go, I think we can seed the groups with an initial
> set of interested individuals then grow from there. Same basic process
> for any other new project(s) on Gerrit.
>>
>> * Decide if we host full upstream sources, or if we point at a remote
>> site with tarballs to extract.
> I don't have much of an opinion here other than I think it would be
> unfun to maintain local forks of all the things we build packages for. I
> am inclined to say point at remote locations until we need to do
> otherwise for some reason.

I agree with Clark - except I see both needs.

I can imagine a few different things we might want to build and/or host:

- temporary packages of things where the packaging source lives
elsewhere (think "run the version of libvirt that's in debian unstable
on our ubuntu trusty build hosts, but stop keeping a copy as soon as it
hits an official repo")

- temporary packages of things where we need to modify the packaging in
some manner (think "the nova team wants a version of libvirt that does
not exist anywhere yet, and us installing it on our build hosts is part
of the step needed to show that it's worthwhile putting into the next
release of a distro - but we'll consume from the upstream distro when it
gets there")

- per-commit or per-tag versions of things where we are the upstream but
the packaging repo is not hosted in gerrit ("think building a package of
nova on each commit and shoving it somewhere and using the installation
of that package as the basis for doing gate testing")

- per-commit or per-tag versions of things where we are the upstream and
where the packaging is hosted in gerrit ("think infra running CD
deployments of zuul per-commit but building a deb of it first")

So I think the answer here is complex and needs to accomodate some
packages where we want to have a packaging repository that is a
first-class citizen in our infrastructure, and some things where we do
not want to import a packaging repository into gerrit but instead want
to either reference an external packaging repository, or even just
generate some packages with no packaging repository based on other forms
of scripting.

Once we have that sketched out, figuring out which repos need to exist
should then be pretty clear where those repos need to go.

>> * Decide how we are going to trigger builds of packages. Per commit may
>> not be the most useful step, since committers may want to pull multiple
>> disparate changes together into one changelog entry, and hence one
>> build.
> If a single commit does not produce a useful artifact then it is an
> incomplete commit. We should be building packages on every commit, we
> may not publish them on every commit but we should build them on every
> commit (this is one way to gate for package changes). If we need to
> control publishing independent of commits we can use tags to publish
> releases. 

I agree with clarkb. I would prefer that we build a package on every
commit and that we "publish" those packages on a tag like what works
with all of our other things

I think, as with the other things, there is a case here we also publish
to a location per-commit. Again, zuul comes to mind as an example of
wanting to do that. If a commit to zuul requires a corresponding commit
and tag to deploy to our zuul server, then we have lost functionality.
That said, if we have packaging infrastructure and want to provide a
zuul repo so that other people can run "releases" - then I could see
tagged versions publishing to a different repo than untagged versions.

>>
>> * Decide where to host the resultant package repositories.
>>
>> * Decide how to deal with removal of packages from the package
>> repositories.
> I would like to see us building packages before we worry too much about
> hosting them. In the past everyone has want to jump straight to these
> problems when we really just need to sort out building packages first.

I agree and disagree with Clark. I think the above questions need to get
sorted first. However, I'd like to mention that, again, there are a few
different use cases.

One could be a long-lived repo for things we care and feed, like zuul,
that is publicly accessible with no warnings.

One could be a repo into which we put, for instance, per-commit packages
of OpenStack because we've decided that we want to run infra-cloud that
way. OR - a repo that we use to run infra-cloud that is only published
to when we tag something.

One could be a per-build repo - where step one in a devstack-gate run is
building packages from the repos that zuul has prepared - and then we
make that repo available to all of the jobs that are associated with
that zuul ref.

One could be a long-lived repo that contains some individually curated
but automatically built things such as a version of libvirt that the
nova team requests. Such a repo would need to be made publically
available such that developers working on OpenStack could run devstack
and it would get the right version.

Finally - we should also always keep in mind that whatever our solution
it needs to be able to handle debian, ubuntu, fedora and centos.

> But if I were to look ahead I would expect that we would run per distro
> package mirrors to host our packages and possibly mirror distro released
> packages as well.

And yes - as Clark says, we may also (almost certainly do based on this
morning) want to have mirrors of each of the upstream distro package
repos that we based things on top of.

Monty