Open Stack

On Tue, Apr 28, 2015 at 2:41 PM, Monty Taylor <mordred at inaugust.com> wrote:
> On 04/28/2015 02:02 PM, Clark Boylan wrote:
>> On Mon, Apr 27, 2015, at 11:13 PM, Steve Kowalik wrote:
>>> Hi,
>>>
>>>      Monty and I have been discussing Infra building and hosting
>>> distribution packages that we may require -- things like third-party
>>> libraries where the distros might lag behind pypi, or where the library
>>> is not provided by the distro. Which means I'm writing a specification
>>> for it, and at the moment, there are far too many unanswered questions
>>> to push up a spec.
>
> Thanks for kicking this off!
>
Agreed.  I am excited to see there is much movement around this.  I
planned to talk to people about this at the summit too.

>>>      I thought I would bring them up here and see if we can reach
>>> rough consensus, or if we want to defer until Vancouver.
>>>
>>>
>>> * Decide where to host said git repositories (or indeed, if we even
>>> want to), and who should have permissions to approve changes.
>> Hopefully we can agree to use Gerrit. If we intend on these packages
>> being reconsumable then using the openstack/ or is probably best. If
>> they are only intended for our testing maybe openstack-infra? Though
>> there is some recent thought that we should stop distinguishing orgs all
>> together like that so maybe just openstack/.
>
> I don't think there is a thing to agree to here. If it's source code
> related to the openstack project, it goes into gerrit.
>
> As far as which namespace - I think that gets informed a little bit by
> the next thing ...
>
+1

>> As far as permissions go, I think we can seed the groups with an initial
>> set of interested individuals then grow from there. Same basic process
>> for any other new project(s) on Gerrit.
>>>
>>> * Decide if we host full upstream sources, or if we point at a remote
>>> site with tarballs to extract.
>> I don't have much of an opinion here other than I think it would be
>> unfun to maintain local forks of all the things we build packages for. I
>> am inclined to say point at remote locations until we need to do
>> otherwise for some reason.
>
> I agree with Clark - except I see both needs.
>
> I can imagine a few different things we might want to build and/or host:
>
> - temporary packages of things where the packaging source lives
> elsewhere (think "run the version of libvirt that's in debian unstable
> on our ubuntu trusty build hosts, but stop keeping a copy as soon as it
> hits an official repo")
>
> - temporary packages of things where we need to modify the packaging in
> some manner (think "the nova team wants a version of libvirt that does
> not exist anywhere yet, and us installing it on our build hosts is part
> of the step needed to show that it's worthwhile putting into the next
> release of a distro - but we'll consume from the upstream distro when it
> gets there")
>
> - per-commit or per-tag versions of things where we are the upstream but
> the packaging repo is not hosted in gerrit ("think building a package of
> nova on each commit and shoving it somewhere and using the installation
> of that package as the basis for doing gate testing")
>
> - per-commit or per-tag versions of things where we are the upstream and
> where the packaging is hosted in gerrit ("think infra running CD
> deployments of zuul per-commit but building a deb of it first")
>
> So I think the answer here is complex and needs to accomodate some
> packages where we want to have a packaging repository that is a
> first-class citizen in our infrastructure, and some things where we do
> not want to import a packaging repository into gerrit but instead want
> to either reference an external packaging repository, or even just
> generate some packages with no packaging repository based on other forms
> of scripting.
>
> Once we have that sketched out, figuring out which repos need to exist
> should then be pretty clear where those repos need to go.
>
>>> * Decide how we are going to trigger builds of packages. Per commit may
>>> not be the most useful step, since committers may want to pull multiple
>>> disparate changes together into one changelog entry, and hence one
>>> build.
>> If a single commit does not produce a useful artifact then it is an
>> incomplete commit. We should be building packages on every commit, we
>> may not publish them on every commit but we should build them on every
>> commit (this is one way to gate for package changes). If we need to
>> control publishing independent of commits we can use tags to publish
>> releases.
>
> I agree with clarkb. I would prefer that we build a package on every
> commit and that we "publish" those packages on a tag like what works
> with all of our other things
>
> I think, as with the other things, there is a case here we also publish
> to a location per-commit. Again, zuul comes to mind as an example of
> wanting to do that. If a commit to zuul requires a corresponding commit
> and tag to deploy to our zuul server, then we have lost functionality.
> That said, if we have packaging infrastructure and want to provide a
> zuul repo so that other people can run "releases" - then I could see
> tagged versions publishing to a different repo than untagged versions.
>
I could also see publishing per commit packages to some temporary
location (24-48hrs) allow developer to consume them.  At least that's
what I was doing locally.

>>>
>>> * Decide where to host the resultant package repositories.
>>>
>>> * Decide how to deal with removal of packages from the package
>>> repositories.
>> I would like to see us building packages before we worry too much about
>> hosting them. In the past everyone has want to jump straight to these
>> problems when we really just need to sort out building packages first.
>
> I agree and disagree with Clark. I think the above questions need to get
> sorted first. However, I'd like to mention that, again, there are a few
> different use cases.
>
> One could be a long-lived repo for things we care and feed, like zuul,
> that is publicly accessible with no warnings.
>
> One could be a repo into which we put, for instance, per-commit packages
> of OpenStack because we've decided that we want to run infra-cloud that
> way. OR - a repo that we use to run infra-cloud that is only published
> to when we tag something.
>
> One could be a per-build repo - where step one in a devstack-gate run is
> building packages from the repos that zuul has prepared - and then we
> make that repo available to all of the jobs that are associated with
> that zuul ref.
>
> One could be a long-lived repo that contains some individually curated
> but automatically built things such as a version of libvirt that the
> nova team requests. Such a repo would need to be made publically
> available such that developers working on OpenStack could run devstack
> and it would get the right version.
>
> Finally - we should also always keep in mind that whatever our solution
> it needs to be able to handle debian, ubuntu, fedora and centos.
>
>> But if I were to look ahead I would expect that we would run per distro
>> package mirrors to host our packages and possibly mirror distro released
>> packages as well.
>
> And yes - as Clark says, we may also (almost certainly do based on this
> morning) want to have mirrors of each of the upstream distro package
> repos that we based things on top of.
>
Not much to add from me right now. I am excited :)

-- 
Paul Belanger | PolyBeacon, Inc.
Jabber: paul.belanger at polybeacon.com | IRC: pabelanger (Freenode)
Github: https://github.com/pabelanger | Twitter: https://twitter.com/pabelanger