Open Stack

Many thanks to Thomas and the other packagers for a great discussion at
the summit and this fast follow-up, explained well. Looking forward to
seeing what can be achieved!

On 27/05/15 16:14, Thomas Goirand wrote:
> Hi all,
> 
> tl;dr:
> - We'd like to push distribution packaging of OpenStack on upstream
> gerrit with reviews.
> - The intention is to better share the workload, and improve the overall
> QA for packaging *and* upstream.
> - The goal is *not* to publish packages upstream
> - There's an ongoing discussion about using stackforge or openstack.
> This isn't, IMO, that important, what's important is to get started.
> - There's an ongoing discussion about using a distribution specific
> namespace, my own opinion here is that using /openstack-pkg-{deb,rpm} or
> /stackforge-pkg-{deb,rpm} would be the most convenient because of a
> number of technical reasons like the amount of Git repository.
> - Finally, let's not discuss for too long and let's do it!!! :)
> 
> Longer version:
> 
> Before I start: some stuff below is just my own opinion, others are just
> given facts. I'm sure the reader is smart enough to guess which is what,
> and we welcome anyone involved in the project to voice an opinion if
> he/she differs.
> 
> During the Vancouver summit, operation, Canonical, Fedora and Debian
> people gathered and collectively expressed the will to maintain
> packaging artifacts within upstream OpenStack Gerrit infrastructure. We
> haven't decided all the details of the implementation, but spent the
> Friday morning together with members of the infra team (hi Paul!) trying
> to figure out what and how.
> 
> A number of topics have been raised, which needs to be shared.
> 
> First, we've been told that such a topic deserved a message to the dev
> list, in order to let groups who were not present at the summit. Yes,
> there was a consensus among distributions that this should happen, but
> still, it's always nice to let everyone know.
> 
> So here it is. Suse people (and other distributions), you're welcome to
> join the effort.
> 
> - Why doing this
> ================
> It's been clear to both Canonical/Ubuntu teams, and Debian (ie: myself)
> that we'd be a way more effective if we worked better together, on a
> collaborative fashion using a review process like on upstream Gerrit.
> But also, we'd like to welcome anyone, and especially the operation
> folks, to contribute and give feedback. Using Gerrit is the obvious way
> to give everyone a say on what we're implementing.
> 
> As OpenStack is welcoming every day more and more projects, it's making
> even more sense to spread the workload.
> 
> This is becoming easier for Ubuntu guys as Launchpad now understand not
> only BZR, but also Git.
> 
> We'd start by merging all of our packages that aren't core packages
> (like all the non-OpenStack maintained dependencies, then the Oslo libs,
> then the clients). Then we'll see how we can try merging core packages.
> 
> Another reason is that we believe working with the infra of OpenStack
> upstream will improve the overall quality of the packages. We want to be
> able to run a set of tests at build time, which we already do on each
> distribution, but now we want this on every proposed patch. Later on,
> when we have everything implemented and working, we may explore doing a
> package based CI on every upstream patch (though, we're far from doing
> this, so let's not discuss this right now please, this is a very long
> term goal only, and we will have a huge improvement already *before*
> this is implemented).
> 
> - What it will *not* be
> =======================
> We do not have the intention (yet?) to publish the resulting packages
> built on upstream infra. Yes, we will share the same Git repositories,
> and yes, the infra will need to keep a copy of all builds (for example,
> because core packages will need oslo.db to build and run unit tests).
> But we will still upload on each distributions on separate repositories.
> So published packages by the infra isn't currently discussed. We could
> get to this topic once everything is implemented, which may be nice
> (because we'd have packages following trunk), though please, refrain to
> engage in this topic right now: having the implementation done is more
> important for the moment. Let's try to stay on tracks and be constructive.
> 
> - Let's keep efficiency in mind
> ===============================
> Over the last few years, I've been able to maintain all of OpenStack in
> Debian with little to no external contribution. Let's hope that the
> Gerrit workflow will not slow down too much the packaging work, even if
> there's an unavoidable overhead. Hopefully, we can implement some
> liberal ACL policies for the core reviewers so that the Gerrit workflow
> don't slow down anyone too much. For example we may be able to create
> new repositories very fast, and it may be possible to self-approve some
> of the most trivial patches (for things like typo in a package
> description, adding new debconf translations, and such obvious fixes, we
> shouldn't waste our time).
> 
> There's a middle ground between the current system (ie: only write
> access ACLs for git.debian.org with no other check what so ever) and a
> too restrictive fully protected gerrit workflow that may slow down
> everyone too much. Currently, there's a small amount of people involved
> in the packaging. While we can expect a lot of operators will be
> interested to work on core packages like Nova, Neutron and so on, I am
> at the same time expecting that nobody will care about a given indirect
> python module dependency (and I may still continue to be the only one
> working on these...). We really don't want to be stuck in situations
> with nobody reviewing these.
> 
> - /stackforge or /openstack namespace
> =====================================
> There's been this question floating around. Should we try joining
> directly as part of OpenStack, as many suggested, or should we go
> through a first round (during one dev cycle, until Liberty is released?)
> through stackforge.
> 
> I have to admit that I'm not strongly opinionated about this myself.
> Sure, being an official OpenStack big-tent project would be nice, and it
> would feel worm, but also there's the issue that we don't have any
> commits within the project yet, and therefore finding who's core and
> who's the PTL could be an issue if we want to follow the official
> guidelines.
> 
> However, without even discussing the topic between us, it's obvious that
> people like James Page (plus others from his team) and myself would be
> core approvers for the Debian packaging. Fedora guys can decide for
> themselves (I must write names I have in mind for the RPM based-OS side
> of things: M. Runge, Haikel Gemmar and Alan Pevec, for example. Please
> add the missing ones here...).
> 
> Another thing is that going for stackforge now means that one day, we'll
> have to suffer from a migration to the openstack namespace, which may be
> 1/ a lot of work and 2/ disruptive. Avoiding such a migration would be good.
> 
> All together, the packaging team will happily accept whatever is decided
> by the TC. What counts for us is that a decision is taken quickly, so
> that we can start implementing it and share our repositories. So, dear
> TC, please let us know ASAP after your next meeting.
> 
> Anyhow, please feel free to discuss the issue within the governance
> patch which I started:
> https://review.openstack.org/#/c/185187/
> 
> - Specific packaging namespace
> ==============================
> Since distribution packaging through Git implies running a lot of git
> repository (one per package), it's been discussed that we may want to
> use a specific namespace for distributions. For example,
> /stackforge-packaging or /openstack-packaging.
> 
> Let's keep in mind that, for OpenStack packaging by the distributions,
> we're talking about more than 200 packages. 235 so far on the OpenStack
> packaging group in Debian [1], and it's increasing all the time...
> 
> Personally, I believe it'd be nice to have a specifc namespace for
> distributions, for multiple reasons. The first one is that we already
> have a lot of projects inside /stackforge or /openstack, and that
> searching through them isn't easy. Now, consider that we'll have
> duplicates for each and every project. Also, we don't have a complete
> match between OpenStack package names. For example, in Debian/Ubuntu, we
> have "python-" as prefix for each and every python libraries (including
> for example all oslo packages, but not only). Plus we'd be forced to use
> a prefix, for example "pkg-nova", to avoid collision. However, it'd be
> nice to keep the same repository names as per the name of the source
> package in the distributions, otherwise we'd have to fix our tooling
> which obviously will become slightly more complex.
> 
> - Distribution specific git repositories or specific branches
> =============================================================
> There's 2 issues if we keep the same repository names for multiple
> distribution.
> 
> First, we'd like to have a different set of core reviewers for a given
> distribution family. This implies that, if we use the same repository
> for RPM and DEB, then we'd have to set ACLs on per-branch basis, which
> may be harder to maintain (compared to giving access to the full set of
> Git repositories for a given distro family). Yes, we could give access
> to both for core reviewers, and trust them to behave. This trust model
> has been proved as successful within the Debian community, but still,
> adding and removing ACLs for new core reviewers would be harder to maintain.
> 
> Then, as written above, it is desirable to have a matching pair between
> distribution source package names and the name of the underlying git
> repository (so that for building X, you git clone X, not prefix-X).
> However, distributions don't have the same names for packages. For
> example, RPM supports upper case, while dpkg systems don't. Also, Red
> Hat uses a prefix "openstack-" for all core projects (like
> "openstack-nova"), while Debian based distribution don't, and so on...
> 
> - Finally
> =========
> Finally, well, please don't discuss the color of the bike-shed for too
> long. We don't really care the color as long as we have a shed to share
> the hosting of our packages Git.
> 
> Yes, I'm open to discussing all of the issues above, but if and only if
> it doesn't become a blocker for implementing packaging on upstream
> Gerrit. For the /openstack namespace, can we agree to set a deadline in
> 6 days, before the next TC meeting? We really need to get started soon,
> at the beginning of the liberty cycle, so that we can achieve important
> tasks like merging our source packages between Debian and Ubuntu (for
> example) and still be on-time for the release of Liberty. The task will
> be non-trivial, if we want to accommodate all parties, and retain the
> features our users had enjoyed over the past few years.
> 
> Last word: thanks to everyone on the infra team (and others) which were
> supportive of the project. All of this is very exciting, and it's really
> great that this is finally happening.
> 
> Cheers,
> 
> Thomas Goirand (zigo)
> 
> [1]
> https://qa.debian.org/developer.php?login=openstack-devel@lists.alioth.debian.org
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>