Open Stack

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi all,

tl;dr:
- - We'd like to push distribution packaging of OpenStack on upstream
gerrit with reviews.
- - The intention is to better share the workload, and improve the overall
QA for packaging *and* upstream.
- - The goal is *not* to publish packages upstream
- - There's an ongoing discussion about using stackforge or openstack.
This isn't, IMO, that important, what's important is to get started.
- - There's an ongoing discussion about using a distribution specific
namespace, my own opinion here is that using /openstack-pkg-{deb,rpm} or
/stackforge-pkg-{deb,rpm} would be the most convenient because of a
number of technical reasons like the amount of Git repository.
- - Finally, let's not discuss for too long and let's do it!!! :)

Longer version:

Before I start: some stuff below is just my own opinion, others are just
given facts. I'm sure the reader is smart enough to guess which is what,
and we welcome anyone involved in the project to voice an opinion if
he/she differs.

During the Vancouver summit, operation, Canonical, Fedora and Debian
people gathered and collectively expressed the will to maintain
packaging artifacts within upstream OpenStack Gerrit infrastructure. We
haven't decided all the details of the implementation, but spent the
Friday morning together with members of the infra team (hi Paul!) trying
to figure out what and how.

A number of topics have been raised, which needs to be shared.

First, we've been told that such a topic deserved a message to the dev
list, in order to let groups who were not present at the summit. Yes,
there was a consensus among distributions that this should happen, but
still, it's always nice to let everyone know.

So here it is. Suse people (and other distributions), you're welcome to
join the effort.

- - Why doing this
================
It's been clear to both Canonical/Ubuntu teams, and Debian (ie: myself)
that we'd be a way more effective if we worked better together, on a
collaborative fashion using a review process like on upstream Gerrit.
But also, we'd like to welcome anyone, and especially the operation
folks, to contribute and give feedback. Using Gerrit is the obvious way
to give everyone a say on what we're implementing.

As OpenStack is welcoming every day more and more projects, it's making
even more sense to spread the workload.

This is becoming easier for Ubuntu guys as Launchpad now understand not
only BZR, but also Git.

We'd start by merging all of our packages that aren't core packages
(like all the non-OpenStack maintained dependencies, then the Oslo libs,
then the clients). Then we'll see how we can try merging core packages.

Another reason is that we believe working with the infra of OpenStack
upstream will improve the overall quality of the packages. We want to be
able to run a set of tests at build time, which we already do on each
distribution, but now we want this on every proposed patch. Later on,
when we have everything implemented and working, we may explore doing a
package based CI on every upstream patch (though, we're far from doing
this, so let's not discuss this right now please, this is a very long
term goal only, and we will have a huge improvement already *before*
this is implemented).

- - What it will *not* be
=======================
We do not have the intention (yet?) to publish the resulting packages
built on upstream infra. Yes, we will share the same Git repositories,
and yes, the infra will need to keep a copy of all builds (for example,
because core packages will need oslo.db to build and run unit tests).
But we will still upload on each distributions on separate repositories.
So published packages by the infra isn't currently discussed. We could
get to this topic once everything is implemented, which may be nice
(because we'd have packages following trunk), though please, refrain to
engage in this topic right now: having the implementation done is more
important for the moment. Let's try to stay on tracks and be constructive.

- - Let's keep efficiency in mind
===============================
Over the last few years, I've been able to maintain all of OpenStack in
Debian with little to no external contribution. Let's hope that the
Gerrit workflow will not slow down too much the packaging work, even if
there's an unavoidable overhead. Hopefully, we can implement some
liberal ACL policies for the core reviewers so that the Gerrit workflow
don't slow down anyone too much. For example we may be able to create
new repositories very fast, and it may be possible to self-approve some
of the most trivial patches (for things like typo in a package
description, adding new debconf translations, and such obvious fixes, we
shouldn't waste our time).

There's a middle ground between the current system (ie: only write
access ACLs for git.debian.org with no other check what so ever) and a
too restrictive fully protected gerrit workflow that may slow down
everyone too much. Currently, there's a small amount of people involved
in the packaging. While we can expect a lot of operators will be
interested to work on core packages like Nova, Neutron and so on, I am
at the same time expecting that nobody will care about a given indirect
python module dependency (and I may still continue to be the only one
working on these...). We really don't want to be stuck in situations
with nobody reviewing these.

- - /stackforge or /openstack namespace
=====================================
There's been this question floating around. Should we try joining
directly as part of OpenStack, as many suggested, or should we go
through a first round (during one dev cycle, until Liberty is released?)
through stackforge.

I have to admit that I'm not strongly opinionated about this myself.
Sure, being an official OpenStack big-tent project would be nice, and it
would feel worm, but also there's the issue that we don't have any
commits within the project yet, and therefore finding who's core and
who's the PTL could be an issue if we want to follow the official
guidelines.

However, without even discussing the topic between us, it's obvious that
people like James Page (plus others from his team) and myself would be
core approvers for the Debian packaging. Fedora guys can decide for
themselves (I must write names I have in mind for the RPM based-OS side
of things: M. Runge, Haikel Gemmar and Alan Pevec, for example. Please
add the missing ones here...).

Another thing is that going for stackforge now means that one day, we'll
have to suffer from a migration to the openstack namespace, which may be
1/ a lot of work and 2/ disruptive. Avoiding such a migration would be good.

All together, the packaging team will happily accept whatever is decided
by the TC. What counts for us is that a decision is taken quickly, so
that we can start implementing it and share our repositories. So, dear
TC, please let us know ASAP after your next meeting.

Anyhow, please feel free to discuss the issue within the governance
patch which I started:
https://review.openstack.org/#/c/185187/

- - Specific packaging namespace
==============================
Since distribution packaging through Git implies running a lot of git
repository (one per package), it's been discussed that we may want to
use a specific namespace for distributions. For example,
/stackforge-packaging or /openstack-packaging.

Let's keep in mind that, for OpenStack packaging by the distributions,
we're talking about more than 200 packages. 235 so far on the OpenStack
packaging group in Debian [1], and it's increasing all the time...

Personally, I believe it'd be nice to have a specifc namespace for
distributions, for multiple reasons. The first one is that we already
have a lot of projects inside /stackforge or /openstack, and that
searching through them isn't easy. Now, consider that we'll have
duplicates for each and every project. Also, we don't have a complete
match between OpenStack package names. For example, in Debian/Ubuntu, we
have "python-" as prefix for each and every python libraries (including
for example all oslo packages, but not only). Plus we'd be forced to use
a prefix, for example "pkg-nova", to avoid collision. However, it'd be
nice to keep the same repository names as per the name of the source
package in the distributions, otherwise we'd have to fix our tooling
which obviously will become slightly more complex.

- - Distribution specific git repositories or specific branches
=============================================================
There's 2 issues if we keep the same repository names for multiple
distribution.

First, we'd like to have a different set of core reviewers for a given
distribution family. This implies that, if we use the same repository
for RPM and DEB, then we'd have to set ACLs on per-branch basis, which
may be harder to maintain (compared to giving access to the full set of
Git repositories for a given distro family). Yes, we could give access
to both for core reviewers, and trust them to behave. This trust model
has been proved as successful within the Debian community, but still,
adding and removing ACLs for new core reviewers would be harder to maintain.

Then, as written above, it is desirable to have a matching pair between
distribution source package names and the name of the underlying git
repository (so that for building X, you git clone X, not prefix-X).
However, distributions don't have the same names for packages. For
example, RPM supports upper case, while dpkg systems don't. Also, Red
Hat uses a prefix "openstack-" for all core projects (like
"openstack-nova"), while Debian based distribution don't, and so on...

- - Finally
=========
Finally, well, please don't discuss the color of the bike-shed for too
long. We don't really care the color as long as we have a shed to share
the hosting of our packages Git.

Yes, I'm open to discussing all of the issues above, but if and only if
it doesn't become a blocker for implementing packaging on upstream
Gerrit. For the /openstack namespace, can we agree to set a deadline in
6 days, before the next TC meeting? We really need to get started soon,
at the beginning of the liberty cycle, so that we can achieve important
tasks like merging our source packages between Debian and Ubuntu (for
example) and still be on-time for the release of Liberty. The task will
be non-trivial, if we want to accommodate all parties, and retain the
features our users had enjoyed over the past few years.

Last word: thanks to everyone on the infra team (and others) which were
supportive of the project. All of this is very exciting, and it's really
great that this is finally happening.

Cheers,

Thomas Goirand (zigo)

[1]
https://qa.debian.org/developer.php?login=openstack-devel@lists.alioth.debian.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCAAGBQJVZXzYAAoJENQWrRWsa0P+M8gP/21TcEfBs/PTxUFRDMSfP97a
JCVj5JIy6GCFjDC+8mbX9WGWFEbU0hRn2CG/dRJNbcrkjeQJaDosSd3BMmEAsrQO
ONyu0Qbm2YgclZ9OyUaewE8NPSH8cGRjxS+IdOm5kzhNqX538ohsxD2XNMknx5rU
cyq3TTDKWL3ot8+SVdUqfvzmZZdESF+8lucneqOiRk9wU/UVt6fdsovuv3QSqM2h
k7etO4Oev46nQFCVvSmofv2mfGTWn5iWDbcCj7qf/blI5uiL4GkTkzFbkr/+f2De
7C4p9rO8WsEaK+Ptg2hayiOm36i6JPnvO6YSOjm6yTeY4LQ3UkULN2/THvftSkYX
I+oICj7XbJMhaUDbg9mfJhVZ+O5L0v9YHV2v12NIWMbDGb6EzSYsLqmvANwYN5wM
Y8zihyETZFYsJVmPdqHN6ZPGeuKAroFo0F5rldMJ/++J5exApf6DJyd+JrE7vmyV
F3sksS+BhmPgmEBbSvcA0ZP8gqEfPWMK5zT1rWBakKy1OHtDaLwnzsCAkaS6kXUJ
G45Sr965H6pkO10EMpIbjFeCcVIP7UpmHDxq1PQbhUx4aogjVGA1mIWwuD0Cscaj
hD18bJyZxe0t0MzhkijxWnQ6holJ1C36SvSD/jha6oZ7k9FHIAwo1MklTJcCEo7J
PBnrpJTi9IkhDmfp21/s
=5T0x
-----END PGP SIGNATURE-----