Open Stack

Hi all!

First of all, I spoke about something yesterday in terms of being a
done-deal, when it should have been prefaced with "I'm looking at". My
bad, and I certainly apologize for freaking anyone out.

That being said - there are several issues with the current packaging
setup that I've been tasked with tackling. This email will outline the
issues, and then I'll discuss what I've been thinking in terms of solutions.

*****
This is a proposal. We want to get feedback on this before the summit
but will not be changing the way we do packaging before the summit.
*****

For reference, the current setup is:
  Jenkins creates source packages from a team
(~openstack-ubuntu-packagers) packaging branch for every trunk commit
for Ubuntu and uploads them to a PPA on Launchpad

Let me start be enumerating the issues with the current setup that we're
trying to solve. (These are not in order of importance)

- Lack of Debian support. One of the largest current deployments of
OpenStack is Rackspace, and they deploy on Debian. (disclaimer, they
also pay my salary) This means they are internally maintaining their own
set of packages, and not interacting with ours at all. That sort of sucks.

- Lack of collaboration. Related to the above, but slightly different.
Since the Rackspace internal deployment team have to maintain their own
packages, it means they are not using the published packages and we are
not benefiting from their real-world deployment experience.
Additionally, the Cloudbuilders consulting team doesn't use our
published packages, and similarly we aren't really benefiting from them
directly working with us on packaging.

- PPAs are Async. This makes integration testing harder. We're about to
roll out deployment/integration testing, which involves installing the
latest set of packages in environments and then testing that it works.
In that workflow, the queue/async nature of using LP as our package
builder becomes more problematic. I cannot trigger a direct test of a
set of packages as part of the testing of a commit if part of the
process is "upload into the LP queue and then wait for a while" -
especially since LP still has not added any sort of callback mechanism
to report completion. This means Jenkins is locally building packages
anyway so that it can pass them to deployment environments to test them.
If it's building them locally, and that's what's being tested, why
wouldn't that be what's published?

- Divergent dev process from the rest of OpenStack. The PPB just
recently voted that all OpenStack projects must use the same set of
tooling. Now granted, packaging is not a core project, so we could get
legalistic and claim that we're not required to comply - but that seems
a little disingenuous - especially since I was a strong proponent of the
one-set-of-tooling model. Additionally, since we have a strong code
review/testing system that's integrated in to our build and deployment
system, it seems very odd not to take advantage of it. (having new
packages created triggered by both approval of packaging branch changes
AND new upstream tarballs seems kindof ideal, no?)

- Not integrated with CI. PPA build errors are not reported inside of
our CI system where all of the rest of our errors are reported.
Additionally, those build errors are not reproducible by devs locally,
as they cannot spin up a launchpad PPA builder.

- Lack of RPM support. Turns out RPM distros are still kind of popular,
and at least one of our major contributors (hi, NTT) rolls out on
RHEL/CentOS, and one of our partners who is going to be contributing
testing hardware (MSFT/Novell) is associated with a distro (SuSE) that
is RPM based. We're a few steps away from this, but it warrants mentioning.

- Murky backported deps. Some of the backported deps in the PPAs came
from a dir on my laptop. Some are from Theirry's laptop. Some are from
Soren's laptop. There is neither documentation on how they got there,
how we decided to put them there, how we built them or how to continue
to collaborate on them.

- Direct injection to Ubuntu not via Debian. The publically stated
best-practice for things entering Ubuntu is to be added to/uploaded to
Debian and then synced, unless there is some compelling reason not to do
this. In this case, we had Ubuntu devs and no Debian devs in the core,
so we keep improving packages and adding packages to Ubuntu but are not
doing the same to Debian.

- PPA Fragmentation. The PPB voted that we are one project, yet we have
a separate PPA for each project.

Ok. I lied - those  might have been in a general order of importance.
Turns out though, that in terms of addressing them, they fall in to two
different groupings:

1) How we manage and work on the packaging code itself
2) How we build and publish packages

The two are related, but not linked. Of the two, the most absolutely
important is 2, so I'll address it first. I'm also going to be a bit
more blunt about it than the other one.

== How we build and publish packages ==

This deals with:
 - Lack of Debian support.
 - PPAs are Async.
 - Not integrated with CI.
 - Lack of RPM support.
 - PPA Fragmentation.

I'm going to be rolling out http://packages.openstack.org, which will at
first contain packages for lucid, maverick, natty, oneiric, squeeze and
wheezy. There will be at the very least a trunk repo and a repo
per-release (so a diablo one) Will probably use the debian trick of
pointing trunk at diablo at first, and then creating an essex repo and
pointing essex at that - but I'm up for suggestions there. We have a
Jenkins, we have plenty of builder slaves and we have pbuilder for
chroot clean builds - there is no compelling reason for us to wait on
the Launchpad build queue. AND - I can hand a puppet module to a dev
that they can run on a cloud server and they can have access to an env
which is EXACTLY like the one that build errors that they experience
occured in.

So big wins here are:
 - single place to get OpenStack as a cohesive whole
 - addition of pure Debian packages
 - possibility of adding YUM repos to packages.openstack.org
 - synchronous publishing for better testing
 - end to end visibility of what build actions were taken

Note that there is nothing stopping PPA packages from existing - we
could even have jenkins continue upload things to PPAs as well - it's
just not really a primary target.

== How we manage and work on the packaging code itself ==

Now to the one that's a little more up in the air because it's the one
that's going to cause people to throw their hands up in the air and
scream and gnash their teeth. Sigh. Sorry.

To address:
 - Lack of collaboration.
 - Divergent dev process from the rest of OpenStack.
 - Murky backported deps.
 - Direct injection to Ubuntu not via Debian.

I'm suggesting:
 - we move all packaging to git
 - we manage it consistently via git-buildpackage
 - we code review packaging branches in gerrit
 - we publish packages only through reviewed branches

Because of the last, point, I'd like to point out that this applies to
backported packages we need in our repo as well. Building any package
from the set of important OpenStack pacakges should go through the same
process. Managing the packaging for our packages and for backports will
be slightly different.

What exactly does that mean?

For our packages:

As Thomas Goirand suggested, we will add a packaging branch to the repo
of each of our projects and have git-buildpackage do the right thing. We
can have gerrit set different ACLs for the packaging branch so that a
different set of folks (the packaging team) can handle the reviewing of
that branch. Combining this with a pristine-tar and having Jenkins push
some tags back to the repo on package release, it means that we can
trace any release of software back to our code repo, and that only the
repo is needed to re-create any particular release.

The jenkins job for this is triggered off of both new commits to trunk
and to new commits to packaging.

For backported packages:

Having branches on top of upstream source is almost useless here, as in
most cases we're doing one-off backports. So here we simply make a new
repo via git-import-dsc, upload it and register it with jenkins. In the
common case, there won't be much more work to do - but in the ongoing
case, it means that anyone in our dev community knows where to go to get
at packaging of specific versions of things carried in our repo.

The jenkins jobs for these are triggered off of new commits to packaging.

For both:

When Jenkins is triggered, it will pull the branches and build packages
for all available targets in base chroots using git-pbuilder/cowbuilder.
Each build will have its full log in Jenkins just as any other build.
>From a repo perspective, for each target, Jenkins will attempt to build
using packaging in the debian branch unless there is a branch named
debian/${target} - so if the packaging needs to be overridden for, say,
lucid (because it's older than dirt) It's easy enough to just create a
debian/lucid branch  and leave debian to handle the other branches.

These packages are then all published to packages.openstack.org and jobs
testing deployment are kicked. At a point in the future (if we start
doing pre-merge deployment testing) we'll need to make a pending archive
so that we can re-publish same versions due to failure -but we're not
there yet.

As a bonus, git-buildpackage does some nice tagging of things, which we
can have Jenkins push back, so we can keep our repos up to date with
meta information. Since these are on ancillary branches, that pushing
doesn't have to interrupt normal dev work. Incidentally, since that meta
info is in the repo, we cease to need the tarball_versions and
ppa_versions files that we currently manage on the filesystem in jenkins.

I know there are other ways to accomplish some of these things, but this
is the one I'm proposing. The fact that it fits in to our development
methodology is appealing, as is the possibility of getting pvo and jesse
and thomas looped in to working directly on this set of packages.

Monty