Open Stack

I was expecting an answer just like that from Soren.
Thanks for being so passionate about Ubuntu! :)

(no sarcasm here intended, I really mean it)

On 08/25/2011 04:11 AM, Soren Hansen wrote:
> This is false. The packages are perfectly capable of building and
> running on Debian (barring currently unknown bugs).

No. After Cactus was released, it didn't build because of issues in
python-eventlet related to the OpenSSL transition. Then it built without
a single error, so I uploaded. Then a few months later it didn't build,
and I got warned about it by:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=634406

So it was building for about 3 or 4 months, and after the 19th of July
(date of the bug report), things got even worse: it fails earlier in the
build process.

> Whether *the rest
> of Debian* is ready for it is a different story. We chose Ubuntu to
> begin with not just because we have Ubuntu core devs on the team, but
> because Ubuntu was the only distro that met our needs:
>  a) A release cycle that lined up with what we wanted.
>  b) A fresh set of packages in its current release.
>  c) A sane, dependable way to get required changes into development releases.

I'm not sure what Monty is proposing, but I think keeping releases in
Ubuntu is a good thing, if you guys are happy with it. Never the less,
I'd love to have more involvement in the Debian side *as well*.

> Debian releases are rare and unpredictable.

Actually, they are predictable. We have now scheduled *freeze* time
(each 2 years). Yes, you wont know when it will be released, but you can
use testing a bit after it's frozen. To my experience, after 1 or 2
months of frozen stage, it's pretty stable. Reasonably, it would be
released about 6 months of time after the freeze (see the release
history, that's about the time-line nearly all the time).

> If you want a fresh set of
> libraries on top of which you can build your stuff any Debian release
> is out of the question and I don't think having a moving target (like
> unstable or testing) as a deployment target is a very sound idea.

I don't think this is anyone's proposal. Monty and I are barely asking
for having more involvement in SID / Stable-backports. I really also
hope that things will be done *before* Wheezy is frozen, so that we have
a nice environment in it for OpenStack. People tend to wake up too late
before the release...

> Imagine Cactus had been targeted for testing and we tested it and it
> all worked on release day, but the next day someone changed something
> in testing, but Cactus was already out of the door, frozen. That'd
> suck.

Well, the issue is precisely that not a lot are caring for working on
SID development! I am (and I guess, Monty) not advocating for targeting
SID or testing for releasing OpenStack. As you mentioned, it's a moving
target.

> Alternatively, you have to maintain a set of backports, but that's a
> major pain (and it only gets worse over time).

Actually, backports are fairly easy to work with, and I don't think they
are a pain. I think they are even more easy than anything else. That's
always what I suggest to anyone willing to go in production using Debian.

> Rackspace isn't doing their own packaging because of (lack of) Debian
> support. If they did, they'd have realised almost immediately that the
> packages actually build on Debian.

Cactus does in Squeeze, thanks to my (few) patches of last winter. I
guess they aren't even trying in SID, since that wouldn't be their
production platform. And as much as I heard, they had few patches that
were not up-streamed since they didn't care about cleanness, but just
having something that worked was ok (I'd like to know more about it from
the involved people here...).

> They're doing it because there'll
> supposedly be differences in the behaviour Ubuntu (or any other distro
> for that matter) will want and what Rackspace will want from the
> packages.

As much as I heard, they do backports because all their infrastructure
is running Debian stable, they are happy with it, and want to continue
this way.

>> - PPAs are Async. This makes integration testing harder.
> 
> PPAs were chosen because
> a) they are almost exactly identical to the builders for Ubuntu
> proper, so the on-commit-to-trunk builds serve as way to know
> immediately if we introduce a change that will break when uploaded to
> Ubuntu proper. So, they're the closest we'll get to integration tests
> for real Ubuntu builds,
> b) we don't have to maintain them, and
> c) adding a PPA as a source to an Ubuntu system is incredibly
> straight-forward and a very well-supported operation.

a) and b) are very valid point, c) isn't (frankly, how hard is it to
just add a line in your sources.list compared to the complexity of
setting-up OpenStack?)

>> If it's building them locally, and that's what's being tested, why
>> wouldn't that be what's published?
> 
> Because of c) above and because PPAs are trustworthy.

I don't think that c) is valid (see above), and I don't think PPAs are
more trustworthy than a private repository with a well maintained GPG
key and a ${whatever}-archive-keyring package published. As long as the
release file is signed, you have the same security, IMHO. Please let me
know if/why I'm wrong here.

By the way, there are ongoing talks inside Debian to have PPAs as well,
because we'd be happy to use the buildd machines to have ports for many
different arch. I really hope this will happen soon.

>> - Lack of RPM support. Turns out RPM distros are still kind of popular,
>> and at least one of our major contributors (hi, NTT) rolls out on
>> RHEL/CentOS, and one of our partners who is going to be contributing
>> testing hardware (MSFT/Novell) is associated with a distro (SuSE) that
>> is RPM based. We're a few steps away from this, but it warrants mentioning.
> 
> This probably warrants a separate discussion. As pointed out further
> up, Ubuntu was chosen in part because it's very up-to-date. With a
> release every 6 months, we're never far behind in terms of access to
> dependencies in the most recent release, and since our release cycles
> line up, we can *ensure* availability of the dependencies we require
> in the upcoming release.  If
> RHEL/CentOS/ScientificLinux/WhatEverOtherFlavourOfRedHatYouPrefer
> becomes a supported target, will this block stuff from landing in Nova
> until RHEL catches up with the rest of the world in terms of libraries
> upon which we depend?

I don't think that's the plan! :)
I think the plan here is to be able to target them both as much as
possible, with private (backport?) repositories for them. I do that too
for other packages inside my own private Debian repository, and with
enough care and time involved, it works. I just hope there will be
enough people with enough involvement. I think we got to make sure of
that before adding any flavor of Unix.

>> - Direct injection to Ubuntu not via Debian. The publically stated
>> best-practice for things entering Ubuntu is to be added to/uploaded to
>> Debian and then synced, unless there is some compelling reason not to do
>> this. In this case, we had Ubuntu devs and no Debian devs in the core,
>> so we keep improving packages and adding packages to Ubuntu but are not
>> doing the same to Debian.
> 
> For the umpteenth time, this is not about OpenStack. It's about the
> dependencies. Getting OpenStack itself into Debian was never the hard
> part. Getting all the dependencies into Debian was. Not just libraries
> that weren't in Debian at all (like gflags, which I got uploaded to
> Debian and synced into Ubuntu afterwards), but existing libraries in
> Debian that we needed either a newer version of or a patch applied to.
> In Ubuntu we have the power to upload these newer versions or patches
> or whatever, by implicitly accepting responsibility for it if it
> should happen to cause problems. We have no way to even begin to
> ensure that we can do this in Debian. Debian packages have
> maintainers. The maintainers decide whether to accept a patch or not.
> If they're not comfortable with a patch, rejecting it is entirely
> their prerogative (and for good reason! They're the ones who
> ultimately have to support it).

This is a pretty bad picture that you are painting of Debian.
Theoretically, you are right, at the exception that if there is a strong
technical disagreement with the maintainer of a given package, then you
can ask the technical committee to decide. But in reality, most package
maintainers will be very happy to have patches sent against their
package, and will accept them. Also, more and more, we are seeing people
forming packaging teams and work collaboratively. Just have a look in
Alioth how many packages are in collab-maint! Having teams for packaging
really is the preferred way, unlike few years ago.

> In every Ubuntu release since we
> started work on OpenStack, I've added at least one patch to libvirt to
> support something we needed for OpenStack. I've submitted them
> upstream (to libvirt) as well, but we can't stop everything while
> waiting for libvirt to make a release, the Debian maintainer to pick
> it up, and for Ubuntu to sync it.

Your example with libvirt is especially wrong. It's team maintained in
Debian, see the maintainer field:

Debian Libvirt Maintainers <pkg-libvirt-maintainers at lists.alioth.debian.org>

Have you tried joining the Alioth project? This group has 8 members
right now.

> Debian support is a massive undertaking and I'm thrilled to see it
> happen

I think that's more a mater of communication. With some maintainers,
it's going to be easy. With others, it will be harder.

> but I don't believe Debian is suitable as a reference platform

Which is the "reference platform" is another debate entirely. When it
goes down to me only, I'm fine with Ubuntu being the reference, as long
as we *also* care for SID/Squeeze-backports, which hasn't been the case
(at least over the last month).

Having jenkins to do unit tests with a SID and Squeeze-backport chroot
would certainly help to get things in a much better shape. I think that
would be the first step. If I can help for that, please let me know.

Cheers,

Thomas Goirand