Open Stack

On 08/24/2011 01:11 PM, Soren Hansen wrote:
> 2011/8/24 Monty Taylor <mordred at inaugust.com>:
>> - Lack of Debian support. One of the largest current deployments of
>> OpenStack is Rackspace, and they deploy on Debian. (disclaimer, they
>> also pay my salary) This means they are internally maintaining their own
>> set of packages, and not interacting with ours at all. That sort of sucks..
> 
> This is false. The packages are perfectly capable of building and
> running on Debian (barring currently unknown bugs). Whether *the rest
> of Debian* is ready for it is a different story. We chose Ubuntu to
> begin with not just because we have Ubuntu core devs on the team, but
> because Ubuntu was the only distro that met our needs:
>  a) A release cycle that lined up with what we wanted.
>  b) A fresh set of packages in its current release.
>  c) A sane, dependable way to get required changes into development releases.
> 
> Debian releases are rare and unpredictable. If you want a fresh set of
> libraries on top of which you can build your stuff any Debian release
> is out of the question and I don't think having a moving target (like
> unstable or testing) as a deployment target is a very sound idea.
> Imagine Cactus had been targeted for testing and we tested it and it
> all worked on release day, but the next day someone changed something
> in testing, but Cactus was already out of the door, frozen. That'd
> suck.
> 
> Alternatively, you have to maintain a set of backports, but that's a
> major pain (and it only gets worse over time).

We already maintain a set of backports. This will not change.

Also, I think that our definitions of some things are different here.
I'm not talking about reference platform for inclusion upstream. I'm not
talking about "I'm on Ubuntu Oneiric and I want to install OpenStack, so
I'm going to apt-get install openstack from the main Ubuntu repository"
I imagine it will be literally YEARS before anyone does that.

I expect that people running OpenStack will, as step one, add the
OpenStack repo, or the OpenStack PPA, or wherever we publish our latest
and greatest packages. We released Cactus 6 months ago, and we did it in
such a way as to get it included into Ubuntu - and less than a month
later the general consensus is "cactus is unusable"

So I'm talking about repository lifecycle from an upstream perspective
with the assumption that I will be maintaining a repository from which
my consumers can run OpenStack. Futher, I am assuming that some of them
will want to run current OpenStack on Lucid. (hi Rackspace Cloud Files)
or on Debian (hi upcoming Rackspace Cloud Servers) or RHEL6. As
upstreams, it is incumbent on us to provide a complete experience for that.

This, btw, is the disconnect in intentions that comes up at every UDS
when I put on my hat as upstream (formerly from Drizzle, before that
from MySQL, now from OpenStack) - which is that main inclusion is all
well and good, but I don't actually care.

In fact, the only reason I really care about main inclusion for
OpenStack for Ubuntu is because Canonical has a (asinine) requirement
that only software in main can run in their data centers, and I would
like for Canonical devs to be able to run internal services using
OpenStack rather than Eucalyptus. Other than that, I do not think any of
the consumers of OpenStack will use the packages directly from Ubuntu.
(especially since to do so in the next few months would mean deploying
Oneiric server, which I think even fewer people will do. Data centers
tend to like to deploy servers running latest RHEL or latest Ubuntu LTS
and then deploy up-to-date packages for the services that that
particular machine runs.

All of that to say:

- PPAs do not allow me to upload packages to be built for wheezy or
squeeze nor host them.
- We need to provide a set of packages that someone who is basing their
environment on squeeze can be assured will work.
- We will be maintaining backported dependency packages from now until
the cows come home.

>> - Lack of collaboration. Related to the above, but slightly different.
>> Since the Rackspace internal deployment team have to maintain their own
>> packages, it means they are not using the published packages and we are
>> not benefiting from their real-world deployment experience.
>> Additionally, the Cloudbuilders consulting team doesn't use our
>> published packages, and similarly we aren't really benefiting from them
>> directly working with us on packaging.
> 
> Rackspace isn't doing their own packaging because of (lack of) Debian
> support. If they did, they'd have realised almost immediately that the
> packages actually build on Debian. They're doing it because there'll
> supposedly be differences in the behaviour Ubuntu (or any other distro
> for that matter) will want and what Rackspace will want from the
> packages. I've becried this divergence countless times, but to no
> avail.

AFAIK, the packages do _not_ build on debian, nor have they for a while.

Actually, this is exactly the point - we are not building them on
debian, so we do not notice when they break. That's part of the point of
a CI system - if it isn't an output target, no one will realize when
they break it.

> That cloudbuilders aren't using the packages either.. I don't really
> know what to say about that.
> 
>> - PPAs are Async. This makes integration testing harder.
> 
> PPAs were chosen because
> a) they are almost exactly identical to the builders for Ubuntu
> proper, so the on-commit-to-trunk builds serve as way to know
> immediately if we introduce a change that will break when uploaded to
> Ubuntu proper. So, they're the closest we'll get to integration tests
> for real Ubuntu builds,
> b) we don't have to maintain them, and

This was a concern when they were chosen. Now? Not a big deal. We
maintain lots of things at this point.

> c) adding a PPA as a source to an Ubuntu system is incredibly
> straight-forward and a very well-supported operation.

So is adding another repo. add-apt-repository "deb
http://packages.openstack.org/trunk main" works just fine. Actually, the
fact that add-apt-repository only does the key fetching for PPAs and not
for other repos is sort of ridiculous and in-bred, but I've bitched at
canonical enough for one day (don't get me started on the broken
2.6.6-3~ thing)

>> If it's building them locally, and that's what's being tested, why
>> wouldn't that be what's published?
> 
> Because of c) above and because PPAs are trustworthy. There's no way
> you can sneak extra packages into the chroot or otherwise compromise
> its integrity, there's no way you can reach the internet from the
> builders (and vice versa), and they're built in a trusted environment
> (run by people who are used to managing this exact sort of
> infrastructure).

This is not unique to PPAs. Our repo can be just as trustworthy. In
fact, we're shipping software which runs large portions of your data
center, so if you don't trust us to make it, well, you're sort of screwed.

Same as our current unit-test runners ... the system is locked down, the
dev can't sneak stuff in.

BTW - the "can't reach the internet from the builders" is only really a
concern in a project like Debian and Ubuntu where you have random people
uploading random un-reviewed things. In our system, where things _ONLY_
hit package builders as a result of peer review, we should actually be
in better shape. (because we don't have to solve the general case like
they do)

>> - Divergent dev process from the rest of OpenStack. The PPB just
>> recently voted that all OpenStack projects must use the same set of
>> tooling. Now granted, packaging is not a core project, so we could get
>> legalistic and claim that we're not required to comply - but that seems
>> a little disingenuous - especially since I was a strong proponent of the
>> one-set-of-tooling model. Additionally, since we have a strong code
>> review/testing system that's integrated in to our build and deployment
>> system, it seems very odd not to take advantage of it. (having new
>> packages created triggered by both approval of packaging branch changes
>> AND new upstream tarballs seems kindof ideal, no?)
> 
> The packaging effort is far, far from an OpenStack-only operation.
> It's a shared effort with the Ubuntu team. We chose tools and
> processes for packaging that lined up with what is customary in Ubuntu
> in an effort to build confidence in the tools and automation we built.
> It maps very, very well with how things are usually done in Ubuntu.

It does map very very well with how thing are usually done in Ubuntu. It
does _NOT_ map very very well with how things are (coming to be) done in
OpenStack.

As I mentioned earlier (and this comes from someone with a great deal of
love for Ubuntu) Ubuntu's goal and needs are not necessarily ours.
Ubuntu devs are primarily focused on release+1 (because that's their
job) WE are primarily focused on release, release-1 and last-LTS. This
makes things like 2.6.6-3~ thing have very different emphasis and
meaning depending on which side of the fence you stand.

So my FIRST concern is making sure that the software that we produce
meets the needs of our community, our devs and our users. My second
concern is making things easier for Ubuntu.

If the Ubuntu devs decide that they cannot work with OpenStack tooling,
that's fine - although I will be very sorry to see them go. I'm more
concerned with getting packages produced from us that match the needs of
people actually deploying and using OpenStack on a regular basis. Right
now, that means notmyname, pvo, anotherjesse, jmckenty

>> - Not integrated with CI. PPA build errors are not reported inside of
>> our CI system where all of the rest of our errors are reported.
>> Additionally, those build errors are not reproducible by devs locally,
>> as they cannot spin up a launchpad PPA builder.
> 
> I don't understand.
> a) Building a chroot and setting up sbuild is a matter of installing
> one pacakge and running one command,

Still doesn't get you exact same as a PPA builder. The PPA buider has
networking turned off, which confused the sphinx build a while back
until we figured out what was going on. The PPA build also has not only
the packages inside of itself, but also other build-dep repos it can
use. Making sure you get a chroot with the right repos takes a decent
amount of knowledge about launchpad and ppas, actually.

> b) further up you argued that building packages ourselves is pretty
> much identical to building in a PPA and further down you say you have
> puppet magic to create a build environment for devs. Those two things
> put together sure sound to me like they can get infinitesimally close
> to the same sort of environment as you see in Launchpad,

Yes. Same sort of env.

> c) it's going to be exactly as difficult/easy to create a build
> environment that mimicks what you're doing in Jenkins, isn't it?

Nope, not really. That's pretty darned simple. (right now it's like,
three commands, I can definitely lower the bar there)

Also, in jenkins I can set up parameterized builders which devs can
submit code branches to and have them build in the actual jenkins env.
That is MUCH harder to accomplish in Launchpad.

More importantly, though, is the fact that all of the build output in
the Launchpad PPA is hidden.

Didja know that Glance hasn't built for Maverick or Lucid in a bit? That
python-novaclient failed all of its latest PPA builds? No? Of course not
- it's not in Jenkins, where all of the rest of the success/failure
reporting is.

For the same reason that I unreasonably yelled at Dan Prince at the last
ODS for having built a parallel CI system that wasn't part of our CI
system, the fact that we have important parts of our infrastructure
building elsewhere with no reporting back into our centralized build
system is becoming more and more singularly unacceptable.

>> - Lack of RPM support. Turns out RPM distros are still kind of popular,
>> and at least one of our major contributors (hi, NTT) rolls out on
>> RHEL/CentOS, and one of our partners who is going to be contributing
>> testing hardware (MSFT/Novell) is associated with a distro (SuSE) that
>> is RPM based. We're a few steps away from this, but it warrants mentioning.
> 
> This probably warrants a separate discussion. As pointed out further
> up, Ubuntu was chosen in part because it's very up-to-date. With a
> release every 6 months, we're never far behind in terms of access to
> dependencies in the most recent release, and since our release cycles
> line up, we can *ensure* availability of the dependencies we require
> in the upcoming release.  If
> RHEL/CentOS/ScientificLinux/WhatEverOtherFlavourOfRedHatYouPrefer
> becomes a supported target, will this block stuff from landing in Nova
> until RHEL catches up with the rest of the world in terms of libraries
> upon which we depend?

Again - we're going to be maintaining repos and backports. RHEL is a
good example, but the same applies to Ubuntu - the distros will NEVER
keep up with our needs, because their needs are different.

>> - Murky backported deps. Some of the backported deps in the PPAs came
>> from a dir on my laptop. Some are from Theirry's laptop. Some are from
>> Soren's laptop. There is neither documentation on how they got there,
>> how we decided to put them there, how we built them or how to continue
>> to collaborate on them.
> 
> Sorry, what? They're all built in PPAs. That's about as clean-room as
> it gets. Everyone is free to go and see and inspect them. The diff
> between that version upon which the backport is based and what got
> uploaded is trivially easy to extract. Documenting how it's done
> sounds pointless. Anyone who's familiar with Ubuntu/Debian packaging
> will find no surprises. If people want to look at OpenStack code, it's
> not our job to document what Python is all about. If people want to
> see how I backported something, it's not my job to teach them
> packaging first.

nono. Not how they're built - where the source tree is that you (or I)
built the source package from that is built in the builders that aren't
controlled by the Jenkins that controls everything else we do.

If you (or I) do a one-off backport, we then upload a source package and
then, often, don't shove the code anywhere. That's because a lot of the
time it's easiest just to do dget, hack version, debuild -S, dput. We
have no way of knowing if there is intention of further work, or where
the canonical location of that work is. If I wanted to do an additional
change to python-kombu, should I just grab the dsc from our PPA and
upload a new one? Or did you already import that in to a bzr repo? Who
knows? How can they find out? (and yeah, UDD was supposed to solve this
- except it didn't)

>> - Direct injection to Ubuntu not via Debian. The publically stated
>> best-practice for things entering Ubuntu is to be added to/uploaded to
>> Debian and then synced, unless there is some compelling reason not to do
>> this. In this case, we had Ubuntu devs and no Debian devs in the core,
>> so we keep improving packages and adding packages to Ubuntu but are not
>> doing the same to Debian.
> 
> For the umpteenth time, this is not about OpenStack. It's about the
> dependencies. Getting OpenStack itself into Debian was never the hard
> part. Getting all the dependencies into Debian was. Not just libraries
> that weren't in Debian at all (like gflags, which I got uploaded to
> Debian and synced into Ubuntu afterwards), but existing libraries in
> Debian that we needed either a newer version of or a patch applied to.
> In Ubuntu we have the power to upload these newer versions or patches
> or whatever, by implicitly accepting responsibility for it if it
> should happen to cause problems. We have no way to even begin to
> ensure that we can do this in Debian. Debian packages have
> maintainers. The maintainers decide whether to accept a patch or not.
> If they're not comfortable with a patch, rejecting it is entirely
> their prerogative (and for good reason! They're the ones who
> ultimately have to support it). In every Ubuntu release since we
> started work on OpenStack, I've added at least one patch to libvirt to
> support something we needed for OpenStack. I've submitted them
> upstream (to libvirt) as well, but we can't stop everything while
> waiting for libvirt to make a release, the Debian maintainer to pick
> it up, and for Ubuntu to sync it.

Yes. I understand all of that ... but if we START with debian-oriented
packages in our own repo and send patches upsteam (as you've already
been doing) we get Debian support as a target. Those same patches can
then also be applied to our packages for oneiric and maverick and natty
and lucid, which can also be uploaded to our repo. And then, as we
either have time or inclination or motivation from downstream, we can
get libraries updated in both debian and ubuntu.

Part of the problem here is that we have an institutional process which
depends on a special quality of one of the OpenStack devs (namely, the
ability of you or ttx to upload directly in to Ubuntu) That's FANTASTIC,
but it can't really be the basis of ongoing OpenStack process, as it is
not a process that someone else can just pick up.

> Debian support is a massive undertaking and I'm thrilled to see it
> happen, but I don't believe Debian is suitable as a reference platform
> (if that means that failure there can block anything at all).

I don't think you and I have the same view of the words "reference
platform", but I believe I've already covered that.

>> - PPA Fragmentation. The PPB voted that we are one project, yet we have
>> a separate PPA for each project.
> 
> For trunk builds, yes. Two reasons:
> a) Avoid cascading failures: Nova depends on glance which depends on
> Swift. It would be a shame if a bug in Swift had to bring everything
> else to a screeching halt.

But it's already bringing things to a halt in terms of actual
deployments. Jesse just said to Jim and I yesterday as we were doing
integration test setups "oh yeah, this stuff breaks all the time"

The point of a good CI system is to head that off at the pass. Nothing
about our current process should allow a bug in Swift to bring DEV to a
screeching halt, but if it causes integration tests to bork, well, then
people probably want to know about it, no?

> b) If someone wants to help test the bleeding edge of Swift, but isn't
> really interested in whatever breakage we introduce in Nova, having
> separate PPA's makes it straightforward to just test one of them.

I've heard this as a theoretical several times, but I haven't heard the
actual call for "I REALLY want to install bleeding edge swift, with
stable nova" on the same machine. In fact, the chances of running both
swift and nova on the same box in a real environment are pretty low.

Interface libraries, on the other hand, sure - except that new interface
libraries should ALWAYS be able to talk to old versions of services.

> For final releases, we put everything in the same PPA. I've written
> tools to keep track of divergence between the PPA's and to bring them
> in sync so the final consolidation doesn't hold any surprises. Putting
> everything in the same PPA is a trivial change, though.

Where are these tools? Where do they run? When? From what source do they
bring PPAs in sync? How do they decide which version of eventlet needs
to live in the nova and the swift PPA? From were do they get that
version of eventlet?

Monty