Open Stack

Hi Clint,

Thanks for your contribution to this thread.

On 06/04/2015 10:35 PM, Clint Adams wrote:
> On Wed, Jun 03, 2015 at 04:30:17PM -0400, Sean Dague wrote:
>> The closer we can get logic about what a service should look like on
>> disk back into that service itself, the less work duplicated by any of
>> the installers, and the more common OpenStack envs would be. The fact
>> that every installer / package needs to copy in a bunch of etc files
>> (because the python packages don't do it) has always seemed rather odd
>> to me.
> 
> I agree with this, and given the disparate and seemingly contradictory
> goals driving these discussions, I think it will be exceedingly
> difficult to make everyone happy.

I don't think anyone involved in the packaging of OpenStack has
expressed disparity or contradiction. Quite the opposite: it is my
strong opinion that all parties involved are on the same page.

> So here's my suggestion:
> 
> 1. Maintain all important data for packaging as first-class members of
>    the respective repositories. Initscripts, systemd service files,
>    licensing (SPDX?), and so on should be in the master branch of each
>    project.

Are you here saying we should move startup scripts upstream, and not on
the packaging repos? If so, that's a bad idea. Let me explain.

The init scripts used to be hard to maintain because they were many, but
since Debian & Ubuntu are using automatic generation out of a tiny
template (with sysv-rc, systemd and upstart all supported), this is a
problem solved.

If individual projects are getting into the business of publishing their
own startup scripts, I doubt that we would even use them, because having
a central place to patch all of the startup scripts at once (ie:
openstack-pkg-tools) is much better than having to maintain each and
every startup script by hand.

As for the licensing, I agree here. I have multiple times express my
opinion about it: the project as a hole needs to make some progress, as
it's nearly impossible for downstream distributions to second guess who
is the copyright holders (please pay attention: copyright holders and
licensing are 2 separate things...).

Though SPDX?!? Do you know the famous quote (Jay Pipes) "Get your dirty
XML out of my Json" ? :) And there's also the fact that Debian uses a
different parseable format. Not sure what the RPM folks use, but I
suppose that's embedded in a .spec file. Also, all of OpenStack is
mostly using the Apache-2.0 license, the licensing issues are with
(build-)dependencies, and that, we can't solve it.

>    Just enough glue should be present such that functional
>    packaging can be programmatically generated from HEAD with debdry
>    or similar tooling, and this is how test jobs should operate (f.ex.: run
>    debdry, mangle version number to something unique, build package in
>    chroot or equivalent, store output for use in other testing).

I already use pkgos-debpypi (from openstack-pkg-tools) to automate a big
chunk of the Python module packaging. Though the automated thing can
only drive you so far. It wont fix missing dependencies in
requirement.txt, wrong lower bounds, SSLv3 removal related issues, and
all of this kind of issues which we constantly need to address to make
sure all unit tests are passing. The thing is, unit & functional tests
in OpenStack are designed for Devstack. I supposed you already saw the
"It worked on Devstack" XKCD t-shirt... Well, I'm sure all package
maintainers of OpenStack have this in mind every day! :)

As you are a DD, you know it too: getting a correct debian/copyright
also represent some work, which cannot be automated, or the FTP masters
will really hate you. :P

Plus there's also the fact that sometimes, distributions don't use the
matching versions. For example, Debian Jessie was released with Django
1.7 and OpenStack Icehouse, and I had to work on patching Horizon to
make it work with it (at the time, Horizon didn't work with something
higher than 1.6). The same thing happened with SQLAlchemy.

This has slowed down a little bit over the years, but we also used to
spend most of our time simply packaging new dependencies. We already
have a big amount of redundancy (alembic vs sqlalchemy-migrate, WSGI
frameworks, nose vs testr, pymysql vs mysqlclient vs mysql-python, you
name it...). And each new project, with its specificities, bring new
dependencies. I used to say that 80% of the packaging work is spent
there: packaging new stuff. These days, it has a bit shifted to
packaging oslo and client libs, which is a good thing (as they are
respecting a standard, so we have less surprise). Though what actually
represent an OpenStack release has grown bigger. After Jessie was
released, uploading all of Kilo to Sid took me about 3 or 4 days, and
maybe about the same amount of time to upload it to the official Jessie
backports (all done in dependency order, with as few dependency breakage
as possible...).

I really hope that the effort on having a gate on the lower bounds of
our global-requirements.txt will help here. But it wont solve all issues.

So, that's where all the time is spent. Yeah, we can do some level of
automation. But believing we can do *ALL* with automation is often a
naive thought of people who never tried. I welcome you to join the
packaging effort and enhance the toolset we have so we get closer to
that goal though! :)

> This way OpenStack could
> 
> -  produce and consume its own package artifacts for testing changes of
>    varying complexity
> -  get early warning of changes which will break packaging, insanity in
>    dependency graphs, and so on

That's some of the goals of this proposal, yes.

> Distributors could
> 
> -  clone the upstream repos and branch to add any special tweaks, then
>    potentially run the same automation to get source/binary packages
> -  collaborate upstream to fix things of common utility

At least for Debian & Ubuntu, we hope to push absolutely all of our
packaging upstream, and only rebuild and publish the final result for
the stable branches (well, I can only speak for myself, so I'll say that
this is what I want to do in Debian...). Even if we can do this for all
but the core projects, then that's a huge win (since it represents the
biggest chunk of the work).

I'll do as much as possible so that Mirantis OpenStack also follows
upstream packaging branches too, and also contributes upstream.

Cheers,

Thomas Goirand (zigo)

P.S: Will you get involved in the packaging for Debian as well?