Open Stack

Hi

On Mon, 2013-06-10 at 20:50 +1200, Robert Collins wrote:
> We have a hole in our otherwise pretty robust dependency management :
> binary packages. At the moment, the most reliable way to get a dev
> environment working is to either copy from a wiki page like
> https://wiki.openstack.org/wiki/DependsOnUbuntu, or to run devstack
> once :).
> 
> This is something I've said before we should automate... I was
> starting to look at tempest in the weekend, and yet again running up
> the curve of 'what packages are needed for this specific project',
> when I hit my tolerance threshold and thus, rage-fueled-code.
> 
> https://github.com/rbtcollins/bindep is a draft project which would
> allow capturing in machine readable form the dependencies needed for
> our CI environments, for developers, and for binary deployment. It
> *isn't* a replacement or wrapper for tools like dpkg and rpm : there
> are good tools for that already. What it would let us do is record
> within each project the dependencies the project actually has for each
> use case : no more patching devstack to educate it about new binary
> dependencies.

Having metadata in each project describing its non-python requirements
is definitely a good thing. As is having some way of turning that
metadata into a "check these deps are satisfied" or "install these deps"
disto-specific command.

I think it's great you've kicked this off - it's a great starting point
for finding a workable solution. I'm actually kinda surprised we never
had e.g. tools/install-deps.sh type scripts long ago.

The first thing I notice is that the list of requirements is a list of
debian packages and versions. Obviously, anyone who doesn't use a debian
distro wouldn't like to have to cater first for debian and then for
their preferred distro when adding a dep. I think the project has gone
beyond having a single first class distro, so I think this needs
improving.

(I've an obvious bias here - but the concern honestly does go beyond
just me)

What might work is if we built in a dependence on the
openstack/requirements repo. Like for python deps, we'd use that repo to
describe the unified set of dependencies for all of openstack and
individual projects would specify a subset of those dependencies.

We could also use openstack/requirements to maintain a mapping of distro
agnostic requirement name/version to distro specific package
name/version. Maybe bindep would look use the distro mapping table to
apply other-requirements.txt or maybe we'd have a utility to copy the
appropriate parts of the mapping table into other-requirements.txt.

Bikeshed - other-requirements could be system-requirements.txt or
distro-requirements.txt ?

Anyway, just some ideas ...

> I think this could live very happily as an oslo component, like pbr
> does. It provides a small CLI, and a Python library which can be used
> from things like pbr or CI scripts, if they want to layer on it.

It could come under oslo, but I'd like to see it gain a good deal of
traction first. Obviously adding other-requirements.txt to projects
would be a good first step. Using it in devstack would be next?

Basically, I think this could move to stackforge and we could see how it
evolves before moving it to oslo.

Why wait?

Well, if we include it in oslo, it will imply some interface stability
guarantees that will make it harder to evolve the idea.

Also, oslo-core is already finding the review workload pretty hard to
keep on top of ... so I'm reluctant to pull in another project which is
very new and isn't in use anywhere yet. We have lots of work with our
core focus on the project's copy-and-paste technical debt.

So, again - I'm up for including it oslo at some point, but let's get it
in use in a bunch of places first.

Cheers,
Mark.