Open Stack

On Sat, 2013-01-12 at 19:30 -0800, Monty Taylor wrote:
> Hey all!
> 
> Once again, we've kind of stalled out on our two-summit old project of
> common requirements across the projects. The initial repo is up at
> http://github.com/openstack/requirements but I think not much happens
> with it because working on it doesn't really get you anything right now.

Just to remind ourselves, here's what we discussed at the last summit:

https://etherpad.openstack.org/process-dependency-management

Thanks for prodding this again - I was starting to get really grumpy
with myself for not doing anything to help it along :)

> So I've got an idea for steps to move forward.
> 
> Step 0: Re-sync the contents of the files with the new state of reality

Take a look at:

  https://github.com/markmc/requirements/

Is that what you're thinking?

> Step 1: Drive updates to the OpenStack PyPI mirror from the requirements
> repo
> Step 2: Configure the Jenkins slaves to only talk to the PyPI mirror for pip
> Step 3: Add a step to the pypi mirror creation that pulls packages fresh
> from requirements into a new proposed state of the PyPI mirror, then
> tests all of the projects against that mirror and only promotes new
> packages into if those tests pass.
> 
> That's it. Done.
> 
> If we do through step 2, it means that before someone can add a new
> requirement to a project, it would have to go into
> openstack/requirements. If we do step 3, it'll trap for things like the
> sqlalchemy 0.8 upgrade issue. It also, as an approach, doesn't enforce
> that projects have to upgrade at any point, because previous versions of
> things will still be in our mirror (we don't purge it)

I think there's 3 things we want to catch:

  1) When whole new libraries are added as a dependency

  2) When the minimum version of a dependency is increased

  3) When a project wants to specify a version range that is 
     incompatible with another project's version range

I think what you're talking about will catch (1) and (3) but not (2).

To catch (2), I guess we'd need a separate pypi mirror with all the
minimum versions of dependencies and a set of jobs to run tests with
that mirror?

> Also - while you're all thinking about it - I've been considering
> getting the code we have in oslo-incubator setup that makes setup.py
> understand pip-requires and test-requires automatically upstreamed
> (seems like a thing we're not the only ones who would want it) The catch
> - those aren't the filenames most people use for those files. SO - what
> if we (and by we, I mean I, and by I, I mean a shell script that submits
> a bajillion reviews) renamed tools/pip-requires to requirements.txt and
> tools/test-requires to test-requirements.txt? I've already updated our
> mirror creation code to recognize both files. If people are cool with
> that, I can then work on trying to get a patch into distribute that
> would allow for that - which would mean less copied code for us.

Sounds good to me.

At the summit, I said I'd take a stab at resolving the differences
between the requires of the core projects so that the effective
requirements of the project as a whole were consistent. I've dumped some
notes about what I think needs fixing here:

  https://github.com/markmc/requirements/blob/master/issues.txt

Cheers,
Mark.