Open Stack

On Sun, 2013-03-03 at 16:23 +0000, Mark McLoughlin wrote:
> On Thu, 2013-02-28 at 15:43 +0000, Mark McLoughlin wrote:
> > On Thu, 2013-02-28 at 11:59 +0000, Mark McLoughlin wrote:
> > > On Wed, 2013-02-27 at 14:43 -0500, Doug Hellmann wrote:
> > > > On Wed, Feb 27, 2013 at 1:44 PM, Monty Taylor <mordred at inaugust.com> wrote:
> > > > > On 02/27/2013 01:38 PM, Sean Dague wrote:
> > > 
> > > > > > Various respected python community members, it would be really awesome
> > > > > > to solve this as a root issue, get pypi to not allow uploads that don't
> > > > > > specify, and have it be part of the installer. Would be good for
> > > > > > everyone, not just OpenStack.
> > > > >
> > > > > Python is talking about serveral major upgrades to PyPI - I'll toss this
> > > > > one in the mix.
> > > > >
> > > > 
> > > > +1 Participating in the packaging summit at PyCon in a few weeks should go
> > > > a long way.
> > > 
> > > I've just sent a slightly desperate plea to psf-members. See attached.
> > 
> > Redirected here:
> > 
> >   http://mail.python.org/pipermail/distutils-sig/2013-February/020030.html
> > 
> > Nick Coghlan points out PEP426 which is relevant:
> > 
> >   http://www.python.org/dev/peps/pep-0426/
> 
> Another follow-up on distutils-sig attached.
> 
> The conclusion I'm coming to from the discussion is that by and large
> Python library maintainers try to avoid incompatible changes and they
> have commitments to stability built into their versioning schemes. We
> need to understand each library's versioning scheme and do e.g.
> 
>   sqlalchemy>=0.7.8,<0.8
> 
> because we know that a minor version number change in sqlalchemy is used
> to warn people of an incompatible change whereas other libraries we can
> do:
> 
>   foo>=1.2.3,<2
> 
> because we know they're using the semantic versioning and will only make
> incompatible API changes in version 2 whereas still others we can do:
> 
>   bar>=2.2
> 
> because we know they will introduce a new 'bar2' library if they need to
> make incompatible changes.
> 
> Not fun, but way better than pinning to specific versions IMHO.

To follow up some more ... the distutils-sig thread is really quite
enlightening IMHO:

  http://mail.python.org/pipermail/distutils-sig/2013-March/thread.html#20054

I've some notes below, but the tl;dr is that unless we "cap" our
requirement ranges we're going to be hit by incompatible updates. Each
project has a different policy on how their version number and we'll
actually need to talk to each project we use about their policy.

What we want is e.g.

  sqlalchemy>=0.7.8,<0.8

but only after talking to sqlalchemy to confirm that they will bump the
minor number iff they are making incompatible changes.

What we need is a combination of (a) the openstack/requirements repo,
(b) capping our requirements as described above and (c) a PyPI mirror
that is only refreshed if the refresh doesn't break the gate

Cheers,
Mark.

Notes on PEP426:

  - A trend towards semantic versioning. Any project that we know is 
    using this versioning scheme, we should cap our requirement range 
    to avoid incompatible updates e.g.

        foo>=1.2.3,<2.0

  - A new operator (sometimes known as the 'spermy' operator in Ruby) 
    is being introduced. In PEP426, it's expressed as:

      Requires-Dist: foo (1.2)

    but in pip-requires, it's likely to be:

      foo~>1.2

    any version in the 1.x series greater than 1.2 satisfies this. If 
    you wanted 1.2.3 or later but also to avoid 2.x, you'd do:

      foo~>1.2,>=1.2.3

  - Not all projects will use semantic versioning, but it is expected 
    that all projects have some policy on how their version number will 
    reflect incompatible updates. There is currently no metadata that 
    projects can use to publish this information, so we actually need 
    to talk to each upstream individually.

  - There is currently no sane support for installing multiple 
    incompatible versions of the same library except "use venv". This 
    isn't a problem for OpenStack per-se, but it is a problem for 
    distributors. Nick Coghlan had some ideas for how this could be 
    improved in future:

      http://mail.python.org/pipermail/distutils-sig/2013-March/020082.html

  - There will be support for a library saying which version of Python 
    it requires. Presumably that would prevent us auto-updating to a 
    library which doesn't support 2.6/2.7. However, dropping 2.6/2.7
    support should surely be seen as an incompatible change and the 
    project's version number increased so ... it's probably covered 
    anyway.

  - All the way through this I've seen mentions of work needed in PyPI 
    and that's completely separate from PEP426 etc. We need to get up 
    to speed there and follow what's happening on catalog-sig.

  - I'd never heard of buildout before and still haven't looked in 
    detail. Maybe it's relevant to OpenStack, maybe not.