[openstack-dev] Dependency version pinning [was Re: Pyparsing 2.0 breaking jenkins jobs]

Joshua Harlow harlowja at yahoo-inc.com
Wed Mar 6 08:44:38 UTC 2013


The dist-utils thread was a little scary, thx mark for doing that. It's an eye opener for me, and hopefully something that we can make sure is done right. I don't want to have to make a super tarball with all openstack and dependencies just so it works. That would be sad...

I've avoided venv in general, since its a model that isn't used in yahoos production deploys (and openstsck provides vms so a vm is a much more realistic and just as fast "venv" to me for dev work i do). Even so I really want the multiple simultaneous installs (java and others are figuring this out, python need to also, not acceptable to just say use venvs since who actually deploys that way).

Yahoo has this similar tension between making dev installs fast (aka the venv) and having packages with strong and repeatable dependencies for production usage, this venv debate seems very similar.

Sent from my really tiny device...

On Mar 5, 2013, at 3:09 PM, "Mark McLoughlin" <markmc at redhat.com> wrote:

> On Sun, 2013-03-03 at 16:23 +0000, Mark McLoughlin wrote:
>> On Thu, 2013-02-28 at 15:43 +0000, Mark McLoughlin wrote:
>>> On Thu, 2013-02-28 at 11:59 +0000, Mark McLoughlin wrote:
>>>> On Wed, 2013-02-27 at 14:43 -0500, Doug Hellmann wrote:
>>>>> On Wed, Feb 27, 2013 at 1:44 PM, Monty Taylor <mordred at inaugust.com> wrote:
>>>>>> On 02/27/2013 01:38 PM, Sean Dague wrote:
>>>> 
>>>>>>> Various respected python community members, it would be really awesome
>>>>>>> to solve this as a root issue, get pypi to not allow uploads that don't
>>>>>>> specify, and have it be part of the installer. Would be good for
>>>>>>> everyone, not just OpenStack.
>>>>>> 
>>>>>> Python is talking about serveral major upgrades to PyPI - I'll toss this
>>>>>> one in the mix.
>>>>> 
>>>>> +1 Participating in the packaging summit at PyCon in a few weeks should go
>>>>> a long way.
>>>> 
>>>> I've just sent a slightly desperate plea to psf-members. See attached.
>>> 
>>> Redirected here:
>>> 
>>>  http://mail.python.org/pipermail/distutils-sig/2013-February/020030.html
>>> 
>>> Nick Coghlan points out PEP426 which is relevant:
>>> 
>>>  http://www.python.org/dev/peps/pep-0426/
>> 
>> Another follow-up on distutils-sig attached.
>> 
>> The conclusion I'm coming to from the discussion is that by and large
>> Python library maintainers try to avoid incompatible changes and they
>> have commitments to stability built into their versioning schemes. We
>> need to understand each library's versioning scheme and do e.g.
>> 
>>  sqlalchemy>=0.7.8,<0.8
>> 
>> because we know that a minor version number change in sqlalchemy is used
>> to warn people of an incompatible change whereas other libraries we can
>> do:
>> 
>>  foo>=1.2.3,<2
>> 
>> because we know they're using the semantic versioning and will only make
>> incompatible API changes in version 2 whereas still others we can do:
>> 
>>  bar>=2.2
>> 
>> because we know they will introduce a new 'bar2' library if they need to
>> make incompatible changes.
>> 
>> Not fun, but way better than pinning to specific versions IMHO.
> 
> To follow up some more ... the distutils-sig thread is really quite
> enlightening IMHO:
> 
>  http://mail.python.org/pipermail/distutils-sig/2013-March/thread.html#20054
> 
> I've some notes below, but the tl;dr is that unless we "cap" our
> requirement ranges we're going to be hit by incompatible updates. Each
> project has a different policy on how their version number and we'll
> actually need to talk to each project we use about their policy.
> 
> What we want is e.g.
> 
>  sqlalchemy>=0.7.8,<0.8
> 
> but only after talking to sqlalchemy to confirm that they will bump the
> minor number iff they are making incompatible changes.
> 
> What we need is a combination of (a) the openstack/requirements repo,
> (b) capping our requirements as described above and (c) a PyPI mirror
> that is only refreshed if the refresh doesn't break the gate
> 
> Cheers,
> Mark.
> 
> Notes on PEP426:
> 
>  - A trend towards semantic versioning. Any project that we know is 
>    using this versioning scheme, we should cap our requirement range 
>    to avoid incompatible updates e.g.
> 
>        foo>=1.2.3,<2.0
> 
>  - A new operator (sometimes known as the 'spermy' operator in Ruby) 
>    is being introduced. In PEP426, it's expressed as:
> 
>      Requires-Dist: foo (1.2)
> 
>    but in pip-requires, it's likely to be:
> 
>      foo~>1.2
> 
>    any version in the 1.x series greater than 1.2 satisfies this. If 
>    you wanted 1.2.3 or later but also to avoid 2.x, you'd do:
> 
>      foo~>1.2,>=1.2.3
> 
>  - Not all projects will use semantic versioning, but it is expected 
>    that all projects have some policy on how their version number will 
>    reflect incompatible updates. There is currently no metadata that 
>    projects can use to publish this information, so we actually need 
>    to talk to each upstream individually.
> 
>  - There is currently no sane support for installing multiple 
>    incompatible versions of the same library except "use venv". This 
>    isn't a problem for OpenStack per-se, but it is a problem for 
>    distributors. Nick Coghlan had some ideas for how this could be 
>    improved in future:
> 
>      http://mail.python.org/pipermail/distutils-sig/2013-March/020082.html
> 
>  - There will be support for a library saying which version of Python 
>    it requires. Presumably that would prevent us auto-updating to a 
>    library which doesn't support 2.6/2.7. However, dropping 2.6/2.7
>    support should surely be seen as an incompatible change and the 
>    project's version number increased so ... it's probably covered 
>    anyway.
> 
>  - All the way through this I've seen mentions of work needed in PyPI 
>    and that's completely separate from PEP426 etc. We need to get up 
>    to speed there and follow what's happening on catalog-sig.
> 
>  - I'd never heard of buildout before and still haven't looked in 
>    detail. Maybe it's relevant to OpenStack, maybe not.
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



More information about the OpenStack-dev mailing list