Open Stack

> capping stable branch requirements so we only have to worry about uncapped dependencies on master

+1 this is a sensible idea not just from the POV of grenade testing,
but general stability of the stable releases.  As per Joe's link [5]
we can see 3rd party libraries may make breaking changes in minor
releases, so we should pin to the exact version (ie ==X.Y.Z, rather
than saying we'll allow increases in the Z for bugfixes).

> how about the deep dependency?

We should pin transitive dependencies too. If we don't then we're open
to new 3rd party releases breaking things just as before.

  mg

On Thu, Jan 15, 2015 at 9:51 AM, ZhiQiang Fan <aji.zqfan at gmail.com> wrote:
> +1
>
> how about the deep dependency? for example, we depends package A, and pin
> it, but A->B->C, then B and C are not pined since they are not directly
> depended, then what should we do? pin everything?
>
> On Thu, Jan 15, 2015 at 5:35 PM, Joe Gordon <joe.gordon0 at gmail.com> wrote:
>>
>> Eventlet released 0.16.1 on 2015-01-14 [0], which removed a deprecated API
>> that nova stable/* still used. This caused nova-compute in stable/juno and
>> stable/icehouse to crash thus breaking grenade on master. In 24 hours this
>> bug caused 671 grenade jobs to fail[1]!
>>
>> After some quick debugging of this cryptic error suppressing stacktrace
>> [2] we got to the real stacktrace:
>> " File "nova/virt/libvirt/__init__.py", line 15, in <module>
>>
>>     from nova.virt.libvirt import driver
>>   File "nova/virt/libvirt/driver.py", line 48, in <module>
>>     from eventlet import util as eventlet_util
>>
>> ImportError: cannot import name util" [3]
>> and capped the eventlet version on stable/* [4]
>>
>> As dependency bugs go this was a pretty typical one, so why am I writing
>> this? Because we knew about this bug before it hit the gate, yet it was
>> still an issue. The util module was removed in 0.16.0 but 'sneaked into' the
>> build [5] so 0.16.1 fixed that. Before 0.16.1 was released this bug only
>> impacted people who installed eventlet 0.16.0 from source and not from pip.
>> Luckily someone did this and filed a bug [6] and a fix for this was landed
>> on master on January 7th, and by the 10th the fix was backported to
>> stable/juno and ready [7].  In the mean time, we had an unusually bad week
>> dealing with new library versions; boto 2.35.0 was released and broke master
>> and stable/juno [8] and sqlalchemy-migrate broke glance on master and stable
>> as well [9].  And by the time these issues were fixed, and the tests would
>> pass, eventlet 0.16.1 was already out. Unfortunately, there are a very small
>> number of people who work on fixing dependency issues, some of them were
>> traveling and the rest were over worked, so figuring out what went wrong
>> with all the new packages came down to a handful of overworked people.
>>
>> So how could we have avoided this problem? By capping stable branch
>> requirements so we only have to worry about uncapped dependencies on master.
>> Capping stable branches has been previous discussed but no action has been
>> taken. So going forward I propose we pin all requirements, including
>> transitive, on stable branches. This way the release of new dependencies
>> cannot automatically break stable branches and thus break grenade on master.
>>
>> I think we can implement this using `pip freeze` to get a list of all the
>> installed libraries and `pip install -r --no-deps` to install the specific
>> package versions.  I am not sure how best to handle package versions being
>> removed from pypi though. Or at the very least we can cap requirements in
>> global requirements and at least reduce the number of packages we install
>> uncapped versions of.
>>
>> Thoughts?
>>
>> best,
>> Joe
>>
>> [0] https://pypi.python.org/pypi/eventlet/0.16.1
>> [1] http://status.openstack.org/elastic-recheck/#1410626
>> [2]
>> http://logs.openstack.org/98/115998/59/check/check-grenade-dsvm/e57eb40/logs/old/screen-n-cpu.txt.gz
>> [3] http://paste.openstack.org/show/157558/
>> [4]
>> https://review.openstack.org/#/q/I4bbbeb5bf9c22ed36f5c9a74fec6b487d2c15697,n,z
>> [5] https://github.com/eventlet/eventlet/releases/tag/v0.16.1
>> [6] https://bugs.launchpad.net/nova/+bug/1407685
>> [7]
>> https://review.openstack.org/#/q/Idbb9d2b53829dae0e807cd1260dee3dce155d5f3,n,z
>> [8] https://bugs.launchpad.net/nova/+bug/1408987
>> [9] https://bugs.launchpad.net/glance/+bug/1410494
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
> --
> blog: zqfan.github.com
> git: github.com/zqfan
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>