Open Stack

+1

how about the deep dependency? for example, we depends package A, and pin
it, but A->B->C, then B and C are not pined since they are not directly
depended, then what should we do? pin everything?

On Thu, Jan 15, 2015 at 5:35 PM, Joe Gordon <joe.gordon0 at gmail.com> wrote:

> Eventlet released 0.16.1 on 2015-01-14 [0], which removed a deprecated
> API that nova stable/* still used. This caused nova-compute in stable/juno
> and stable/icehouse to crash thus breaking grenade on master. In 24 hours
> this bug caused 671 grenade jobs to fail[1]!
>
> After some quick debugging of this cryptic error suppressing stacktrace
> [2] we got to the real stacktrace:
> " File "nova/virt/libvirt/__init__.py", line 15, in <module>
>
>     from nova.virt.libvirt import driver
>   File "nova/virt/libvirt/driver.py", line 48, in <module>
>     from eventlet import util as eventlet_util
>
> ImportError: cannot import name util" [3]
> and capped the eventlet version on stable/* [4]
>
> As dependency bugs go this was a pretty typical one, so why am I writing
> this? Because we knew about this bug before it hit the gate, yet it was
> still an issue. The util module was removed in 0.16.0 but 'sneaked into'
> the build [5] so 0.16.1 fixed that. Before 0.16.1 was released this bug
> only impacted people who installed eventlet 0.16.0 from source and not from
> pip. Luckily someone did this and filed a bug [6] and a fix for this was
> landed on master on January 7th, and by the 10th the fix was backported to
> stable/juno and ready [7].  In the mean time, we had an unusually bad week
> dealing with new library versions; boto 2.35.0 was released and broke
> master and stable/juno [8] and sqlalchemy-migrate broke glance on master
> and stable as well [9].  And by the time these issues were fixed, and the
> tests would pass, eventlet 0.16.1 was already out. Unfortunately, there are
> a very small number of people who work on fixing dependency issues, some of
> them were traveling and the rest were over worked, so figuring out what
> went wrong with all the new packages came down to a handful of overworked
> people.
>
> So how could we have avoided this problem? By capping stable branch
> requirements so we only have to worry about uncapped dependencies on
> master. Capping stable branches has been previous discussed but no action
> has been taken. So going forward I propose we pin all requirements,
> including transitive, on stable branches. This way the release of new
> dependencies cannot automatically break stable branches and thus break
> grenade on master.
>
> I think we can implement this using `pip freeze` to get a list of all the
> installed libraries and `pip install -r --no-deps` to install the specific
> package versions.  I am not sure how best to handle package versions being
> removed from pypi though. Or at the very least we can cap requirements in
> global requirements and at least reduce the number of packages we install
> uncapped versions of.
>
> Thoughts?
>
> best,
> Joe
>
> [0] https://pypi.python.org/pypi/eventlet/0.16.1
> [1] http://status.openstack.org/elastic-recheck/#1410626
> [2]
> http://logs.openstack.org/98/115998/59/check/check-grenade-dsvm/e57eb40/logs/old/screen-n-cpu.txt.gz
> [3] http://paste.openstack.org/show/157558/
> [4]
> https://review.openstack.org/#/q/I4bbbeb5bf9c22ed36f5c9a74fec6b487d2c15697,n,z
> [5] https://github.com/eventlet/eventlet/releases/tag/v0.16.1
> [6] https://bugs.launchpad.net/nova/+bug/1407685
> [7]
> https://review.openstack.org/#/q/Idbb9d2b53829dae0e807cd1260dee3dce155d5f3,n,z
> [8] https://bugs.launchpad.net/nova/+bug/1408987
> [9] https://bugs.launchpad.net/glance/+bug/1410494
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
blog: zqfan.github.com
git: github.com/zqfan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150115/4666ff27/attachment.html>