Open Stack

Eventlet released 0.16.1 on 2015-01-14 [0], which removed a deprecated API
that nova stable/* still used. This caused nova-compute in stable/juno and
stable/icehouse to crash thus breaking grenade on master. In 24 hours this
bug caused 671 grenade jobs to fail[1]!

After some quick debugging of this cryptic error suppressing stacktrace [2]
we got to the real stacktrace:
" File "nova/virt/libvirt/__init__.py", line 15, in <module>

    from nova.virt.libvirt import driver
  File "nova/virt/libvirt/driver.py", line 48, in <module>
    from eventlet import util as eventlet_util

ImportError: cannot import name util" [3]
and capped the eventlet version on stable/* [4]

As dependency bugs go this was a pretty typical one, so why am I writing
this? Because we knew about this bug before it hit the gate, yet it was
still an issue. The util module was removed in 0.16.0 but 'sneaked into'
the build [5] so 0.16.1 fixed that. Before 0.16.1 was released this bug
only impacted people who installed eventlet 0.16.0 from source and not from
pip. Luckily someone did this and filed a bug [6] and a fix for this was
landed on master on January 7th, and by the 10th the fix was backported to
stable/juno and ready [7].  In the mean time, we had an unusually bad week
dealing with new library versions; boto 2.35.0 was released and broke
master and stable/juno [8] and sqlalchemy-migrate broke glance on master
and stable as well [9].  And by the time these issues were fixed, and the
tests would pass, eventlet 0.16.1 was already out. Unfortunately, there are
a very small number of people who work on fixing dependency issues, some of
them were traveling and the rest were over worked, so figuring out what
went wrong with all the new packages came down to a handful of overworked
people.

So how could we have avoided this problem? By capping stable branch
requirements so we only have to worry about uncapped dependencies on
master. Capping stable branches has been previous discussed but no action
has been taken. So going forward I propose we pin all requirements,
including transitive, on stable branches. This way the release of new
dependencies cannot automatically break stable branches and thus break
grenade on master.

I think we can implement this using `pip freeze` to get a list of all the
installed libraries and `pip install -r --no-deps` to install the specific
package versions.  I am not sure how best to handle package versions being
removed from pypi though. Or at the very least we can cap requirements in
global requirements and at least reduce the number of packages we install
uncapped versions of.

Thoughts?

best,
Joe

[0] https://pypi.python.org/pypi/eventlet/0.16.1
[1] http://status.openstack.org/elastic-recheck/#1410626
[2]
http://logs.openstack.org/98/115998/59/check/check-grenade-dsvm/e57eb40/logs/old/screen-n-cpu.txt.gz
[3] http://paste.openstack.org/show/157558/
[4]
https://review.openstack.org/#/q/I4bbbeb5bf9c22ed36f5c9a74fec6b487d2c15697,n,z
[5] https://github.com/eventlet/eventlet/releases/tag/v0.16.1
[6] https://bugs.launchpad.net/nova/+bug/1407685
[7]
https://review.openstack.org/#/q/Idbb9d2b53829dae0e807cd1260dee3dce155d5f3,n,z
[8] https://bugs.launchpad.net/nova/+bug/1408987
[9] https://bugs.launchpad.net/glance/+bug/1410494
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150115/772a8485/attachment.html>