Open Stack

Excerpts from Henry Gessau's message of 2016-02-17 13:00:03 -0500:
> Doug Hellmann <doug at doughellmann.com> wrote:
> > Excerpts from Henry Gessau's message of 2016-02-17 11:00:53 -0500:
> >> Doug Hellmann <doug at doughellmann.com> wrote:
> >>> Excerpts from Morgan Fainberg's message of 2016-02-17 07:10:34 -0800:
> >>>> On Wed, Feb 17, 2016 at 5:55 AM, Sean Dague <sean at dague.net> wrote:
> >>>>
> >>>>> On 02/17/2016 08:42 AM, Doug Hellmann wrote:
> >>>>>> Excerpts from Victor Stinner's message of 2016-02-17 14:14:18 +0100:
> >>>>>>> Le 17/02/2016 13:43, Henry Gessau a écrit :
> >>>>>>>> And it looks like eventlet 0.18.3 breaks neutron:
> >>>>>>>> https://bugs.launchpad.net/neutron/+bug/1546506
> >>>>>>>
> >>>>>>> 2 releases, 2 regressions in OpenStack. Should we cap eventlet version?
> >>>>>>> The requirement bot can produce patches to update eventlet, patches
> >>>>>>> which would run integration tests using Nova, Keystone, Neutron on the
> >>>>>>> new eventlet version.
> >>>>>>>
> >>>>>>> eventlet 0.18.2 broke OpenStack Keystone and OpenStack Nova
> >>>>>>> https://github.com/eventlet/eventlet/issues/296
> >>>>>>> https://github.com/eventlet/eventlet/issues/299
> >>>>>>> https://review.openstack.org/#/c/278147/
> >>>>>>> https://bugs.launchpad.net/nova/+bug/1544801
> >>>>>>>
> >>>>>>> eventlet 0.18.3 broke OpenStack Neutron
> >>>>>>> https://github.com/eventlet/eventlet/issues/301
> >>>>>>> https://bugs.launchpad.net/neutron/+bug/1546506
> >>>>>>>
> >>>>>>> FYI eventlet 0.18.0 broke WSGI servers:
> >>>>>>> https://github.com/eventlet/eventlet/issues/295
> >>>>>>>
> >>>>>>> It was followed quickly by eventlet 0.18.2 to fix this issue.
> >>>>>>>
> >>>>>>> Sadly, it looks like bugfix releases of eventlet don't include a single
> >>>>>>> bugfix, but include also other changes. For example, 0.18.3 fixed the
> >>>>>>> bug #296 but introduced "wsgi: TCP_NODELAY enabled by default"
> >>>>> optimization.
> >>>>>>>
> >>>>>>> IMHO the problem is not the release manager of eventlet, but more the
> >>>>>>> lack of tests on eventlet, especially on OpenStack services.
> >>>>>>>
> >>>>>>> Current "Continious Delivery"-like with gates do detect bugs, yeah, but
> >>>>>>> also block a lot of developers when the gates are broken. It doesn't
> >>>>>>> seem trivial to investigate and fix eventlet issues.
> >>>>>>>
> >>>>>>> Victor
> >>>>>>>
> >>>>>>
> >>>>>> Whether we cap or not, we should exclude the known broken versions.
> >>>>>> It looks like getting back to a good version will also require
> >>>>>> lowering the minimum version we support, since we have >=0.18.2
> >>>>>> now.
> >>>>>>
> >>>>>> What was the last version of eventlet known to work?
> >>>>>
> >>>>> 0.18.2 works. On the Nova side we had a failure around unit tests which
> >>>>> was quite synthetic that we fixed. I don' know what the keystone issue
> >>>>> turned out to be.
> >>>>>
> >>>>
> >>>> I believe the keystone issue was a test specific issue, not a runtime
> >>>> issue. We disabled the test.
> >>>> --Morgan
> >>>
> >>> OK. Can someone from the neutron team verify that 0.18.2 works? If so,
> >>> we can just exclude 0.18.3 and reset the constraint.
> >>
> >> I can confirm that neutron works with 0.18.2 as far as we know.
> >>
> > 
> > Great. If you (or someone else) wants to submit a requirements update, I
> > can approve it. Ping me in #openstack-release.
> 
> If it's only neutron that is affected by 0.18.3 then we already have our
> workaround in place [1]. Additionally, eventlet 0.18.4 will replace the
> breaking change with a different approach [2].

I suspect, given the phase of our cycle we're in and the nature of the
past couple of eventlet releases, we're going to want to do more
extensive testing before taking a new release. We're approaching a
requirements freeze *anyway* so it may be moot, depending on when they
get 0.18.4 out.

Doug

> 
> [1] https://review.openstack.org/281278
> [2] https://github.com/eventlet/eventlet/issues/301
>