Open Stack

Doug Hellmann <doug at doughellmann.com> wrote:
> Excerpts from Henry Gessau's message of 2016-02-17 11:00:53 -0500:
>> Doug Hellmann <doug at doughellmann.com> wrote:
>>> Excerpts from Morgan Fainberg's message of 2016-02-17 07:10:34 -0800:
>>>> On Wed, Feb 17, 2016 at 5:55 AM, Sean Dague <sean at dague.net> wrote:
>>>>
>>>>> On 02/17/2016 08:42 AM, Doug Hellmann wrote:
>>>>>> Excerpts from Victor Stinner's message of 2016-02-17 14:14:18 +0100:
>>>>>>> Le 17/02/2016 13:43, Henry Gessau a écrit :
>>>>>>>> And it looks like eventlet 0.18.3 breaks neutron:
>>>>>>>> https://bugs.launchpad.net/neutron/+bug/1546506
>>>>>>>
>>>>>>> 2 releases, 2 regressions in OpenStack. Should we cap eventlet version?
>>>>>>> The requirement bot can produce patches to update eventlet, patches
>>>>>>> which would run integration tests using Nova, Keystone, Neutron on the
>>>>>>> new eventlet version.
>>>>>>>
>>>>>>> eventlet 0.18.2 broke OpenStack Keystone and OpenStack Nova
>>>>>>> https://github.com/eventlet/eventlet/issues/296
>>>>>>> https://github.com/eventlet/eventlet/issues/299
>>>>>>> https://review.openstack.org/#/c/278147/
>>>>>>> https://bugs.launchpad.net/nova/+bug/1544801
>>>>>>>
>>>>>>> eventlet 0.18.3 broke OpenStack Neutron
>>>>>>> https://github.com/eventlet/eventlet/issues/301
>>>>>>> https://bugs.launchpad.net/neutron/+bug/1546506
>>>>>>>
>>>>>>> FYI eventlet 0.18.0 broke WSGI servers:
>>>>>>> https://github.com/eventlet/eventlet/issues/295
>>>>>>>
>>>>>>> It was followed quickly by eventlet 0.18.2 to fix this issue.
>>>>>>>
>>>>>>> Sadly, it looks like bugfix releases of eventlet don't include a single
>>>>>>> bugfix, but include also other changes. For example, 0.18.3 fixed the
>>>>>>> bug #296 but introduced "wsgi: TCP_NODELAY enabled by default"
>>>>> optimization.
>>>>>>>
>>>>>>> IMHO the problem is not the release manager of eventlet, but more the
>>>>>>> lack of tests on eventlet, especially on OpenStack services.
>>>>>>>
>>>>>>> Current "Continious Delivery"-like with gates do detect bugs, yeah, but
>>>>>>> also block a lot of developers when the gates are broken. It doesn't
>>>>>>> seem trivial to investigate and fix eventlet issues.
>>>>>>>
>>>>>>> Victor
>>>>>>>
>>>>>>
>>>>>> Whether we cap or not, we should exclude the known broken versions.
>>>>>> It looks like getting back to a good version will also require
>>>>>> lowering the minimum version we support, since we have >=0.18.2
>>>>>> now.
>>>>>>
>>>>>> What was the last version of eventlet known to work?
>>>>>
>>>>> 0.18.2 works. On the Nova side we had a failure around unit tests which
>>>>> was quite synthetic that we fixed. I don' know what the keystone issue
>>>>> turned out to be.
>>>>>
>>>>
>>>> I believe the keystone issue was a test specific issue, not a runtime
>>>> issue. We disabled the test.
>>>> --Morgan
>>>
>>> OK. Can someone from the neutron team verify that 0.18.2 works? If so,
>>> we can just exclude 0.18.3 and reset the constraint.
>>
>> I can confirm that neutron works with 0.18.2 as far as we know.
>>
> 
> Great. If you (or someone else) wants to submit a requirements update, I
> can approve it. Ping me in #openstack-release.

If it's only neutron that is affected by 0.18.3 then we already have our
workaround in place [1]. Additionally, eventlet 0.18.4 will replace the
breaking change with a different approach [2].

[1] https://review.openstack.org/281278
[2] https://github.com/eventlet/eventlet/issues/301