Open Stack

On Wed, Feb 17, 2016 at 12:44:11PM -0500, Doug Hellmann wrote:
> Excerpts from Henry Gessau's message of 2016-02-17 11:00:53 -0500:
> > Doug Hellmann <doug at doughellmann.com> wrote:
> > > Excerpts from Morgan Fainberg's message of 2016-02-17 07:10:34 -0800:
> > >> On Wed, Feb 17, 2016 at 5:55 AM, Sean Dague <sean at dague.net> wrote:
> > >>
> > >>> On 02/17/2016 08:42 AM, Doug Hellmann wrote:
> > >>>> Excerpts from Victor Stinner's message of 2016-02-17 14:14:18 +0100:
> > >>>>> Le 17/02/2016 13:43, Henry Gessau a écrit :
> > >>>>>> And it looks like eventlet 0.18.3 breaks neutron:
> > >>>>>> https://bugs.launchpad.net/neutron/+bug/1546506
> > >>>>>
> > >>>>> 2 releases, 2 regressions in OpenStack. Should we cap eventlet version?
> > >>>>> The requirement bot can produce patches to update eventlet, patches
> > >>>>> which would run integration tests using Nova, Keystone, Neutron on the
> > >>>>> new eventlet version.
> > >>>>>
> > >>>>> eventlet 0.18.2 broke OpenStack Keystone and OpenStack Nova
> > >>>>> https://github.com/eventlet/eventlet/issues/296
> > >>>>> https://github.com/eventlet/eventlet/issues/299
> > >>>>> https://review.openstack.org/#/c/278147/
> > >>>>> https://bugs.launchpad.net/nova/+bug/1544801
> > >>>>>
> > >>>>> eventlet 0.18.3 broke OpenStack Neutron
> > >>>>> https://github.com/eventlet/eventlet/issues/301
> > >>>>> https://bugs.launchpad.net/neutron/+bug/1546506
> > >>>>>
> > >>>>> FYI eventlet 0.18.0 broke WSGI servers:
> > >>>>> https://github.com/eventlet/eventlet/issues/295
> > >>>>>
> > >>>>> It was followed quickly by eventlet 0.18.2 to fix this issue.
> > >>>>>
> > >>>>> Sadly, it looks like bugfix releases of eventlet don't include a single
> > >>>>> bugfix, but include also other changes. For example, 0.18.3 fixed the
> > >>>>> bug #296 but introduced "wsgi: TCP_NODELAY enabled by default"
> > >>> optimization.
> > >>>>>
> > >>>>> IMHO the problem is not the release manager of eventlet, but more the
> > >>>>> lack of tests on eventlet, especially on OpenStack services.
> > >>>>>
> > >>>>> Current "Continious Delivery"-like with gates do detect bugs, yeah, but
> > >>>>> also block a lot of developers when the gates are broken. It doesn't
> > >>>>> seem trivial to investigate and fix eventlet issues.
> > >>>>>
> > >>>>> Victor
> > >>>>>
> > >>>>
> > >>>> Whether we cap or not, we should exclude the known broken versions.
> > >>>> It looks like getting back to a good version will also require
> > >>>> lowering the minimum version we support, since we have >=0.18.2
> > >>>> now.
> > >>>>
> > >>>> What was the last version of eventlet known to work?
> > >>>
> > >>> 0.18.2 works. On the Nova side we had a failure around unit tests which
> > >>> was quite synthetic that we fixed. I don' know what the keystone issue
> > >>> turned out to be.
> > >>>
> > >>
> > >> I believe the keystone issue was a test specific issue, not a runtime
> > >> issue. We disabled the test.
> > >> --Morgan
> > > 
> > > OK. Can someone from the neutron team verify that 0.18.2 works? If so,
> > > we can just exclude 0.18.3 and reset the constraint.
> > 
> > I can confirm that neutron works with 0.18.2 as far as we know.
> > 
> 
> Great. If you (or someone else) wants to submit a requirements update, I
> can approve it. Ping me in #openstack-release.

I think we're in a tough spot.

My $0.02 is that we have to cap at <0.18.0 however ....

We're (the openstack community) finding issues with eventlet which is good
(painful I agree, but good) for both communities.  However we *are* trying to
lock down our release, and test/gate failures are very unwelcome in this
phase.

0.18.2 was "mostly" good but shows the following bug:
  https://bugs.launchpad.net/nova/+bug/1544801

The eventlet team have been very responsive to our issue reports.  I'm worried
if we cap at 0.18.2 (or 0.17.4) the underlying problems will go undressed and
unnoticed for too long and then we'll all loose :(  Not least of all because
it's *very* likely if we cap this we'll release with the cap which means we're
basically stuck with it for the whole life-cycle of mitaka[1][2]

Calls for "extra vetting" of new eventlet releases are slightly problematic,
and really require a short cross-project team to verify, as the "Yup is passes
dsvm-tempest-full" doesn't cut it.

Yours Tony.

[1] Given we're @ R-7 and requirements freeze is R-5
[2] Assuming we cap at <0.18.0, we don't generally allow raising the minimum
    version of $library in a stable release
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160218/54bccd7d/attachment.pgp>