Open Stack

Excerpts from Tony Breeds's message of 2016-02-18 08:04:00 +1100:
> On Wed, Feb 17, 2016 at 12:44:11PM -0500, Doug Hellmann wrote:
> > Excerpts from Henry Gessau's message of 2016-02-17 11:00:53 -0500:
> > > Doug Hellmann <doug at doughellmann.com> wrote:
> > > > Excerpts from Morgan Fainberg's message of 2016-02-17 07:10:34 -0800:
> > > >> On Wed, Feb 17, 2016 at 5:55 AM, Sean Dague <sean at dague.net> wrote:
> > > >>
> > > >>> On 02/17/2016 08:42 AM, Doug Hellmann wrote:
> > > >>>> Excerpts from Victor Stinner's message of 2016-02-17 14:14:18 +0100:
> > > >>>>> Le 17/02/2016 13:43, Henry Gessau a écrit :
> > > >>>>>> And it looks like eventlet 0.18.3 breaks neutron:
> > > >>>>>> https://bugs.launchpad.net/neutron/+bug/1546506
> > > >>>>>
> > > >>>>> 2 releases, 2 regressions in OpenStack. Should we cap eventlet version?
> > > >>>>> The requirement bot can produce patches to update eventlet, patches
> > > >>>>> which would run integration tests using Nova, Keystone, Neutron on the
> > > >>>>> new eventlet version.
> > > >>>>>
> > > >>>>> eventlet 0.18.2 broke OpenStack Keystone and OpenStack Nova
> > > >>>>> https://github.com/eventlet/eventlet/issues/296
> > > >>>>> https://github.com/eventlet/eventlet/issues/299
> > > >>>>> https://review.openstack.org/#/c/278147/
> > > >>>>> https://bugs.launchpad.net/nova/+bug/1544801
> > > >>>>>
> > > >>>>> eventlet 0.18.3 broke OpenStack Neutron
> > > >>>>> https://github.com/eventlet/eventlet/issues/301
> > > >>>>> https://bugs.launchpad.net/neutron/+bug/1546506
> > > >>>>>
> > > >>>>> FYI eventlet 0.18.0 broke WSGI servers:
> > > >>>>> https://github.com/eventlet/eventlet/issues/295
> > > >>>>>
> > > >>>>> It was followed quickly by eventlet 0.18.2 to fix this issue.
> > > >>>>>
> > > >>>>> Sadly, it looks like bugfix releases of eventlet don't include a single
> > > >>>>> bugfix, but include also other changes. For example, 0.18.3 fixed the
> > > >>>>> bug #296 but introduced "wsgi: TCP_NODELAY enabled by default"
> > > >>> optimization.
> > > >>>>>
> > > >>>>> IMHO the problem is not the release manager of eventlet, but more the
> > > >>>>> lack of tests on eventlet, especially on OpenStack services.
> > > >>>>>
> > > >>>>> Current "Continious Delivery"-like with gates do detect bugs, yeah, but
> > > >>>>> also block a lot of developers when the gates are broken. It doesn't
> > > >>>>> seem trivial to investigate and fix eventlet issues.
> > > >>>>>
> > > >>>>> Victor
> > > >>>>>
> > > >>>>
> > > >>>> Whether we cap or not, we should exclude the known broken versions.
> > > >>>> It looks like getting back to a good version will also require
> > > >>>> lowering the minimum version we support, since we have >=0.18.2
> > > >>>> now.
> > > >>>>
> > > >>>> What was the last version of eventlet known to work?
> > > >>>
> > > >>> 0.18.2 works. On the Nova side we had a failure around unit tests which
> > > >>> was quite synthetic that we fixed. I don' know what the keystone issue
> > > >>> turned out to be.
> > > >>>
> > > >>
> > > >> I believe the keystone issue was a test specific issue, not a runtime
> > > >> issue. We disabled the test.
> > > >> --Morgan
> > > > 
> > > > OK. Can someone from the neutron team verify that 0.18.2 works? If so,
> > > > we can just exclude 0.18.3 and reset the constraint.
> > > 
> > > I can confirm that neutron works with 0.18.2 as far as we know.
> > > 
> > 
> > Great. If you (or someone else) wants to submit a requirements update, I
> > can approve it. Ping me in #openstack-release.
> 
> I think we're in a tough spot.
> 
> My $0.02 is that we have to cap at <0.18.0 however ....
> 
> We're (the openstack community) finding issues with eventlet which is good
> (painful I agree, but good) for both communities.  However we *are* trying to
> lock down our release, and test/gate failures are very unwelcome in this
> phase.
> 
> 0.18.2 was "mostly" good but shows the following bug:
>   https://bugs.launchpad.net/nova/+bug/1544801
> 
> The eventlet team have been very responsive to our issue reports.  I'm worried
> if we cap at 0.18.2 (or 0.17.4) the underlying problems will go undressed and
> unnoticed for too long and then we'll all loose :(  Not least of all because
> it's *very* likely if we cap this we'll release with the cap which means we're
> basically stuck with it for the whole life-cycle of mitaka[1][2]

The current plan is to not cap, but exclude the bad versions. The patch
to do that (https://review.openstack.org/#/c/281479/) does set the
constraint to 0.18.2, so if we need to change that please drop a comment
on the review.

> Calls for "extra vetting" of new eventlet releases are slightly problematic,
> and really require a short cross-project team to verify, as the "Yup is passes
> dsvm-tempest-full" doesn't cut it.

Can we construct a better automated test scenario?

Doug

> 
> Yours Tony.
> 
> [1] Given we're @ R-7 and requirements freeze is R-5
> [2] Assuming we cap at <0.18.0, we don't generally allow raising the minimum
>     version of $library in a stable release