Open Stack

On Wed, Feb 17, 2016 at 06:58:18PM -0500, Sean Dague wrote:
> On 02/17/2016 05:05 PM, Doug Hellmann wrote:
> > Excerpts from Tony Breeds's message of 2016-02-18 08:04:00 +1100:
> <snip>
> >> I think we're in a tough spot.
> >>
> >> My $0.02 is that we have to cap at <0.18.0 however ....
> >>
> >> We're (the openstack community) finding issues with eventlet which is good
> >> (painful I agree, but good) for both communities.  However we *are* trying to
> >> lock down our release, and test/gate failures are very unwelcome in this
> >> phase.
> >>
> >> 0.18.2 was "mostly" good but shows the following bug:
> >>   https://bugs.launchpad.net/nova/+bug/1544801
> >>
> >> The eventlet team have been very responsive to our issue reports.  I'm worried
> >> if we cap at 0.18.2 (or 0.17.4) the underlying problems will go undressed and
> >> unnoticed for too long and then we'll all loose :(  Not least of all because
> >> it's *very* likely if we cap this we'll release with the cap which means we're
> >> basically stuck with it for the whole life-cycle of mitaka[1][2]
> > 
> > The current plan is to not cap, but exclude the bad versions. The patch
> > to do that (https://review.openstack.org/#/c/281479/) does set the
> > constraint to 0.18.2, so if we need to change that please drop a comment
> > on the review.
> > 
> >> Calls for "extra vetting" of new eventlet releases are slightly problematic,
> >> and really require a short cross-project team to verify, as the "Yup is passes
> >> dsvm-tempest-full" doesn't cut it.
> > 
> > Can we construct a better automated test scenario?
> 
> Question. Are we only tripping this up in unit tests because the tests
> are doing things we'd never really do in real life?

In the case of (nova):
 https://bugs.launchpad.net/nova/+bug/1544801
 https://github.com/eventlet/eventlet/issues/299

It was seen in a "in the wild".  That particular issue didn't seem to break
anything (for nova) but did record lots of tracebacks in logs.

The keystone failure with 0.18.2 *was* limited to tests and has been worked
around for the time being.
  https://github.com/eventlet/eventlet/issues/296

The Neutron issue:
  https://bugs.launchpad.net/neutron/+bug/1546506
  https://github.com/eventlet/eventlet/issues/301

I don't think this was a test-only scenario, and was a clear regression between
0.18.3 and 0.18.3

In my opinion we'll need something to increase faith in a new release, be that
manual effort or better tests/tools

The challenge seems to be getting back to a point where we can function.

I suggested moving back to 0.17.4, but that's going to be hard[1] as we've made
several library releases that have "evelntlet>=0.18.2"

So perhaps [2] is the right thing to do, as it was mostly good.  I think we can
deal with the log spam while we test 0.18.4

Like I said earlier the eventlet team has been very responsive and I'm
certainly not ignoring that.  We just need to be confident that whichever
approach we take we can complete it my R-5

> For instance, the nova bug was because someone decided to test the
> eventlet wsgi server with a sample app with a custom written eventlet
> http client that was doing raw socket reads naively. Which was never a
> good idea. The fix: use requests as our http client. Life was fine.

Yeah, there have certainly been some test-only issues.

> I think it's fine if we're going to want some better tests for bumping
> upper-constraints.txt, even if it's tests we only run for certain touchy
> packages like eventlet. However we should make sure it wasn't just our
> tests being kind of bonkers.

I *think* we've passed the point where we're seeing issues restricted to the
unit/functional/tempest tests.

Yours Tony.

[1] https://review.openstack.org/#/c/281479/2
[2] https://review.openstack.org/#/c/281479/1
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160218/02d13564/attachment.pgp>