Open Stack

On Fri, May 3, 2019 at 7:59 PM Michele Baldessari <michele at acksyn.org>
wrote:

> On Mon, Apr 22, 2019 at 01:21:03PM -0500, Ben Nemec wrote:
> >
> >
> > On 4/22/19 12:53 PM, Alex Schultz wrote:
> > > On Mon, Apr 22, 2019 at 11:28 AM Ben Nemec <openstack at nemebean.com>
> wrote:
> > > >
> > > >
> > > >
> > > > On 4/20/19 1:38 AM, Michele Baldessari wrote:
> > > > > On Fri, Apr 19, 2019 at 03:20:44PM -0700,
> iain.macdonnell at oracle.com wrote:
> > > > > >
> > > > > > Today I discovered that this problem appears to be caused by
> eventlet
> > > > > > monkey-patching. I've created a bug for it:
> > > > > >
> > > > > > https://bugs.launchpad.net/nova/+bug/1825584
> > > > >
> > > > > Hi,
> > > > >
> > > > > just for completeness we see this very same issue also with
> > > > > mistral (actually it was the first service where we noticed the
> missed
> > > > > heartbeats). iirc Alex Schultz mentioned seeing it in ironic as
> well,
> > > > > although I have not personally observed it there yet.
> > > >
> > > > Is Mistral also mixing eventlet monkeypatching and WSGI?
> > > >
> > >
> > > Looks like there is monkey patching, however we noticed it with the
> > > engine/executor. So it's likely not just wsgi.  I think I also saw it
> > > in the ironic-conductor, though I'd have to try it out again.  I'll
> > > spin up an undercloud today and see if I can get a more complete list
> > > of affected services. It was pretty easy to reproduce.
> >
> > Okay, I asked because if there's no WSGI/Eventlet combination then this
> may
> > be different from the Nova issue that prompted this thread. It sounds
> like
> > that was being caused by a bad interaction between WSGI and some Eventlet
> > timers. If there's no WSGI involved then I wouldn't expect that to
> happen.
> >
> > I guess we'll see what further investigation turns up, but based on the
> > preliminary information there may be two bugs here.
>
> So just to get some closure on this error that we have seen around
> mistral executor and tripleo with python3: this was due to the ansible
> action that called subprocess which has a different implementation in
> python3 and so the monkeypatching needs to be adapted.
>
> Review which fixes it for us is here:
> https://review.opendev.org/#/c/656901/
>
> Damien and I think the nova_api/eventlet/mod_wsgi has a separate root-cause
> (although we have not spent all too much time on that one yet)
>
>
Right, after further investigation, it appears that the problem we saw
under mod_wsgi was due to monkey patching, as Iain originally
reported. It has nothing to do with our work on healthchecks.

It turns out that running the AMQP heartbeat thread under mod_wsgi
doesn't work when the threading library is monkey_patched, because the
thread waits on a data structure [1] that has been monkey patched [2],
which makes it yield its execution instead of sleeping for 15s.

Because mod_wsgi stops the execution of its embedded interpreter, the
AMQP heartbeat thread can't be resumed until there's a message to be
processed in the mod_wsgi queue, which would resume the python
interpreter and make eventlet resume the thread.

Disabling monkey-patching in nova_api makes the scheduling issue go
away.

Note: other services like heat-api do not use monkey patching and
aren't affected, so this seem to confirm that monkey-patching
shouldn't happen in nova_api running under mod_wsgi in the first
place.

[1]
https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L904
[2]
https://github.com/openstack/oslo.utils/blob/master/oslo_utils/eventletutils.py#L182
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190504/ad13be84/attachment-0001.html>