On Fri, May 3, 2019 at 7:59 PM Michele Baldessari <michele@acksyn.org> wrote:
On Mon, Apr 22, 2019 at 01:21:03PM -0500, Ben Nemec wrote:
On 4/22/19 12:53 PM, Alex Schultz wrote:
On Mon, Apr 22, 2019 at 11:28 AM Ben Nemec <openstack@nemebean.com>
On 4/20/19 1:38 AM, Michele Baldessari wrote:
On Fri, Apr 19, 2019 at 03:20:44PM -0700,
iain.macdonnell@oracle.com wrote:
Today I discovered that this problem appears to be caused by
eventlet
monkey-patching. I've created a bug for it:
Hi,
just for completeness we see this very same issue also with mistral (actually it was the first service where we noticed the missed heartbeats). iirc Alex Schultz mentioned seeing it in ironic as well, although I have not personally observed it there yet.
Is Mistral also mixing eventlet monkeypatching and WSGI?
Looks like there is monkey patching, however we noticed it with the engine/executor. So it's likely not just wsgi. I think I also saw it in the ironic-conductor, though I'd have to try it out again. I'll spin up an undercloud today and see if I can get a more complete list of affected services. It was pretty easy to reproduce.
Okay, I asked because if there's no WSGI/Eventlet combination then this may be different from the Nova issue that prompted this thread. It sounds
wrote: like
that was being caused by a bad interaction between WSGI and some Eventlet timers. If there's no WSGI involved then I wouldn't expect that to happen.
I guess we'll see what further investigation turns up, but based on the preliminary information there may be two bugs here.
So just to get some closure on this error that we have seen around mistral executor and tripleo with python3: this was due to the ansible action that called subprocess which has a different implementation in python3 and so the monkeypatching needs to be adapted.
Review which fixes it for us is here: https://review.opendev.org/#/c/656901/
Damien and I think the nova_api/eventlet/mod_wsgi has a separate root-cause (although we have not spent all too much time on that one yet)
Right, after further investigation, it appears that the problem we saw under mod_wsgi was due to monkey patching, as Iain originally reported. It has nothing to do with our work on healthchecks. It turns out that running the AMQP heartbeat thread under mod_wsgi doesn't work when the threading library is monkey_patched, because the thread waits on a data structure [1] that has been monkey patched [2], which makes it yield its execution instead of sleeping for 15s. Because mod_wsgi stops the execution of its embedded interpreter, the AMQP heartbeat thread can't be resumed until there's a message to be processed in the mod_wsgi queue, which would resume the python interpreter and make eventlet resume the thread. Disabling monkey-patching in nova_api makes the scheduling issue go away. Note: other services like heat-api do not use monkey patching and aren't affected, so this seem to confirm that monkey-patching shouldn't happen in nova_api running under mod_wsgi in the first place. [1] https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_driv... [2] https://github.com/openstack/oslo.utils/blob/master/oslo_utils/eventletutils...