<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 18, 2019 at 7:51 AM <<a href="mailto:iain.macdonnell@oracle.com">iain.macdonnell@oracle.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
Is anyone running Stein nova-api under uWSGI? I'm trying to do this, in <br>
exactly the same way as I did with Rocky, but encountering a new problem...<br>
<br></blockquote><div><br></div><div>This sounds similar to a problem that we're currently debugging with John <br></div><div>Eckersberg and Michele Baldessari (in CC) in Stein, but when nova-api runs</div><div>under Apache mod_wsgi.</div><div><br></div><div>The symptoms for us seem similar: sometimes the scheduling of the AMQP</div><div>heartbeat greenlet seems heratic (no run for 75 seconds), RabbitMQ</div><div>drops the connection to the client during this idle period, and next time</div><div>nova-api needs to communicate over AMQP it logs a "connection reset</div><div>by peer" and reestablish a new connection.</div><div><br></div><div>This is fresh news from yesterday so this is still to be confirmed/disproved;</div><div>but currently we're investigating how eventlet schedules recurring tasks</div><div>such as the AMQP heartbeat greenlet under Apache mod_wsgi's execution</div><div>model.<br></div><div><br></div><div>Once the AMQP heartbeat greenlet has sent its heartbeat, it sleeps for a</div><div>fixed amount of time; i.e. it suspends its execution, eventlet registers a timer</div><div>for this greenlet, and continues scheduling other greenlets. Once the timer <br></div><div>expires, eventlet marks the heartbeat greenlet as "eligible for scheduling"</div><div>and eventually this greenlet resumes its execution.</div><div>For that mechanism to work, eventlet regularly watches for all the timers</div><div>that are registered in its event loop (with select/epoll/whatnot).<br></div><div><br></div><div>Now, Apache mod_wsgi embeds its own Python interpreter, and calls it <br></div><div>whenever there's an API call to process. It seems that - the precise course</div><div>of event is unclear to me yet - once the API call is processed, the Python</div><div>Interpreter returns a result to mod_wsgi and stops its execution even if</div><div>some timers were still registered in eventlet.</div><div><br></div><div>The state of the Python Interpreter is conserved though, and next time</div><div>mod_wsgi receives an API call, it resumes the execution of the Python</div><div>Interpreter: eventlet resumes its execution and process every timer</div><div>that were registered previously. So ultimately, the AMQP heartbeat <br></div><div>greenlet is rescheduled, but that might be a long time already since the</div><div>last AMQP heartbeat.<br><div><br></div><div>So I have the feeling that when there's no traffic, mod_wsgi doesn't execute</div><div>the Python interpreter, which prevents eventlet from running, watching for</div><div>timers, and triggering the AMQP heartbeat greenlet. So far, the evidences</div><div>in favour of this theory are:</div><div> . When there's no traffic for nova-api running under mod_wsgi, and I use</div><div> gdb to dump the call stack of the mod_wsgi process, I see no CPython</div><div> interpreter in the stack. Whereas when there's traffic, I see many CPython</div><div> stack frames.<br></div><div> . When there's no traffic to nova-api and the AMQP heartbeat greenlet seem</div><div> to be unscheduled for a long time, running a simple 'nova list' command</div><div class="gmail_quote"> seems to unblock everything and trigger a AMQP heartbeat.</div><div class="gmail_quote"><br></div><div class="gmail_quote">This would also explain why running the nova-api service in standalone works</div><div class="gmail_quote">just fine as you said. because the Python interpreter is never stopped in this</div><div class="gmail_quote">execution model.</div><div class="gmail_quote"><br></div><div class="gmail_quote">I'm still trying to figure out when the Python interpreter returns to mod_wsgi</div><div class="gmail_quote">to make sure what I'm saying makes sense. And I also need to double check</div><div class="gmail_quote">that running in standalone works fine for me as well.<br></div><br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
The symptom is intermittent failure of API operations that involve RPC <br>
calls - notably getting the noVNC console URL for an instance. The <br>
nova-api log shows RPC timeouts and "connection reset by peer" for AMQP. <br>
I've ruled out environmental issues, as a Rocky installation on the same <br>
hosts works fine. With oslo debug enabled, I see that the first API call <br>
causes a AMQP connections to be established, and the RPC call gets its <br>
reply, and all is well... but there are no heartbeats logged after that. <br>
3 minutes later, rabbitmq-server reports that it has received no <br>
heartbeats, and drops the connection. On a subsequent API call (but not <br>
before), nova-api logging "connection reset by peer" and the RPC call <br>
times out. With the Rocky code, once the first API call causes the AMQP <br>
connections to be established, regular heartbeats start happening, the <br>
connections are maintained, and subsequent API calls work as expected.<br>
<br>
It seems like a lockup in a thread that's supposed to maintain the AMQP <br>
connection, or something. I tried to use SIGUSR2 to trigger guru <br>
meditation, but it seems that uWSGI has its own interpretation of <br>
SIGUSR2, so nova-api never sees it (anyone know a way around this?).<br>
<br>
Running nova-api as a standalone service, instead of under uWSGI, the <br>
problem does not occur (I see the heartbeats, and everything works fine).<br>
<br>
I've tried updating oslo-messaging (and python-amqp) on a Rocky <br>
installation to the Stein version, but the problem does not reproduce.<br>
<br>
Ideas?<br>
<br>
~iain<br>
<br>
<br>
<br>
</blockquote></div></div></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">Damien<br></div></div></div>