Open Stack

Is anyone running Stein nova-api under uWSGI? I'm trying to do this, in 
exactly the same way as I did with Rocky, but encountering a new problem...

The symptom is intermittent failure of API operations that involve RPC 
calls - notably getting the noVNC console URL for an instance. The 
nova-api log shows RPC timeouts and "connection reset by peer" for AMQP. 
I've ruled out environmental issues, as a Rocky installation on the same 
hosts works fine. With oslo debug enabled, I see that the first API call 
causes a AMQP connections to be established, and the RPC call gets its 
reply, and all is well... but there are no heartbeats logged after that. 
3 minutes later, rabbitmq-server reports that it has received no 
heartbeats, and drops the connection. On a subsequent API call (but not 
before), nova-api logging "connection reset by peer" and the RPC call 
times out. With the Rocky code, once the first API call causes the AMQP 
connections to be established, regular heartbeats start happening, the 
connections are maintained, and subsequent API calls work as expected.

It seems like a lockup in a thread that's supposed to maintain the AMQP 
connection, or something. I tried to use SIGUSR2 to trigger guru 
meditation, but it seems that uWSGI has its own interpretation of 
SIGUSR2, so nova-api never sees it (anyone know a way around this?).

Running nova-api as a standalone service, instead of under uWSGI, the 
problem does not occur (I see the heartbeats, and everything works fine).

I've tried updating oslo-messaging (and python-amqp) on a Rocky 
installation to the Stein version, but the problem does not reproduce.

Ideas?

     ~iain