<html><body>
<p><font size="2" face="sans-serif">We're running into a problem periodically where we lose our qpid connection for one of our compute services. We're on Folsom in a 2-node setup with the compute services running on one node and qpidd, scheduler, network, etc., running on the other.</font><br>
<br>
<font size="2" face="sans-serif">We've scaled this environment up to where we have 2800 instances created. When we hit this problem, the scheduler continues to get updates from the compute service so the service is still active, however looking at the qpid queues with "qpid-config queues", we see that the queue no longer exists and the compute service no longer receives spawn requests. The scheduler continues to select this compute service for new boot requests which get stuck in BUILD state.</font><br>
<br>
<font size="2" face="sans-serif">I have a trace here on pastebin <a href="http://pastebin.com/rDid7Egm">http://pastebin.com/rDid7Egm</a> </font><br>
<br>
<font size="2" face="sans-serif">The first error appears to be an RPC Timeout "</font><font size="2" face="sans-serif">Timed out waiting for RPC response: None " followed by an AssertionError in the qpid/messaging/driver.py.</font><br>
<br>
<font size="2" face="sans-serif">Any ideas about what might be happening would be appreciated. Also if you have thoughts on how to debug this further I'd love to hear them.</font><br>
<font size="2" face="sans-serif">Thanks!</font><br>
<font size="2" face="sans-serif">-Paul</font><br>
<br>
</body></html>