<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" id="owaParaStyle"></style>
</head>
<body>
<div style="direction:ltr; font-family:Tahoma; color:#000000; font-size:10pt">I have a CentOS 6.5 Folsom cluster running 716 VMs across 16 hypervisors. This past week I have started seeing problems which I believe are related to message queueing.
<div><br>
</div>
<div>Symptom one: Horizon stalls for a long time upon login or when changing tenants, and eventually displays a "Could not retrieve quota information" message.</div>
<div><br>
</div>
<div>Symptom two: Metadata services calls are not being serviced, and nova api logs show messages similar to the following: <span style="font-size:10pt">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler Timeout: Timeout while waiting on RPC response.</span></div>
<div>
<div><br>
</div>
</div>
<div>Using qpid-tool, I can see that the exchange count (currently more than 10,000) is growing larger and larger. </div>
<div><br>
</div>
<div>
<div>qpid: list</div>
<div>Summary of Objects by Type:</div>
<div> Package Class Active Deleted</div>
<div> =======================================================</div>
<div> org.apache.qpid.broker binding 110 184</div>
<div> org.apache.qpid.broker broker 1 0</div>
<div> org.apache.qpid.broker system 1 0</div>
<div> org.apache.qpid.broker subscription 66 92</div>
<div> org.apache.qpid.broker connection 40 0</div>
<div> org.apache.qpid.broker session 40 157</div>
<div> org.apache.qpid.broker queue 54 92</div>
<div> org.apache.qpid.broker exchange 10025 0</div>
<div> org.apache.qpid.broker vhost 1 0</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Can anyone help me identify what is going on here and get things back to normal? Please let me know what other information would be helpful to provide.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Mike</div>
<div>Overstock.com</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Here is a more complete traceback:</div>
<div><br>
</div>
<div><span style="font-size:10pt">2014-05-28 09:57:15 16759 ERROR nova.openstack.common.rpc.impl_qpid [-] Timed out waiting for RPC response: None</span></div>
<div>2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid Traceback (most recent call last):</div>
<div>2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 358, in ensure</div>
<div>2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid return method(*args, **kwargs)</div>
<div>2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 407, in _consume</div>
<div>2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid nxt_receiver = self.session.next_receiver(timeout=timeout)</div>
<div>2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid File "<string>", line 6, in next_receiver</div>
<div>2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 663, in next_receiver</div>
<div>2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid raise Empty</div>
<div>2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid Empty: None</div>
<div>2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid</div>
<div>2014-05-28 09:57:15 16759 ERROR nova.openstack.common.rpc.impl_qpid [-] Timed out waiting for RPC response: None</div>
<div>2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid Traceback (most recent call last):</div>
<div>2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 358, in ensure</div>
<div>2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid return method(*args, **kwargs)</div>
<div>2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 407, in _consume</div>
<div>2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid nxt_receiver = self.session.next_receiver(timeout=timeout)</div>
<div>2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid File "<string>", line 6, in next_receiver</div>
<div>2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 663, in next_receiver</div>
<div>2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid raise Empty</div>
<div>2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid Empty: None</div>
<div>2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid</div>
<div>2014-05-28 09:57:16 ERROR nova.api.metadata.handler [req-93548aee-cd7e-46af-8e95-df3e0d6b96a2 None None] Failed to get metadata for ip: 10.8.55.11</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler Traceback (most recent call last):</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/api/metadata/handler.py", line 75, in __call__</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler meta_data = self.get_metadata(remote_address)</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/api/metadata/handler.py", line 57, in get_metadata</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler data = base.get_metadata_by_address(address)</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/api/metadata/base.py", line 385, in get_metadata_by_address</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler fixed_ip = network.API().get_fixed_ip_by_address(ctxt, address)</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 126, in get_fixed_ip_by_address</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler 'args': {'address': address}})</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/__init__.py", line 108, in call</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler return _get_impl().call(cfg.CONF, context, topic, msg, timeout)</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 543, in call</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler rpc_amqp.get_connection_pool(conf, Connection))</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 369, in call</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler rv = list(rv)</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 330, in __iter__</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler self.done()</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler self.gen.next()</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 327, in __iter__</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler self._iterator.next()</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 416, in iterconsume</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler yield self.ensure(_error_callback, _consume)</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 362, in ensure</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler error_callback(e)</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 401, in _error_callback</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler raise rpc_common.Timeout()</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler Timeout: Timeout while waiting on RPC response.</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler</div>
<div>2014-05-28 09:57:16 16759 INFO nova.api.ec2 [-] 13656.257934s 10.8.55.11 GET / None:None 500 [None] text/plain text/html</div>
<div>2014-05-28 09:57:16 16759 INFO nova.metadata.wsgi.server [-] 10.8.55.11 - - [28/May/2014 09:57:16] "GET / HTTP/1.1" 500 343 13656.259023</div>
<div><br>
</div>
<div>2014-05-28 09:57:16 ERROR nova.api.metadata.handler [req-da2813bf-69a9-491e-b655-ca99c6cf97a2 None None] Failed to get metadata for ip: 10.8.11.17</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler Traceback (most recent call last):</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/api/metadata/handler.py", line 75, in __call__</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler meta_data = self.get_metadata(remote_address)</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/api/metadata/handler.py", line 57, in get_metadata</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler data = base.get_metadata_by_address(address)</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/api/metadata/base.py", line 385, in get_metadata_by_address</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler fixed_ip = network.API().get_fixed_ip_by_address(ctxt, address)</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 126, in get_fixed_ip_by_address</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler 'args': {'address': address}})</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/__init__.py", line 108, in call</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler return _get_impl().call(cfg.CONF, context, topic, msg, timeout)</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 543, in call</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler rpc_amqp.get_connection_pool(conf, Connection))</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 369, in call</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler rv = list(rv)</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 330, in __iter__</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler self.done()</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler self.gen.next()</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 327, in __iter__</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler self._iterator.next()</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 416, in iterconsume</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler yield self.ensure(_error_callback, _consume)</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 362, in ensure</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler error_callback(e)</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 401, in _error_callback</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler raise rpc_common.Timeout()</div>
<div>2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler Timeout: Timeout while waiting on RPC response.</div>
<div><span style="font-size:10pt">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler</span></div>
<div><br>
</div>
<div> </div>
</div>
<br>
<hr>
<font face="Arial" color="Gray" size="1"><br>
CONFIDENTIALITY NOTICE: This message is intended only for the use and review of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this message is not the intended recipient, or
the employee or agent responsible for delivering the message solely to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in
error, please notify sender immediately by telephone or return email. Thank you.<br>
</font>
</body>
</html>