<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body class="" style="word-wrap:break-word">
This issue that we experienced in May has hit us again, although this time I don’t see a large number of exchanges using qpid-tool. I believe the “fix” that I will have to perform is for me to failover/restart my nova-network service, which somehow unwedged
the RPC subsystem last time.
<div class=""><br class="">
</div>
<div class="">Before I do so, I would love to know if anyone has an idea as to why this would be occurring. Restarting nova-network is not something I love doing because we lose network connectivity to our instances for several minutes.</div>
<div class=""><br class="">
</div>
<div class="">Thanks</div>
<div class=""><br class="">
<div class=""><br class="">
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On May 28, 2014, at 10:06 AM, Mike Smith <<a href="mailto:mismith@overstock.com" class="">mismith@overstock.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="" style="font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; line-height:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; direction:ltr; font-family:Tahoma; font-size:10pt">
I have a CentOS 6.5 Folsom cluster running 716 VMs across 16 hypervisors. This past week I have started seeing problems which I believe are related to message queueing.
<div class=""><br class="">
</div>
<div class="">Symptom one: Horizon stalls for a long time upon login or when changing tenants, and eventually displays a "Could not retrieve quota information" message.</div>
<div class=""><br class="">
</div>
<div class="">Symptom two: Metadata services calls are not being serviced, and nova api logs show messages similar to the following: <span class="" style="font-size:10pt">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler Timeout: Timeout while waiting
on RPC response.</span></div>
<div class="">
<div class=""><br class="">
</div>
</div>
<div class="">Using qpid-tool, I can see that the exchange count (currently more than 10,000) is growing larger and larger. </div>
<div class=""><br class="">
</div>
<div class="">
<div class="">qpid: list</div>
<div class="">Summary of Objects by Type:</div>
<div class=""> Package Class Active Deleted</div>
<div class=""> =======================================================</div>
<div class=""> org.apache.qpid.broker binding 110 184</div>
<div class=""> org.apache.qpid.broker broker 1 0</div>
<div class=""> org.apache.qpid.broker system 1 0</div>
<div class=""> org.apache.qpid.broker subscription 66 92</div>
<div class=""> org.apache.qpid.broker connection 40 0</div>
<div class=""> org.apache.qpid.broker session 40 157</div>
<div class=""> org.apache.qpid.broker queue 54 92</div>
<div class=""> org.apache.qpid.broker exchange 10025 0</div>
<div class=""> org.apache.qpid.broker vhost 1 0</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Can anyone help me identify what is going on here and get things back to normal? Please let me know what other information would be helpful to provide.</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Mike</div>
<div class=""><a href="http://Overstock.com" class="">Overstock.com</a></div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Here is a more complete traceback:</div>
<div class=""><br class="">
</div>
<div class=""><span class="" style="font-size:10pt">2014-05-28 09:57:15 16759 ERROR nova.openstack.common.rpc.impl_qpid [-] Timed out waiting for RPC response: None</span></div>
<div class="">2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid Traceback (most recent call last):</div>
<div class="">2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 358, in ensure</div>
<div class="">2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid return method(*args, **kwargs)</div>
<div class="">2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 407, in _consume</div>
<div class="">2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid nxt_receiver = self.session.next_receiver(timeout=timeout)</div>
<div class="">2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid File "<string>", line 6, in next_receiver</div>
<div class="">2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 663, in next_receiver</div>
<div class="">2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid raise Empty</div>
<div class="">2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid Empty: None</div>
<div class="">2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid</div>
<div class="">2014-05-28 09:57:15 16759 ERROR nova.openstack.common.rpc.impl_qpid [-] Timed out waiting for RPC response: None</div>
<div class="">2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid Traceback (most recent call last):</div>
<div class="">2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 358, in ensure</div>
<div class="">2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid return method(*args, **kwargs)</div>
<div class="">2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 407, in _consume</div>
<div class="">2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid nxt_receiver = self.session.next_receiver(timeout=timeout)</div>
<div class="">2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid File "<string>", line 6, in next_receiver</div>
<div class="">2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 663, in next_receiver</div>
<div class="">2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid raise Empty</div>
<div class="">2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid Empty: None</div>
<div class="">2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid</div>
<div class="">2014-05-28 09:57:16 ERROR nova.api.metadata.handler [req-93548aee-cd7e-46af-8e95-df3e0d6b96a2 None None] Failed to get metadata for ip: 10.8.55.11</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler Traceback (most recent call last):</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/api/metadata/handler.py", line 75, in __call__</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler meta_data = self.get_metadata(remote_address)</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/api/metadata/handler.py", line 57, in get_metadata</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler data = base.get_metadata_by_address(address)</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/api/metadata/base.py", line 385, in get_metadata_by_address</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler fixed_ip = network.API().get_fixed_ip_by_address(ctxt, address)</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 126, in get_fixed_ip_by_address</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler 'args': {'address': address}})</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/__init__.py", line 108, in call</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler return _get_impl().call(cfg.CONF, context, topic, msg, timeout)</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 543, in call</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler rpc_amqp.get_connection_pool(conf, Connection))</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 369, in call</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler rv = list(rv)</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 330, in __iter__</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler self.done()</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler self.gen.next()</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 327, in __iter__</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler self._iterator.next()</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 416, in iterconsume</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler yield self.ensure(_error_callback, _consume)</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 362, in ensure</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler error_callback(e)</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 401, in _error_callback</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler raise rpc_common.Timeout()</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler Timeout: Timeout while waiting on RPC response.</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler</div>
<div class="">2014-05-28 09:57:16 16759 INFO nova.api.ec2 [-] 13656.257934s 10.8.55.11 GET / None:None 500 [None] text/plain text/html</div>
<div class="">2014-05-28 09:57:16 16759 INFO nova.metadata.wsgi.server [-] 10.8.55.11 - - [28/May/2014 09:57:16] "GET / HTTP/1.1" 500 343 13656.259023</div>
<div class=""><br class="">
</div>
<div class="">2014-05-28 09:57:16 ERROR nova.api.metadata.handler [req-da2813bf-69a9-491e-b655-ca99c6cf97a2 None None] Failed to get metadata for ip: 10.8.11.17</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler Traceback (most recent call last):</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/api/metadata/handler.py", line 75, in __call__</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler meta_data = self.get_metadata(remote_address)</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/api/metadata/handler.py", line 57, in get_metadata</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler data = base.get_metadata_by_address(address)</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/api/metadata/base.py", line 385, in get_metadata_by_address</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler fixed_ip = network.API().get_fixed_ip_by_address(ctxt, address)</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 126, in get_fixed_ip_by_address</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler 'args': {'address': address}})</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/__init__.py", line 108, in call</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler return _get_impl().call(cfg.CONF, context, topic, msg, timeout)</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 543, in call</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler rpc_amqp.get_connection_pool(conf, Connection))</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 369, in call</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler rv = list(rv)</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 330, in __iter__</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler self.done()</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler self.gen.next()</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 327, in __iter__</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler self._iterator.next()</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 416, in iterconsume</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler yield self.ensure(_error_callback, _consume)</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 362, in ensure</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler error_callback(e)</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 401, in _error_callback</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler raise rpc_common.Timeout()</div>
<div class="">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler Timeout: Timeout while waiting on RPC response.</div>
<div class=""><span class="" style="font-size:10pt">2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler</span></div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br>
<hr>
<font face="Arial" color="Gray" size="1"><br>
CONFIDENTIALITY NOTICE: This message is intended only for the use and review of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this message is not the intended recipient, or
the employee or agent responsible for delivering the message solely to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in
error, please notify sender immediately by telephone or return email. Thank you.<br>
</font>
</body>
</html>