Open Stack

I have a CentOS 6.5 Folsom cluster running 716 VMs across 16 hypervisors.   This past week I have started seeing problems which I believe are related to message queueing.

Symptom one:  Horizon stalls for a long time upon login or when changing tenants, and eventually displays a "Could not retrieve quota information" message.

Symptom two:  Metadata services calls are not being serviced, and nova api logs show messages similar to the following:   2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler Timeout: Timeout while waiting on RPC response.

Using qpid-tool, I can see that the exchange count (currently more than 10,000) is growing larger and larger.

qpid: list
Summary of Objects by Type:
    Package                 Class         Active  Deleted
    =======================================================
    org.apache.qpid.broker  binding       110     184
    org.apache.qpid.broker  broker        1       0
    org.apache.qpid.broker  system        1       0
    org.apache.qpid.broker  subscription  66      92
    org.apache.qpid.broker  connection    40      0
    org.apache.qpid.broker  session       40      157
    org.apache.qpid.broker  queue         54      92
    org.apache.qpid.broker  exchange      10025   0
    org.apache.qpid.broker  vhost         1       0


Can anyone help me identify what is going on here and get things back to normal?   Please let me know what other information would be helpful to provide.

Thanks,
Mike
Overstock.com




Here is a more complete traceback:

2014-05-28 09:57:15 16759 ERROR nova.openstack.common.rpc.impl_qpid [-] Timed out waiting for RPC response: None
2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid Traceback (most recent call last):
2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 358, in ensure
2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 407, in _consume
2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid     nxt_receiver = self.session.next_receiver(timeout=timeout)
2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid   File "<string>", line 6, in next_receiver
2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 663, in next_receiver
2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid     raise Empty
2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid Empty: None
2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid
2014-05-28 09:57:15 16759 ERROR nova.openstack.common.rpc.impl_qpid [-] Timed out waiting for RPC response: None
2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid Traceback (most recent call last):
2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 358, in ensure
2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 407, in _consume
2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid     nxt_receiver = self.session.next_receiver(timeout=timeout)
2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid   File "<string>", line 6, in next_receiver
2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 663, in next_receiver
2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid     raise Empty
2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid Empty: None
2014-05-28 09:57:15 16759 TRACE nova.openstack.common.rpc.impl_qpid
2014-05-28 09:57:16 ERROR nova.api.metadata.handler [req-93548aee-cd7e-46af-8e95-df3e0d6b96a2 None None] Failed to get metadata for ip: 10.8.55.11
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler Traceback (most recent call last):
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/api/metadata/handler.py", line 75, in __call__
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     meta_data = self.get_metadata(remote_address)
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/api/metadata/handler.py", line 57, in get_metadata
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     data = base.get_metadata_by_address(address)
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/api/metadata/base.py", line 385, in get_metadata_by_address
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     fixed_ip = network.API().get_fixed_ip_by_address(ctxt, address)
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 126, in get_fixed_ip_by_address
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     'args': {'address': address}})
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/__init__.py", line 108, in call
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     return _get_impl().call(cfg.CONF, context, topic, msg, timeout)
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 543, in call
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     rpc_amqp.get_connection_pool(conf, Connection))
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 369, in call
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     rv = list(rv)
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 330, in __iter__
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     self.done()
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     self.gen.next()
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 327, in __iter__
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     self._iterator.next()
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 416, in iterconsume
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     yield self.ensure(_error_callback, _consume)
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 362, in ensure
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     error_callback(e)
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 401, in _error_callback
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     raise rpc_common.Timeout()
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler Timeout: Timeout while waiting on RPC response.
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler
2014-05-28 09:57:16 16759 INFO nova.api.ec2 [-] 13656.257934s 10.8.55.11 GET / None:None 500 [None] text/plain text/html
2014-05-28 09:57:16 16759 INFO nova.metadata.wsgi.server [-] 10.8.55.11 - - [28/May/2014 09:57:16] "GET / HTTP/1.1" 500 343 13656.259023

2014-05-28 09:57:16 ERROR nova.api.metadata.handler [req-da2813bf-69a9-491e-b655-ca99c6cf97a2 None None] Failed to get metadata for ip: 10.8.11.17
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler Traceback (most recent call last):
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/api/metadata/handler.py", line 75, in __call__
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     meta_data = self.get_metadata(remote_address)
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/api/metadata/handler.py", line 57, in get_metadata
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     data = base.get_metadata_by_address(address)
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/api/metadata/base.py", line 385, in get_metadata_by_address
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     fixed_ip = network.API().get_fixed_ip_by_address(ctxt, address)
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 126, in get_fixed_ip_by_address
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     'args': {'address': address}})
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/__init__.py", line 108, in call
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     return _get_impl().call(cfg.CONF, context, topic, msg, timeout)
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 543, in call
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     rpc_amqp.get_connection_pool(conf, Connection))
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 369, in call
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     rv = list(rv)
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 330, in __iter__
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     self.done()
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     self.gen.next()
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 327, in __iter__
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     self._iterator.next()
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 416, in iterconsume
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     yield self.ensure(_error_callback, _consume)
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 362, in ensure
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     error_callback(e)
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 401, in _error_callback
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler     raise rpc_common.Timeout()
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler Timeout: Timeout while waiting on RPC response.
2014-05-28 09:57:16 16759 TRACE nova.api.metadata.handler



________________________________

CONFIDENTIALITY NOTICE: This message is intended only for the use and review of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message solely to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify sender immediately by telephone or return email. Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140528/dab9f4dd/attachment.html>