[Openstack] impl_kombu.FanoutConsumer uses the same name for an exclusive queue on reconnecting
alex at zadarastorage.com
Wed Dec 26 18:05:11 UTC 2012
I am seeing an issue with the FanoutConsumer: currently each nova service
declares a FanoutConsumer (in addition to TopicConsumer and DirectConsumer).
The FanoutConsumer declares a queue on the rabbitmq server, whose name is
“<topic>_fanout_<UUID>”, for example:
This queue is marked as “exclusive” and “auto-delete”. “Exclusive” means
that the queue can be used by a single Connection. However, when the
FanoutConsumer detects disconnection and attempts to reconnect, it creates a
new BrokerConnection, but uses the same queue name. In most cases, since the
queue is marked as “auto-delete”, the rabbitmq server deletes the queue
automatically on disconnection (after some timeout). So when FanoutConsumer
tries to reconnect, the previous queue is gone. In some cases, however, the
queue has not been deleted on the server yet, so when FanoutConsumer
reconnects, there is an exception on the server:
=ERROR REPORT==== 26-Dec-2012::18:44:55 ===
connection <0.747.0>, channel 1 - error:
"cannot obtain exclusive access to locked queue
'volume_fanout_dcc7e1a2d71845d29cd53cfab8920289' in vhost '/'",
As a result, the call to queue.declare() that leads to exception on the
server is stuck and never returns. So the service is not able to receive any
Possible ways to fix this:
1) Use non-exclusive queues (but still auto-delete). (Note: exclusive queues
are also used by DirectConsumers.) I am not sure what is the implication of
that in nova. I think it should work, because all exclusive queues defined
by nova have unique names anyways.
2) Use a different queue name each time FanoutConsumer reconnects. For
DirectConsumer, this still leave the issue not fixed, although with lower
probability to happen, I believe.
I implemented a quick fix as in 2), and ensured that it resolves the issue.
According to the latest nova source code, the same issue still exists there.
Another note: “exclusive=true” argument is also used when creating Exchanges
on the publisher side. However, Exchange C’tor does not receive “exclusive”
parameter at all (looking at kombu sources), so I am not sure what this does
(latest nova code also has this).
Can anybody of nova rpc developers please comment on how you think this
issue should be addressed?
More information about the Openstack