Open Stack

Switching to openstack-dev as this seems like a more apporopriate place to have this discussion.  We have been running into an intermittent issue where workers stop consuming messages and I wonder if it is the same issue. It would be great to see the code for the quick fix, but it seems like switching to non-exclusive queues might be more correct.

Vish

On Dec 26, 2012, at 10:05 AM, Alex Lyakas <alex at zadarastorage.com> wrote:

> Hi,
> I am seeing an issue with the FanoutConsumer: currently each nova service declares a FanoutConsumer (in addition to TopicConsumer and DirectConsumer). The FanoutConsumer declares a queue on the rabbitmq server, whose name is “<topic>_fanout_<UUID>”, for example: compute_fanout_e494cb57953b4e6d9dc68c72fb3e26b6.
> 
> This queue is marked as “exclusive” and “auto-delete”. “Exclusive” means that the queue can be used by a single Connection. However, when the FanoutConsumer detects disconnection and attempts to reconnect, it creates a new BrokerConnection, but uses the same queue name. In most cases, since the queue is marked as “auto-delete”, the rabbitmq server deletes the queue automatically on disconnection (after some timeout). So when FanoutConsumer tries to reconnect, the previous queue is gone. In some cases, however, the queue has not been deleted on the server yet, so when FanoutConsumer reconnects, there is an exception on the server:
> =ERROR REPORT==== 26-Dec-2012::18:44:55 ===
> connection <0.747.0>, channel 1 - error:
> {amqp_error,resource_locked,
> "cannot obtain exclusive access to locked queue 'volume_fanout_dcc7e1a2d71845d29cd53cfab8920289' in vhost '/'",
> 'queue.declare'}
> 
> As a result, the call to queue.declare() that leads to exception on the server is stuck and never returns. So the service is not able to receive any messages.
> 
> Possible ways to fix this:
> 1) Use non-exclusive queues (but still auto-delete). (Note: exclusive queues are also used by DirectConsumers.) I am not sure what is the implication of that in nova. I think it should work, because all exclusive queues defined by nova have unique names anyways.
> 2) Use a different queue name each time FanoutConsumer reconnects. For DirectConsumer, this still leave the issue not fixed, although with lower probability to happen, I believe.
> 
> I implemented a quick fix as in 2), and ensured that it resolves the issue.
> 
> According to the latest nova source code, the same issue still exists there.
> 
> Another note: “exclusive=true” argument is also used when creating Exchanges on the publisher side. However, Exchange C’tor does not receive “exclusive” parameter at all (looking at kombu sources), so I am not sure what this does (latest nova code also has this).
> 
> Can anybody of nova rpc developers please comment on how you think this issue should be addressed?
> 
> Thanks,
> Alex.
> 
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack at lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp