[openstack-dev] auto-delete in amqp "reply_*" queues in OpenStack

Dmitry Mescheryakov dmescheryakov at mirantis.com
Mon Mar 24 19:27:50 UTC 2014


I see two possible explanations for these 5 remaining queues:

 * They were indeed recreated by 'compute' services. I.e. controller
service send some command over rpc and then it was shut down. Its
reply queue was automatically deleted, since its only consumer was
disconnected. The compute services replied after that and so recreated
the queue. According to Rabbit MQ docs, such queue will be stuck alive
indefinitely, since it will never have a consumer.

 * Possibly there are services on compute nodes which initiate RPC
calls themselves. I don't know OpenStack architecture enough to say if
services running on compute nodes do so. In that case these 5 queues
are still used by compute services.

Do Rabbit MQ management tools (web or cli) allow to view active
consumers for queues? If yes, then you can find out which of the cases
above you encountered. Or it maybe be some third case I didn't account
for :-)

> I assume that those 5 queues are (re)created by the services running on the
> compute nodes, but if that's the case then how would the services running on
> the controller node find out about the names of the queues?

When process initiating rpc call is restarted, there is no way for it
to know about queue it used before for receiving replies. The replies
simply never got back. On the other hand, the restarted process does
not know about calls it did before the restart, so it is not a big
loss anyway.

For clarity, here is a simplified algorithm RPC client (the one
initiating RPC call) uses:

msg_id = uuid.uuid4().hex

if not self.reply_q:
    self.reply_q = 'reply_' + uuid.uuid4().hex

message = {
    'msg_id': msg_id,
    'reply_q': self.reply_q,
    'payload': payload,}

send(message)

reply = wait_for_reply(queue=self.reply_q, msg_id=msg_id)


Dmitry



2014-03-24 19:52 GMT+04:00 Chris Friesen <chris.friesen at windriver.com>:
> On 03/24/2014 02:59 AM, Dmitry Mescheryakov wrote:
>>
>> Chris,
>>
>> In oslo.messaging a single reply queue is used to gather results from
>> all the calls. It is created lazily on the first call and is used
>> until the process is killed. I did a quick look at oslo.rpc from
>> oslo-incubator and it seems like it uses the same pattern, which is
>> not surprising since oslo.messaging descends from oslo.rpc. So if you
>> restart some process which does rpc calls (nova-api, I guess), you
>> should see one reply queue gone and another one created instead after
>> some time.
>
>
> Okay, that makes a certain amount of sense.
>
> How does it work for queues used by both the controller and the compute
> node?
>
> If I do a controlled switchover from one controller to another (killing and
> restarting rabbit, nova-api, nova-conductor, nova-scheduler, neutron,
> cinder, etc.) I see that the number of reply queues drops from 28 down to 5,
> but those 5 are all ones that existed before.
>
> I assume that those 5 queues are (re)created by the services running on the
> compute nodes, but if that's the case then how would the services running on
> the controller node find out about the names of the queues?
>
>
> Chris
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



More information about the OpenStack-dev mailing list