On 8/6/19 5:10 PM, Ben Nemec wrote:
Another thing to check if you're having seemingly inexplicable messaging issues is that there isn't a notification queue filling up somewhere. If notifications are enabled somewhere but nothing is consuming them the size of the queue will eventually grind rabbit to a halt.
I used to check queue sizes through the rabbit web ui, so I have to admit I'm not sure how to do it through the cli.
On the cli. Purging Rabbit notification queues: rabbitmqctl purge_queue versioned_notifications.info rabbitmqctl purge_queue notifications.info Getting the total number of messages in Rabbit: NUM_MESSAGE=$(curl -k -uuser:pass https://192.168.0.1:15671/api/overview 2>/dev/null | jq '.["queue_totals"]["messages"]') The same way, you can get a json output of all queues using this URL: https://192.168.0.1:15671/api/queues and playing with jq, you can do many things like: jq '.[] | select(.name == "versioned_notifications.info") | .messages' jq '.[] | select(.name == "notifications.info") | .messages' jq '.[] | select(.name == "versioned_notifications.error") | .messages' jq '.[] | select(.name == "notifications.error") | .messages' If sum add the output of all of the above 4 queues, you get the total number of notification messages. What I did is outputing to graphite like this: echo "`hostname`.rabbitmq.notifications ${NUM_TOTAL_NOTIF} `date +%s`" \ | nc -w 2 graphite-node-hostname 2003 for the amount of notif + the other types of messages. Doing this every minute makes it possible to graph the number of messages in Grafana, which gives me a nice overview of what's going on with notifications and the rest. I hope this will help someone, Cheers, Thomas Goirand (zigo)