On 8/6/19 5:10 PM, Ben Nemec wrote:
Another thing to check if you're having seemingly inexplicable messaging issues is that there isn't a notification queue filling up somewhere. If notifications are enabled somewhere but nothing is consuming them the size of the queue will eventually grind rabbit to a halt.
I used to check queue sizes through the rabbit web ui, so I have to admit I'm not sure how to do it through the cli.
On the cli. Purging Rabbit notification queues:
rabbitmqctl purge_queue versioned_notifications.info rabbitmqctl purge_queue notifications.info
Getting the total number of messages in Rabbit:
NUM_MESSAGE=$(curl -k -uuser:pass https://192.168.0.1:15671/api/overview 2>/dev/null | jq '.["queue_totals"]["messages"]')
The same way, you can get a json output of all queues using this URL:
https://192.168.0.1:15671/api/queues
and playing with jq, you can do many things like: jq '.[] | select(.name == "versioned_notifications.info") | .messages' jq '.[] | select(.name == "notifications.info") | .messages' jq '.[] | select(.name == "versioned_notifications.error") | .messages' jq '.[] | select(.name == "notifications.error") | .messages'
If sum add the output of all of the above 4 queues, you get the total number of notification messages. What I did is outputing to graphite like this:
echo "`hostname`.rabbitmq.notifications ${NUM_TOTAL_NOTIF} `date +%s`" \ | nc -w 2 graphite-node-hostname 2003
for the amount of notif + the other types of messages.
Doing this every minute makes it possible to graph the number of messages in Grafana, which gives me a nice overview of what's going on with notifications and the rest.
I hope this will help someone, Cheers,
Thomas Goirand (zigo)