RMQ notifications.info has no consumers
I installed the RMQ management plugin in my dev and qa clusters and noticed a queue called notifications.info that has 0 consumers: root@us01odc-qa-ctrl1:~# rabbitmqctl list_queues|grep notific notifications.info 15496 If I cold-start RMQ it starts without a notifications.info queue but after I create a VM it appears with 1 or 2 messages and then gradually grows over time. The messages in it appear to be Openstack operations: {"oslo.message": "{\"_context_domain\": null, \"_context_roles\": [\"admin\"], \"_context_global_request_id\": null, \"event_type\": \"dns.domain.exists\", \"_context_edit_managed_records\": false, \"_context_service_catalog\": null, \"timestamp\": \"2019-10-10 21:02:01.524052\", \"_context_all_tenants\": true, \"_unique_id\": \"326b8e30d10f40279d806384e7023954\", \"_context_resource_uuid\": null, \"_context_request_id\": \"req-a16c0cd5-94c3-4bf7-90ed-3ab677cc8e66\", \"_context_is_admin_project\": true, \"_context_tsigkey_id\": null, \"_context_client_addr\": null, \"payload\": {\"updated_at\": \"2019-09-24T15:08:55.000000\", \"minimum\": 3600, \"ttl\": 60, \"serial\": 1569337730, \"deleted_at\": null, \"id\": \"67840d5f-6b98-449c-80c4-4d3e3ee74d19\", \"parent_zone_id\": null, \"retry\": 600, \"transferred_at\": null, \"version\": 470, \"type\": \"PRIMARY\", \"email\": \"linux-admin-core@synopsys.com\", \"status\": \"ACTIVE\", \"description\": null, \"deleted\": \"0\", \"shard\": 1656, \"action\": \"NONE\", \"expire\": 86400, \"audit_period_beginning\": \"2019-10-10T20:02:01.486946\", \"masters\": [], \"name\": \"sg.us01-qa.synopsys.com.\", \"tenant_id\": \"80a60cda79be4ec48e12683478f0cf3b\", \"created_at\": \"2019-09-05T17:18:47.000000\", \"pool_id\": \"794ccc2c-d751-44fe-b57f-8894c9f5c842\", \"refresh\": 3545, \"delayed_notify\": false, \"audit_period_ending\": \"2019-10-10T21:02:01.486946\", \"attributes\": []}, \"_context_auth_token\": null, \"_context_system_scope\": null, \"_context_original_tenant\": null, \"_context_user_identity\": \"- - - - -\", \"_context_show_deleted\": false, \"_context_tenant\": null, \"_context_hide_counts\": false, \"priority\": \"INFO\", \"_context_read_only\": false, \"_context_is_admin\": true, \"_context_abandon\": null, \"_context_project_domain\": null, \"_context_user\": null, \"_context_user_domain\": null, \"publisher_id\": \"producer.us01odc-qa-ctrl2\", \"message_id\": \"0ceae8e3-e6df-4b08-8b1e-c6d8e75d456d\", \"_context_project\": null}", "oslo.version": "2.0"} I didn't install the management plugin in prod, but it appears that I have the same issue there: root@us01odc-p01-ctrl1:~# rabbitmqctl list_queues|grep notific notifications.info 226869 What should be consuming the notifications.info queue?
On 21/10/2019 17:54, Albert Braden wrote:
I installed the RMQ management plugin in my dev and qa clusters and noticed a queue called notifications.info that has 0 consumers:
root@us01odc-qa-ctrl1:~# rabbitmqctl list_queues|grep notific
notifications.info 15496
If I cold-start RMQ it starts without a notifications.info queue but after I create a VM it appears with 1 or 2 messages and then gradually grows over time. The messages in it appear to be Openstack operations: <snip/>
I didn’t install the management plugin in prod, but it appears that I have the same issue there:
root@us01odc-p01-ctrl1:~# rabbitmqctl list_queues|grep notific
notifications.info 226869
What should be consuming the notifications.info queue?
Usually something like ceilometer or other telemetry projects, or projects like Designate and searchlight that don't always have hooks into nova or neutron, or custom tooling. If nothing is consuming them,. you can either set the ttl on the RMQ queue [1] to something small (just in case you want to look at them) or disable them [2]
1 - https://www.rabbitmq.com/ttl.html 2 - e.g. in Nova https://docs.openstack.org/nova/latest/reference/notifications.html
Yes, fully agree with Graham. I do rabbitmqctl queue_purge notifications.* In cron every hour + add policy to remove them after 1 min... Also double check other queues: rabbitmqctl list_qeues | awk '{if($2>5)print $0}' or smth like that. On Mon, 21 Oct 2019, 20:17 Graham Hayes, <gr@ham.ie> wrote:
On 21/10/2019 17:54, Albert Braden wrote:
I installed the RMQ management plugin in my dev and qa clusters and noticed a queue called notifications.info that has 0 consumers:
root@us01odc-qa-ctrl1:~# rabbitmqctl list_queues|grep notific
notifications.info 15496
If I cold-start RMQ it starts without a notifications.info queue but after I create a VM it appears with 1 or 2 messages and then gradually grows over time. The messages in it appear to be Openstack operations: <snip/>
I didn’t install the management plugin in prod, but it appears that I have the same issue there:
root@us01odc-p01-ctrl1:~# rabbitmqctl list_queues|grep notific
notifications.info 226869
What should be consuming the notifications.info queue?
Usually something like ceilometer or other telemetry projects, or projects like Designate and searchlight that don't always have hooks into nova or neutron, or custom tooling.
If nothing is consuming them,. you can either set the ttl on the RMQ queue [1] to something small (just in case you want to look at them) or disable them [2]
1 - https://www.rabbitmq.com/ttl.html 2 - e.g. in Nova https://docs.openstack.org/nova/latest/reference/notifications.html
I just set max length on the queue to something like 1k. Sent from my iPhone On Oct 21, 2019, at 11:46 AM, Ruslanas Gžibovskis <ruslanas@lpic.lt> wrote: Yes, fully agree with Graham. I do rabbitmqctl queue_purge notifications.* In cron every hour + add policy to remove them after 1 min... Also double check other queues: rabbitmqctl list_qeues | awk '{if($2>5)print $0}' or smth like that. On Mon, 21 Oct 2019, 20:17 Graham Hayes, <gr@ham.ie<mailto:gr@ham.ie>> wrote: On 21/10/2019 17:54, Albert Braden wrote:
I installed the RMQ management plugin in my dev and qa clusters and noticed a queue called notifications.info<https://urldefense.com/v3/__http://notifications.info__;!2E0gRdhhnqPNNL0!zCpV0FxF5MFe5k4lsJIfGupVIchBnF7f0j56B6N1aU9SCSliAtL8Rvs88f4lYl7L7Q$> that has 0 consumers:
root@us01odc-qa-ctrl1:~# rabbitmqctl list_queues|grep notific
notifications.info<https://urldefense.com/v3/__http://notifications.info__;!2E0gRdhhnqPNNL0!zCpV0FxF5MFe5k4lsJIfGupVIchBnF7f0j56B6N1aU9SCSliAtL8Rvs88f4lYl7L7Q$> 15496
If I cold-start RMQ it starts without a notifications.info<https://urldefense.com/v3/__http://notifications.info__;!2E0gRdhhnqPNNL0!zCpV0FxF5MFe5k4lsJIfGupVIchBnF7f0j56B6N1aU9SCSliAtL8Rvs88f4lYl7L7Q$> queue but after I create a VM it appears with 1 or 2 messages and then gradually grows over time. The messages in it appear to be Openstack operations: <snip/>
I didn’t install the management plugin in prod, but it appears that I have the same issue there:
root@us01odc-p01-ctrl1:~# rabbitmqctl list_queues|grep notific
notifications.info<https://urldefense.com/v3/__http://notifications.info__;!2E0gRdhhnqPNNL0!zCpV0FxF5MFe5k4lsJIfGupVIchBnF7f0j56B6N1aU9SCSliAtL8Rvs88f4lYl7L7Q$> 226869
What should be consuming the notifications.info<https://urldefense.com/v3/__http://notifications.info__;!2E0gRdhhnqPNNL0!zCpV0FxF5MFe5k4lsJIfGupVIchBnF7f0j56B6N1aU9SCSliAtL8Rvs88f4lYl7L7Q$> queue?
Usually something like ceilometer or other telemetry projects, or projects like Designate and searchlight that don't always have hooks into nova or neutron, or custom tooling. If nothing is consuming them,. you can either set the ttl on the RMQ queue [1] to something small (just in case you want to look at them) or disable them [2]
1 - https://www.rabbitmq.com/ttl.html<https://urldefense.com/v3/__https://www.rabbitmq.com/ttl.html__;!2E0gRdhhnqPNNL0!zCpV0FxF5MFe5k4lsJIfGupVIchBnF7f0j56B6N1aU9SCSliAtL8Rvs88f78akNFdg$> 2 - e.g. in Nova https://docs.openstack.org/nova/latest/reference/notifications.html<https://urldefense.com/v3/__https://docs.openstack.org/nova/latest/reference/notifications.html__;!2E0gRdhhnqPNNL0!zCpV0FxF5MFe5k4lsJIfGupVIchBnF7f0j56B6N1aU9SCSliAtL8Rvs88f4RsJCBjw$>
participants (4)
-
Albert Braden
-
Erik Olof Gunnar Andersson
-
Graham Hayes
-
Ruslanas Gžibovskis