[ops] RabbitMQ Config suggestions for > 100 nodes?
Hi, we run some openstack clusters and currently see issues with a lot connections in rabbitmq. So I would like to ask if someone here has some knowhow about running openstack on >100 nodes he is able to share. Our setup: * 3 Nodes Control-Plane * RabbitMQ Cluster with "sync_mode: all" (had issues during node-failure in the past without this) * transport_url contains all 3 nodes Most of the connections are neutron-based. We already played a bit with max_pool and max_pool_overload, but didnt change anything. As next try I would add some haproxy-lb and change the transport_url to the VIP, this should afaik reduce the amount of connections to 1/3, but I dont know if there are any drawbacks jet. Any suggestions? Thanks a lot, Fabian
On 4/17/20 9:02 AM, Fabian Zimmermann wrote:
As next try I would add some haproxy-lb and change the transport_url to the VIP, this should afaik reduce the amount of connections to 1/3, but I dont know if there are any drawbacks jet.
To my experience, you really don't need to do this, oslo.messaging takes care of reconnecting to another node if one fails. Though you may want to randomize the order in transport_url. If you use puppet, you can use the fqdn_rotate() function to achieve it. Cheers, Thomas Goirand (zigo)
On Fri, 2020-04-17 at 14:14 +0200, Thomas Goirand wrote:
On 4/17/20 9:02 AM, Fabian Zimmermann wrote:
As next try I would add some haproxy-lb and change the transport_url to the VIP, this should afaik reduce the amount of connections to 1/3, but I dont know if there are any drawbacks jet.
To my experience, you really don't need to do this, oslo.messaging takes care of reconnecting to another node if one fails. Though you may want to randomize the order in transport_url. If you use puppet, you can use the fqdn_rotate() function to achieve it. i think that that behavior is consider deprecated. at least i recall converstaion in nova irc channel before about not advising operators to rely on that for loadblancing and instead suggesting they take the ha proxy approch.
Cheers,
Thomas Goirand (zigo)
participants (3)
-
Fabian Zimmermann
-
Sean Mooney
-
Thomas Goirand