[kolla] Restarting RMQ
When I learned OpenStack at eBay we ran RMQ on dedicated VMs. My new employer runs kolla and everything is in containers. When I was running RMQ on VMs, it would lock up and we would have to restart it on all 3 VMs. If that didn't work, we had a "cold start" procedure where we would stop all 3, delete the contents of /var/lib/rabbitmq/mnesia/ and then run some commands to set the correct config and permissions before starting.
What is the correct way to restart RMQ in kolla? Should I log into the containers and restart services there, or use rabbitmqctl, or just stop and start the containers? Is stop/starting the containers the equivalent of the "cold start" procedure?
I apologize for the nonsense below. So far I have not been able to stop it from being attached to my external emails. I'm working on it.
E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
On Wed, 7 Oct 2020 at 13:11, Braden, Albert C-Albert.Braden@charter.com wrote:
When I learned OpenStack at eBay we ran RMQ on dedicated VMs. My new employer runs kolla and everything is in containers. When I was running RMQ on VMs, it would lock up and we would have to restart it on all 3 VMs. If that didn't work, we had a "cold start" procedure where we would stop all 3, delete the contents of /var/lib/rabbitmq/mnesia/ and then run some commands to set the correct config and permissions before starting.
What is the correct way to restart RMQ in kolla? Should I log into the containers and restart services there, or use rabbitmqctl, or just stop and start the containers? Is stop/starting the containers the equivalent of the "cold start" procedure?
Hi Albert. You shouldn't ever need to exec into containers to restart services - restart the containers. Kolla Ansible has some orchestration in place to avoid restarting all nodes at once. However, the deploy command won't restart containers unless something has changed. For a cold start, you would need to stop the containers (you could use kolla-ansible stop --tags rabbitmq), then run a deploy again. Note that state in Kolla is stored in Docker volumes, which get bind mounted into containers. Mark
I apologize for the nonsense below. So far I have not been able to stop it from being attached to my external emails. I'm working on it.
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
Hi Albert. In my case, I usually do restart the RMQ container directly when RMQ got some issue.
BTW, for the env which use only 2 ethernets ( 1 for Neutron External & another for other interfaces.) and both 1Gb/s speed, or disk I/O is not so powerful for system, often met RMQ brain split. That made me need to restart whole RMA cluster sometimes. I still investigate this issue without hardware or network changes. The temp workaround is increase net.ticktime in RMQ configuration. Although the issue still exist but not so often at least.
Mark Goddard mark@stackhpc.com 於 2020年10月7日 週三 下午9:19寫道：
On Wed, 7 Oct 2020 at 13:11, Braden, Albert C-Albert.Braden@charter.com wrote:
When I learned OpenStack at eBay we ran RMQ on dedicated VMs. My new
employer runs kolla and everything is in containers. When I was running RMQ on VMs, it would lock up and we would have to restart it on all 3 VMs. If that didn't work, we had a "cold start" procedure where we would stop all 3, delete the contents of /var/lib/rabbitmq/mnesia/ and then run some commands to set the correct config and permissions before starting.
What is the correct way to restart RMQ in kolla? Should I log into the
containers and restart services there, or use rabbitmqctl, or just stop and start the containers? Is stop/starting the containers the equivalent of the "cold start" procedure?
Hi Albert. You shouldn't ever need to exec into containers to restart services - restart the containers. Kolla Ansible has some orchestration in place to avoid restarting all nodes at once. However, the deploy command won't restart containers unless something has changed. For a cold start, you would need to stop the containers (you could use kolla-ansible stop --tags rabbitmq), then run a deploy again. Note that state in Kolla is stored in Docker volumes, which get bind mounted into containers. Mark
I apologize for the nonsense below. So far I have not been able to stop
it from being attached to my external emails. I'm working on it.
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
participants (3)
-
Braden, Albert
-
Eddie Yen
-
Mark Goddard