[openstack-ansible]: Move one control node to different chasis
Hi All
We are currently running 3 control nodes with LXC containers deployed using openstack-ansible release ussuri and need to move 1 control node on HP blade from one chasis to another chasis.
Is there any recommended way/procedure to do that? can we directly move the HP blade from one chasis to another? or we need to install new control node and decommission the older one?
Regards Rambo
Hey!
So by design your deployment should be fault-tolerant in case you are using keepalived or anything like that for VIP failovers. So you should be able to safely move blades between chassis. The only thing to remember about, which might cause some effect - is memcached, as it can not be set in any HA mode. What makes things worse, is that memcached clients might be waiting for connection timeout when connection is not refused by remote host, which might happen during host shutdown - where it does not accept connections anymore, but also still reachable. You can check on more details here: https://docs.openstack.org/openstack-ansible-memcached_server/latest/alterna...
But in general it should be really fine.
пн, 9 сент. 2024 г. в 12:32, Ram Ram ram.ramb2412@gmail.com:
Hi All
We are currently running 3 control nodes with LXC containers deployed using openstack-ansible release ussuri and need to move 1 control node on HP blade from one chasis to another chasis.
Is there any recommended way/procedure to do that? can we directly move the HP blade from one chasis to another? or we need to install new control node and decommission the older one?
Regards Rambo
Hi Dmitriy
Thanks for the confirmation/ Just for the memcached containers, if a client is connected to memcached container that is turned off. It doesn't get the cache data from the memcached , it will directly query the database (Galera) , so it will be only impact of performance or any other issue can be expected? Is there any timeout on the openstack services for memcached connection timeout?
Regards Keshav Bareja
On Tue, Sep 10, 2024 at 4:58 PM Dmitriy Rabotyagov noonedeadpunk@gmail.com wrote:
Hey!
So by design your deployment should be fault-tolerant in case you are using keepalived or anything like that for VIP failovers. So you should be able to safely move blades between chassis. The only thing to remember about, which might cause some effect - is memcached, as it can not be set in any HA mode. What makes things worse, is that memcached clients might be waiting for connection timeout when connection is not refused by remote host, which might happen during host shutdown - where it does not accept connections anymore, but also still reachable. You can check on more details here:
https://docs.openstack.org/openstack-ansible-memcached_server/latest/alterna...
But in general it should be really fine.
пн, 9 сент. 2024 г. в 12:32, Ram Ram ram.ramb2412@gmail.com:
Hi All
We are currently running 3 control nodes with LXC containers deployed
using openstack-ansible release ussuri and need to move 1 control node on HP blade from one chasis to another chasis.
Is there any recommended way/procedure to do that? can we directly move the HP blade from one chasis to another? or we need to install new control node and decommission the older one?
Regards Rambo
So yes, the problem I was describing, is that a client can wait 10 seconds [1] for memcached to reply in case connection is not just refused instantly, which might happen due to firewall/network setup/during node shutting down. It does not always happen, but it's a possibility anyway, which is good to be aware about. And the client won't go to the galera until this timeout is reached. Since some requests go through multiple APIs, this means that a bunch of clients can have the same behaviour up to a point of getting reply timeouts intermittently from HAProxy.
[1] https://github.com/openstack/oslo.cache/blob/unmaintained/wallaby/oslo_cache...
вт, 10 сент. 2024 г. в 18:59, Rambo Rambo ram.ramb2412@gmail.com:
Hi Dmitriy
Thanks for the confirmation/ Just for the memcached containers, if a client is connected to memcached container that is turned off. It doesn't get the cache data from the memcached , it will directly query the database (Galera) , so it will be only impact of performance or any other issue can be expected? Is there any timeout on the openstack services for memcached connection timeout?
Regards Keshav Bareja
On Tue, Sep 10, 2024 at 4:58 PM Dmitriy Rabotyagov noonedeadpunk@gmail.com wrote:
Hey!
So by design your deployment should be fault-tolerant in case you are using keepalived or anything like that for VIP failovers. So you should be able to safely move blades between chassis. The only thing to remember about, which might cause some effect - is memcached, as it can not be set in any HA mode. What makes things worse, is that memcached clients might be waiting for connection timeout when connection is not refused by remote host, which might happen during host shutdown - where it does not accept connections anymore, but also still reachable. You can check on more details here: https://docs.openstack.org/openstack-ansible-memcached_server/latest/alterna...
But in general it should be really fine.
пн, 9 сент. 2024 г. в 12:32, Ram Ram ram.ramb2412@gmail.com:
Hi All
We are currently running 3 control nodes with LXC containers deployed using openstack-ansible release ussuri and need to move 1 control node on HP blade from one chasis to another chasis.
Is there any recommended way/procedure to do that? can we directly move the HP blade from one chasis to another? or we need to install new control node and decommission the older one?
Regards Rambo
participants (3)
-
Dmitriy Rabotyagov
-
Ram Ram
-
Rambo Rambo