[kolla] [train] haproxy and controller restart causes user impact
We have our haproxy and controller nodes on KVM hosts. When those KVM hosts are restarted, customers who are building or deleting VMs see impact. VMs may go into error status, fail to get DNS records, fail to delete, etc. The obvious reason is because traffic that is being routed to the haproxy on the restarting KVM is lost. If we manually fail over haproxy before restarting the KVM, will that be sufficient to stop traffic being lost, or do we also need to do something with the controller?
Are you running your stack on top of the kvm virtual machine? How many controller nodes do you have? mostly rabbitMQ causing issues if you restart controller nodes. On Thu, May 11, 2023 at 8:34 AM Albert Braden <ozzzo@yahoo.com> wrote:
We have our haproxy and controller nodes on KVM hosts. When those KVM hosts are restarted, customers who are building or deleting VMs see impact. VMs may go into error status, fail to get DNS records, fail to delete, etc. The obvious reason is because traffic that is being routed to the haproxy on the restarting KVM is lost. If we manually fail over haproxy before restarting the KVM, will that be sufficient to stop traffic being lost, or do we also need to do something with the controller?
Hi Albert, how is your haproxy placement controlled, something like pacemaker or similar? I would always do a failover when I'm aware of interruptions (maintenance window), that should speed things up for clients. We have a pacemaker controlled HA control plane, it takes more time until pacemaker realizes that the resource is gone if I just rebooted a server without failing over. I have no benchmarks though. There's always a risk of losing a couple of requests during the failover but we didn't have complaints yet, I believe most of the components try to resend the lost messages. In one of our customer's cluster with many resources (they also use terraform) I haven't seen issues during a regular maintenance window. When they had a DNS outage a few months back it resulted in a mess, manual cleaning was necessary, but the regular failovers seem to work just fine. And I don't see rabbitmq issues either after rebooting a server, usually the haproxy (and virtual IP) failover suffice to prevent interruptions. Regards, Eugen Zitat von Satish Patel <satish.txt@gmail.com>:
Are you running your stack on top of the kvm virtual machine? How many controller nodes do you have? mostly rabbitMQ causing issues if you restart controller nodes.
On Thu, May 11, 2023 at 8:34 AM Albert Braden <ozzzo@yahoo.com> wrote:
We have our haproxy and controller nodes on KVM hosts. When those KVM hosts are restarted, customers who are building or deleting VMs see impact. VMs may go into error status, fail to get DNS records, fail to delete, etc. The obvious reason is because traffic that is being routed to the haproxy on the restarting KVM is lost. If we manually fail over haproxy before restarting the KVM, will that be sufficient to stop traffic being lost, or do we also need to do something with the controller?
We use keepalived and exabgp to manage failover for haproxy. That works but it takes a few minutes, and during those few minutes customers experience impact. We tell them to not build/delete VMs during patching, but they still do, and then complain about the failures. We're planning to experiment with adding a "manual" haproxy failover to our patching automation, but I'm wondering if there is anything on the controller that needs to be failed over or disabled before rebooting the KVM. I looked at the "remove from cluster" and "add to cluster" procedures but that seems unnecessarily cumbersome for rebooting the KVM. On Friday, May 12, 2023, 03:42:42 AM EDT, Eugen Block <eblock@nde.ag> wrote: Hi Albert, how is your haproxy placement controlled, something like pacemaker or similar? I would always do a failover when I'm aware of interruptions (maintenance window), that should speed things up for clients. We have a pacemaker controlled HA control plane, it takes more time until pacemaker realizes that the resource is gone if I just rebooted a server without failing over. I have no benchmarks though. There's always a risk of losing a couple of requests during the failover but we didn't have complaints yet, I believe most of the components try to resend the lost messages. In one of our customer's cluster with many resources (they also use terraform) I haven't seen issues during a regular maintenance window. When they had a DNS outage a few months back it resulted in a mess, manual cleaning was necessary, but the regular failovers seem to work just fine. And I don't see rabbitmq issues either after rebooting a server, usually the haproxy (and virtual IP) failover suffice to prevent interruptions. Regards, Eugen Zitat von Satish Patel <satish.txt@gmail.com>:
Are you running your stack on top of the kvm virtual machine? How many controller nodes do you have? mostly rabbitMQ causing issues if you restart controller nodes.
On Thu, May 11, 2023 at 8:34 AM Albert Braden <ozzzo@yahoo.com> wrote:
We have our haproxy and controller nodes on KVM hosts. When those KVM hosts are restarted, customers who are building or deleting VMs see impact. VMs may go into error status, fail to get DNS records, fail to delete, etc. The obvious reason is because traffic that is being routed to the haproxy on the restarting KVM is lost. If we manually fail over haproxy before restarting the KVM, will that be sufficient to stop traffic being lost, or do we also need to do something with the controller?
Don't expect zero issue when you reboot the controller. It won't be user transparent. your computer nodes and other services still hang on old connections (rabbitmq/amqp) etc and that takes some time to get settled. Curious why are you running control plane service on KVM and second question why do you need them reboot frequently? I have physical nodes for the control plane and we see strange issues whenever we shouldn't use one of the controllers for maintenance. On Fri, May 12, 2023 at 2:59 PM Albert Braden <ozzzo@yahoo.com> wrote:
We use keepalived and exabgp to manage failover for haproxy. That works but it takes a few minutes, and during those few minutes customers experience impact. We tell them to not build/delete VMs during patching, but they still do, and then complain about the failures.
We're planning to experiment with adding a "manual" haproxy failover to our patching automation, but I'm wondering if there is anything on the controller that needs to be failed over or disabled before rebooting the KVM. I looked at the "remove from cluster" and "add to cluster" procedures but that seems unnecessarily cumbersome for rebooting the KVM. On Friday, May 12, 2023, 03:42:42 AM EDT, Eugen Block <eblock@nde.ag> wrote:
Hi Albert,
how is your haproxy placement controlled, something like pacemaker or similar? I would always do a failover when I'm aware of interruptions (maintenance window), that should speed things up for clients. We have a pacemaker controlled HA control plane, it takes more time until pacemaker realizes that the resource is gone if I just rebooted a server without failing over. I have no benchmarks though. There's always a risk of losing a couple of requests during the failover but we didn't have complaints yet, I believe most of the components try to resend the lost messages. In one of our customer's cluster with many resources (they also use terraform) I haven't seen issues during a regular maintenance window. When they had a DNS outage a few months back it resulted in a mess, manual cleaning was necessary, but the regular failovers seem to work just fine. And I don't see rabbitmq issues either after rebooting a server, usually the haproxy (and virtual IP) failover suffice to prevent interruptions.
Regards, Eugen
Zitat von Satish Patel <satish.txt@gmail.com>:
Are you running your stack on top of the kvm virtual machine? How many controller nodes do you have? mostly rabbitMQ causing issues if you restart controller nodes.
On Thu, May 11, 2023 at 8:34 AM Albert Braden <ozzzo@yahoo.com> wrote:
We have our haproxy and controller nodes on KVM hosts. When those KVM hosts are restarted, customers who are building or deleting VMs see impact. VMs may go into error status, fail to get DNS records, fail to delete, etc. The obvious reason is because traffic that is being routed to the haproxy on the restarting KVM is lost. If we manually fail over haproxy before restarting the KVM, will that be sufficient to stop traffic being lost, or do we also need to do something with the controller?
My two cents, If you are still running Train (which is EOL) then please upgrade to the next or latest release, you never know what bug causing the issue. On Fri, May 12, 2023 at 4:51 PM Satish Patel <satish.txt@gmail.com> wrote:
Don't expect zero issue when you reboot the controller. It won't be user transparent. your computer nodes and other services still hang on old connections (rabbitmq/amqp) etc and that takes some time to get settled.
Curious why are you running control plane service on KVM and second question why do you need them reboot frequently?
I have physical nodes for the control plane and we see strange issues whenever we shouldn't use one of the controllers for maintenance.
On Fri, May 12, 2023 at 2:59 PM Albert Braden <ozzzo@yahoo.com> wrote:
We use keepalived and exabgp to manage failover for haproxy. That works but it takes a few minutes, and during those few minutes customers experience impact. We tell them to not build/delete VMs during patching, but they still do, and then complain about the failures.
We're planning to experiment with adding a "manual" haproxy failover to our patching automation, but I'm wondering if there is anything on the controller that needs to be failed over or disabled before rebooting the KVM. I looked at the "remove from cluster" and "add to cluster" procedures but that seems unnecessarily cumbersome for rebooting the KVM. On Friday, May 12, 2023, 03:42:42 AM EDT, Eugen Block <eblock@nde.ag> wrote:
Hi Albert,
how is your haproxy placement controlled, something like pacemaker or similar? I would always do a failover when I'm aware of interruptions (maintenance window), that should speed things up for clients. We have a pacemaker controlled HA control plane, it takes more time until pacemaker realizes that the resource is gone if I just rebooted a server without failing over. I have no benchmarks though. There's always a risk of losing a couple of requests during the failover but we didn't have complaints yet, I believe most of the components try to resend the lost messages. In one of our customer's cluster with many resources (they also use terraform) I haven't seen issues during a regular maintenance window. When they had a DNS outage a few months back it resulted in a mess, manual cleaning was necessary, but the regular failovers seem to work just fine. And I don't see rabbitmq issues either after rebooting a server, usually the haproxy (and virtual IP) failover suffice to prevent interruptions.
Regards, Eugen
Zitat von Satish Patel <satish.txt@gmail.com>:
Are you running your stack on top of the kvm virtual machine? How many controller nodes do you have? mostly rabbitMQ causing issues if you restart controller nodes.
On Thu, May 11, 2023 at 8:34 AM Albert Braden <ozzzo@yahoo.com> wrote:
We have our haproxy and controller nodes on KVM hosts. When those KVM hosts are restarted, customers who are building or deleting VMs see impact. VMs may go into error status, fail to get DNS records, fail to delete, etc. The obvious reason is because traffic that is being routed to the haproxy on the restarting KVM is lost. If we manually fail over haproxy before restarting the KVM, will that be sufficient to stop traffic being lost, or do we also need to do something with the controller?
We reboot quarterly for patching. On Friday, May 12, 2023, 04:58:45 PM EDT, Satish Patel <satish.txt@gmail.com> wrote: Don't expect zero issue when you reboot the controller. It won't be user transparent. your computer nodes and other services still hang on old connections (rabbitmq/amqp) etc and that takes some time to get settled. Curious why are you running control plane service on KVM and second question why do you need them reboot frequently? I have physical nodes for the control plane and we see strange issues whenever we shouldn't use one of the controllers for maintenance. On Fri, May 12, 2023 at 2:59 PM Albert Braden <ozzzo@yahoo.com> wrote: We use keepalived and exabgp to manage failover for haproxy. That works but it takes a few minutes, and during those few minutes customers experience impact. We tell them to not build/delete VMs during patching, but they still do, and then complain about the failures. We're planning to experiment with adding a "manual" haproxy failover to our patching automation, but I'm wondering if there is anything on the controller that needs to be failed over or disabled before rebooting the KVM. I looked at the "remove from cluster" and "add to cluster" procedures but that seems unnecessarily cumbersome for rebooting the KVM. On Friday, May 12, 2023, 03:42:42 AM EDT, Eugen Block <eblock@nde.ag> wrote: Hi Albert, how is your haproxy placement controlled, something like pacemaker or similar? I would always do a failover when I'm aware of interruptions (maintenance window), that should speed things up for clients. We have a pacemaker controlled HA control plane, it takes more time until pacemaker realizes that the resource is gone if I just rebooted a server without failing over. I have no benchmarks though. There's always a risk of losing a couple of requests during the failover but we didn't have complaints yet, I believe most of the components try to resend the lost messages. In one of our customer's cluster with many resources (they also use terraform) I haven't seen issues during a regular maintenance window. When they had a DNS outage a few months back it resulted in a mess, manual cleaning was necessary, but the regular failovers seem to work just fine. And I don't see rabbitmq issues either after rebooting a server, usually the haproxy (and virtual IP) failover suffice to prevent interruptions. Regards, Eugen Zitat von Satish Patel <satish.txt@gmail.com>:
Are you running your stack on top of the kvm virtual machine? How many controller nodes do you have? mostly rabbitMQ causing issues if you restart controller nodes.
On Thu, May 11, 2023 at 8:34 AM Albert Braden <ozzzo@yahoo.com> wrote:
We have our haproxy and controller nodes on KVM hosts. When those KVM hosts are restarted, customers who are building or deleting VMs see impact. VMs may go into error status, fail to get DNS records, fail to delete, etc. The obvious reason is because traffic that is being routed to the haproxy on the restarting KVM is lost. If we manually fail over haproxy before restarting the KVM, will that be sufficient to stop traffic being lost, or do we also need to do something with the controller?
What's the recommended method for rebooting controllers? Do we need to use the "remove from cluster" and "add to cluster" procedures or is there a better way? https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-host... On Friday, May 12, 2023, 03:04:26 PM EDT, Albert Braden <ozzzo@yahoo.com> wrote: We use keepalived and exabgp to manage failover for haproxy. That works but it takes a few minutes, and during those few minutes customers experience impact. We tell them to not build/delete VMs during patching, but they still do, and then complain about the failures. We're planning to experiment with adding a "manual" haproxy failover to our patching automation, but I'm wondering if there is anything on the controller that needs to be failed over or disabled before rebooting the KVM. I looked at the "remove from cluster" and "add to cluster" procedures but that seems unnecessarily cumbersome for rebooting the KVM. On Friday, May 12, 2023, 03:42:42 AM EDT, Eugen Block <eblock@nde.ag> wrote: Hi Albert, how is your haproxy placement controlled, something like pacemaker or similar? I would always do a failover when I'm aware of interruptions (maintenance window), that should speed things up for clients. We have a pacemaker controlled HA control plane, it takes more time until pacemaker realizes that the resource is gone if I just rebooted a server without failing over. I have no benchmarks though. There's always a risk of losing a couple of requests during the failover but we didn't have complaints yet, I believe most of the components try to resend the lost messages. In one of our customer's cluster with many resources (they also use terraform) I haven't seen issues during a regular maintenance window. When they had a DNS outage a few months back it resulted in a mess, manual cleaning was necessary, but the regular failovers seem to work just fine. And I don't see rabbitmq issues either after rebooting a server, usually the haproxy (and virtual IP) failover suffice to prevent interruptions. Regards, Eugen Zitat von Satish Patel <satish.txt@gmail.com>:
Are you running your stack on top of the kvm virtual machine? How many controller nodes do you have? mostly rabbitMQ causing issues if you restart controller nodes.
On Thu, May 11, 2023 at 8:34 AM Albert Braden <ozzzo@yahoo.com> wrote:
We have our haproxy and controller nodes on KVM hosts. When those KVM hosts are restarted, customers who are building or deleting VMs see impact. VMs may go into error status, fail to get DNS records, fail to delete, etc. The obvious reason is because traffic that is being routed to the haproxy on the restarting KVM is lost. If we manually fail over haproxy before restarting the KVM, will that be sufficient to stop traffic being lost, or do we also need to do something with the controller?
Hi Albert, sorry, I'm swamped with different stuff right now. I just took a glance at the docs you mentioned and it seems way too much for something simple as a controller restart to actually remove hosts, that should definitely not be necessary. I'm not familiar with kolla or exabgp, but can you describe what exactly takes that long to failover? Maybe that could be improved? And can you limit the failing requests to a specific service (volumes, network ports, etc.) or do they all fail? Maybe rabbitmq should be considered after all, you could share your rabbitmq settings from the different openstack services and I will collect mine to compare. And then also the rabbitmq config (policies, vhosts, queues). Regards, Eugen Zitat von Albert Braden <ozzzo@yahoo.com>:
What's the recommended method for rebooting controllers? Do we need to use the "remove from cluster" and "add to cluster" procedures or is there a better way?
https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-host... On Friday, May 12, 2023, 03:04:26 PM EDT, Albert Braden <ozzzo@yahoo.com> wrote:
We use keepalived and exabgp to manage failover for haproxy. That works but it takes a few minutes, and during those few minutes customers experience impact. We tell them to not build/delete VMs during patching, but they still do, and then complain about the failures.
We're planning to experiment with adding a "manual" haproxy failover to our patching automation, but I'm wondering if there is anything on the controller that needs to be failed over or disabled before rebooting the KVM. I looked at the "remove from cluster" and "add to cluster" procedures but that seems unnecessarily cumbersome for rebooting the KVM. On Friday, May 12, 2023, 03:42:42 AM EDT, Eugen Block <eblock@nde.ag> wrote:
Hi Albert,
how is your haproxy placement controlled, something like pacemaker or similar? I would always do a failover when I'm aware of interruptions (maintenance window), that should speed things up for clients. We have a pacemaker controlled HA control plane, it takes more time until pacemaker realizes that the resource is gone if I just rebooted a server without failing over. I have no benchmarks though. There's always a risk of losing a couple of requests during the failover but we didn't have complaints yet, I believe most of the components try to resend the lost messages. In one of our customer's cluster with many resources (they also use terraform) I haven't seen issues during a regular maintenance window. When they had a DNS outage a few months back it resulted in a mess, manual cleaning was necessary, but the regular failovers seem to work just fine. And I don't see rabbitmq issues either after rebooting a server, usually the haproxy (and virtual IP) failover suffice to prevent interruptions.
Regards, Eugen
Zitat von Satish Patel <satish.txt@gmail.com>:
Are you running your stack on top of the kvm virtual machine? How many controller nodes do you have? mostly rabbitMQ causing issues if you restart controller nodes.
On Thu, May 11, 2023 at 8:34 AM Albert Braden <ozzzo@yahoo.com> wrote:
We have our haproxy and controller nodes on KVM hosts. When those KVM hosts are restarted, customers who are building or deleting VMs see impact. VMs may go into error status, fail to get DNS records, fail to delete, etc. The obvious reason is because traffic that is being routed to the haproxy on the restarting KVM is lost. If we manually fail over haproxy before restarting the KVM, will that be sufficient to stop traffic being lost, or do we also need to do something with the controller?
Before we switched to durable queues we were seeing RMQ issues after a restart. Now RMQ is fine after restart, but operations in progress will fail. VMs will fail to build, or not get DNS records. Volumes don't get attached or detached. It looks like haproxy is the issue now; connections continue going to the down node. I think we can fix that by failing over haproxy before rebooting. The problem is, I'm not sure that haproxy is the only issue. All 3 controllers are doing stuff, and when I reboot one, whatever it is doing is likely to fail. Is there an orderly way to stop work from being done on a controller without ruining work that is already in progress, besides removing it from the cluster? Would "kolla-ansible stop" do it? On Tuesday, May 16, 2023, 02:23:59 PM EDT, Eugen Block <eblock@nde.ag> wrote: Hi Albert, sorry, I'm swamped with different stuff right now. I just took a glance at the docs you mentioned and it seems way too much for something simple as a controller restart to actually remove hosts, that should definitely not be necessary. I'm not familiar with kolla or exabgp, but can you describe what exactly takes that long to failover? Maybe that could be improved? And can you limit the failing requests to a specific service (volumes, network ports, etc.) or do they all fail? Maybe rabbitmq should be considered after all, you could share your rabbitmq settings from the different openstack services and I will collect mine to compare. And then also the rabbitmq config (policies, vhosts, queues). Regards, Eugen Zitat von Albert Braden <ozzzo@yahoo.com>:
What's the recommended method for rebooting controllers? Do we need to use the "remove from cluster" and "add to cluster" procedures or is there a better way?
https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-host... On Friday, May 12, 2023, 03:04:26 PM EDT, Albert Braden <ozzzo@yahoo.com> wrote:
We use keepalived and exabgp to manage failover for haproxy. That works but it takes a few minutes, and during those few minutes customers experience impact. We tell them to not build/delete VMs during patching, but they still do, and then complain about the failures.
We're planning to experiment with adding a "manual" haproxy failover to our patching automation, but I'm wondering if there is anything on the controller that needs to be failed over or disabled before rebooting the KVM. I looked at the "remove from cluster" and "add to cluster" procedures but that seems unnecessarily cumbersome for rebooting the KVM. On Friday, May 12, 2023, 03:42:42 AM EDT, Eugen Block <eblock@nde.ag> wrote:
Hi Albert,
how is your haproxy placement controlled, something like pacemaker or similar? I would always do a failover when I'm aware of interruptions (maintenance window), that should speed things up for clients. We have a pacemaker controlled HA control plane, it takes more time until pacemaker realizes that the resource is gone if I just rebooted a server without failing over. I have no benchmarks though. There's always a risk of losing a couple of requests during the failover but we didn't have complaints yet, I believe most of the components try to resend the lost messages. In one of our customer's cluster with many resources (they also use terraform) I haven't seen issues during a regular maintenance window. When they had a DNS outage a few months back it resulted in a mess, manual cleaning was necessary, but the regular failovers seem to work just fine. And I don't see rabbitmq issues either after rebooting a server, usually the haproxy (and virtual IP) failover suffice to prevent interruptions.
Regards, Eugen
Zitat von Satish Patel <satish.txt@gmail.com>:
Are you running your stack on top of the kvm virtual machine? How many controller nodes do you have? mostly rabbitMQ causing issues if you restart controller nodes.
On Thu, May 11, 2023 at 8:34 AM Albert Braden <ozzzo@yahoo.com> wrote:
We have our haproxy and controller nodes on KVM hosts. When those KVM hosts are restarted, customers who are building or deleting VMs see impact. VMs may go into error status, fail to get DNS records, fail to delete, etc. The obvious reason is because traffic that is being routed to the haproxy on the restarting KVM is lost. If we manually fail over haproxy before restarting the KVM, will that be sufficient to stop traffic being lost, or do we also need to do something with the controller?
Hi, I found this [1] reference, it recommends to reduce the kernel option for tcp_retries to reduce the impact of a service interruption: # /etc/kolla/globals.yml haproxy_host_ipv4_tcp_retries2: 6 Apparently, this option was introduced in Victoria [2], it states:
Added a new haproxy configuration variable, haproxy_host_ipv4_tcp_retries2, which allows users to modify this kernel option. This option sets maximum number of times a TCP packet is retransmitted in established state before giving up. The default kernel value is 15, which corresponds to a duration of approximately between 13 to 30 minutes, depending on the retransmission timeout. This variable can be used to mitigate an issue with stuck connections in case of VIP failover, see bug 1917068 for details.
It reads like exactly what you're describing. If I remember correctly, you're still on Train? In that case you'll probably have to configure that setting manually (scripted maybe), it is this value: /proc/sys/net/ipv4/tcp_retries2 The solution in [3] even talks about setting it to 3 for HA deployments. # sysctl -a | grep net.ipv4.tcp_retries2 net.ipv4.tcp_retries2 = 15 Regards, Eugen [1] https://docs.openstack.org/kolla-ansible/latest/reference/high-availability/... [2] https://docs.openstack.org/releasenotes/kolla-ansible/victoria.html [3] https://access.redhat.com/solutions/726753 Zitat von Albert Braden <ozzzo@yahoo.com>:
Before we switched to durable queues we were seeing RMQ issues after a restart. Now RMQ is fine after restart, but operations in progress will fail. VMs will fail to build, or not get DNS records. Volumes don't get attached or detached. It looks like haproxy is the issue now; connections continue going to the down node. I think we can fix that by failing over haproxy before rebooting.
The problem is, I'm not sure that haproxy is the only issue. All 3 controllers are doing stuff, and when I reboot one, whatever it is doing is likely to fail. Is there an orderly way to stop work from being done on a controller without ruining work that is already in progress, besides removing it from the cluster? Would "kolla-ansible stop" do it? On Tuesday, May 16, 2023, 02:23:59 PM EDT, Eugen Block <eblock@nde.ag> wrote:
Hi Albert,
sorry, I'm swamped with different stuff right now. I just took a glance at the docs you mentioned and it seems way too much for something simple as a controller restart to actually remove hosts, that should definitely not be necessary. I'm not familiar with kolla or exabgp, but can you describe what exactly takes that long to failover? Maybe that could be improved? And can you limit the failing requests to a specific service (volumes, network ports, etc.) or do they all fail? Maybe rabbitmq should be considered after all, you could share your rabbitmq settings from the different openstack services and I will collect mine to compare. And then also the rabbitmq config (policies, vhosts, queues).
Regards, Eugen
Zitat von Albert Braden <ozzzo@yahoo.com>:
What's the recommended method for rebooting controllers? Do we need to use the "remove from cluster" and "add to cluster" procedures or is there a better way?
https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-host... On Friday, May 12, 2023, 03:04:26 PM EDT, Albert Braden <ozzzo@yahoo.com> wrote:
We use keepalived and exabgp to manage failover for haproxy. That works but it takes a few minutes, and during those few minutes customers experience impact. We tell them to not build/delete VMs during patching, but they still do, and then complain about the failures.
We're planning to experiment with adding a "manual" haproxy failover to our patching automation, but I'm wondering if there is anything on the controller that needs to be failed over or disabled before rebooting the KVM. I looked at the "remove from cluster" and "add to cluster" procedures but that seems unnecessarily cumbersome for rebooting the KVM. On Friday, May 12, 2023, 03:42:42 AM EDT, Eugen Block <eblock@nde.ag> wrote:
Hi Albert,
how is your haproxy placement controlled, something like pacemaker or similar? I would always do a failover when I'm aware of interruptions (maintenance window), that should speed things up for clients. We have a pacemaker controlled HA control plane, it takes more time until pacemaker realizes that the resource is gone if I just rebooted a server without failing over. I have no benchmarks though. There's always a risk of losing a couple of requests during the failover but we didn't have complaints yet, I believe most of the components try to resend the lost messages. In one of our customer's cluster with many resources (they also use terraform) I haven't seen issues during a regular maintenance window. When they had a DNS outage a few months back it resulted in a mess, manual cleaning was necessary, but the regular failovers seem to work just fine. And I don't see rabbitmq issues either after rebooting a server, usually the haproxy (and virtual IP) failover suffice to prevent interruptions.
Regards, Eugen
Zitat von Satish Patel <satish.txt@gmail.com>:
Are you running your stack on top of the kvm virtual machine? How many controller nodes do you have? mostly rabbitMQ causing issues if you restart controller nodes.
On Thu, May 11, 2023 at 8:34 AM Albert Braden <ozzzo@yahoo.com> wrote:
We have our haproxy and controller nodes on KVM hosts. When those KVM hosts are restarted, customers who are building or deleting VMs see impact. VMs may go into error status, fail to get DNS records, fail to delete, etc. The obvious reason is because traffic that is being routed to the haproxy on the restarting KVM is lost. If we manually fail over haproxy before restarting the KVM, will that be sufficient to stop traffic being lost, or do we also need to do something with the controller?
There must be a way to stop traffic from being sent to a controller, so that it can be rebooted in an orderly fashion. If that's not possible, then reducing the period of disruption with network settings would be my second choice. Can someone from the kolla team give advice about this? What is the recommended method for rebooting a kolla-ansible controller in an orderly fashion? Do I need to use the "remove from cluster" and "add to cluster" procedures, or is there a better way? On Wednesday, May 17, 2023, 07:25:34 AM EDT, Eugen Block <eblock@nde.ag> wrote: Hi, I found this [1] reference, it recommends to reduce the kernel option for tcp_retries to reduce the impact of a service interruption: # /etc/kolla/globals.yml haproxy_host_ipv4_tcp_retries2: 6 Apparently, this option was introduced in Victoria [2], it states:
Added a new haproxy configuration variable, haproxy_host_ipv4_tcp_retries2, which allows users to modify this kernel option. This option sets maximum number of times a TCP packet is retransmitted in established state before giving up. The default kernel value is 15, which corresponds to a duration of approximately between 13 to 30 minutes, depending on the retransmission timeout. This variable can be used to mitigate an issue with stuck connections in case of VIP failover, see bug 1917068 for details.
It reads like exactly what you're describing. If I remember correctly, you're still on Train? In that case you'll probably have to configure that setting manually (scripted maybe), it is this value: /proc/sys/net/ipv4/tcp_retries2 The solution in [3] even talks about setting it to 3 for HA deployments. # sysctl -a | grep net.ipv4.tcp_retries2 net.ipv4.tcp_retries2 = 15 Regards, Eugen [1] https://docs.openstack.org/kolla-ansible/latest/reference/high-availability/... [2] https://docs.openstack.org/releasenotes/kolla-ansible/victoria.html [3] https://access.redhat.com/solutions/726753 Zitat von Albert Braden <ozzzo@yahoo.com>:
Before we switched to durable queues we were seeing RMQ issues after a restart. Now RMQ is fine after restart, but operations in progress will fail. VMs will fail to build, or not get DNS records. Volumes don't get attached or detached. It looks like haproxy is the issue now; connections continue going to the down node. I think we can fix that by failing over haproxy before rebooting.
The problem is, I'm not sure that haproxy is the only issue. All 3 controllers are doing stuff, and when I reboot one, whatever it is doing is likely to fail. Is there an orderly way to stop work from being done on a controller without ruining work that is already in progress, besides removing it from the cluster? Would "kolla-ansible stop" do it? On Tuesday, May 16, 2023, 02:23:59 PM EDT, Eugen Block <eblock@nde.ag> wrote:
Hi Albert,
sorry, I'm swamped with different stuff right now. I just took a glance at the docs you mentioned and it seems way too much for something simple as a controller restart to actually remove hosts, that should definitely not be necessary. I'm not familiar with kolla or exabgp, but can you describe what exactly takes that long to failover? Maybe that could be improved? And can you limit the failing requests to a specific service (volumes, network ports, etc.) or do they all fail? Maybe rabbitmq should be considered after all, you could share your rabbitmq settings from the different openstack services and I will collect mine to compare. And then also the rabbitmq config (policies, vhosts, queues).
Regards, Eugen
Zitat von Albert Braden <ozzzo@yahoo.com>:
What's the recommended method for rebooting controllers? Do we need to use the "remove from cluster" and "add to cluster" procedures or is there a better way?
https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-host... On Friday, May 12, 2023, 03:04:26 PM EDT, Albert Braden <ozzzo@yahoo.com> wrote:
We use keepalived and exabgp to manage failover for haproxy. That works but it takes a few minutes, and during those few minutes customers experience impact. We tell them to not build/delete VMs during patching, but they still do, and then complain about the failures.
We're planning to experiment with adding a "manual" haproxy failover to our patching automation, but I'm wondering if there is anything on the controller that needs to be failed over or disabled before rebooting the KVM. I looked at the "remove from cluster" and "add to cluster" procedures but that seems unnecessarily cumbersome for rebooting the KVM. On Friday, May 12, 2023, 03:42:42 AM EDT, Eugen Block <eblock@nde.ag> wrote:
Hi Albert,
how is your haproxy placement controlled, something like pacemaker or similar? I would always do a failover when I'm aware of interruptions (maintenance window), that should speed things up for clients. We have a pacemaker controlled HA control plane, it takes more time until pacemaker realizes that the resource is gone if I just rebooted a server without failing over. I have no benchmarks though. There's always a risk of losing a couple of requests during the failover but we didn't have complaints yet, I believe most of the components try to resend the lost messages. In one of our customer's cluster with many resources (they also use terraform) I haven't seen issues during a regular maintenance window. When they had a DNS outage a few months back it resulted in a mess, manual cleaning was necessary, but the regular failovers seem to work just fine. And I don't see rabbitmq issues either after rebooting a server, usually the haproxy (and virtual IP) failover suffice to prevent interruptions.
Regards, Eugen
Zitat von Satish Patel <satish.txt@gmail.com>:
Are you running your stack on top of the kvm virtual machine? How many controller nodes do you have? mostly rabbitMQ causing issues if you restart controller nodes.
On Thu, May 11, 2023 at 8:34 AM Albert Braden <ozzzo@yahoo.com> wrote:
We have our haproxy and controller nodes on KVM hosts. When those KVM hosts are restarted, customers who are building or deleting VMs see impact. VMs may go into error status, fail to get DNS records, fail to delete, etc. The obvious reason is because traffic that is being routed to the haproxy on the restarting KVM is lost. If we manually fail over haproxy before restarting the KVM, will that be sufficient to stop traffic being lost, or do we also need to do something with the controller?
We have rebooted the controller many times one by one without removing it from the cluster and things survived. We have 200 compute nodes in the cluster and the cluster is quite chatty too. If they are not, that means you have something wrong either in your configuration of the openstack version you are running. In my experience RabbitMQ is the biggest culprit when it comes to reboot controller nodes. This is the two bit I have configure for RabbitMQ # /etc/kolla/config/global.conf [oslo_messaging_rabbit] kombu_reconnect_delay=0.5 rabbit_transient_queues_ttl=60 # /etc/kolla/global.yml om_enable_rabbitmq_high_availability: True I encountered one bug in amqp during ocata release which caused lots of issues but they are all fixed. I highly doubt it's related to HAProxy. On Thu, May 18, 2023 at 8:23 AM Albert Braden <ozzzo@yahoo.com> wrote:
There must be a way to stop traffic from being sent to a controller, so that it can be rebooted in an orderly fashion. If that's not possible, then reducing the period of disruption with network settings would be my second choice.
Can someone from the kolla team give advice about this? What is the recommended method for rebooting a kolla-ansible controller in an orderly fashion? Do I need to use the "remove from cluster" and "add to cluster" procedures, or is there a better way? On Wednesday, May 17, 2023, 07:25:34 AM EDT, Eugen Block <eblock@nde.ag> wrote:
Hi, I found this [1] reference, it recommends to reduce the kernel option for tcp_retries to reduce the impact of a service interruption:
# /etc/kolla/globals.yml haproxy_host_ipv4_tcp_retries2: 6
Apparently, this option was introduced in Victoria [2], it states:
Added a new haproxy configuration variable, haproxy_host_ipv4_tcp_retries2, which allows users to modify this kernel option. This option sets maximum number of times a TCP packet is retransmitted in established state before giving up. The default kernel value is 15, which corresponds to a duration of approximately between 13 to 30 minutes, depending on the retransmission timeout. This variable can be used to mitigate an issue with stuck connections in case of VIP failover, see bug 1917068 for details.
It reads like exactly what you're describing. If I remember correctly, you're still on Train? In that case you'll probably have to configure that setting manually (scripted maybe), it is this value: /proc/sys/net/ipv4/tcp_retries2 The solution in [3] even talks about setting it to 3 for HA deployments.
# sysctl -a | grep net.ipv4.tcp_retries2 net.ipv4.tcp_retries2 = 15
Regards, Eugen
[1]
https://docs.openstack.org/kolla-ansible/latest/reference/high-availability/... [2] https://docs.openstack.org/releasenotes/kolla-ansible/victoria.html [3] https://access.redhat.com/solutions/726753
Zitat von Albert Braden <ozzzo@yahoo.com>:
Before we switched to durable queues we were seeing RMQ issues after a restart. Now RMQ is fine after restart, but operations in progress will fail. VMs will fail to build, or not get DNS records. Volumes don't get attached or detached. It looks like haproxy is the issue now; connections continue going to the down node. I think we can fix that by failing over haproxy before rebooting.
The problem is, I'm not sure that haproxy is the only issue. All 3 controllers are doing stuff, and when I reboot one, whatever it is doing is likely to fail. Is there an orderly way to stop work from being done on a controller without ruining work that is already in progress, besides removing it from the cluster? Would "kolla-ansible stop" do it? On Tuesday, May 16, 2023, 02:23:59 PM EDT, Eugen Block <eblock@nde.ag> wrote:
Hi Albert,
sorry, I'm swamped with different stuff right now. I just took a glance at the docs you mentioned and it seems way too much for something simple as a controller restart to actually remove hosts, that should definitely not be necessary. I'm not familiar with kolla or exabgp, but can you describe what exactly takes that long to failover? Maybe that could be improved? And can you limit the failing requests to a specific service (volumes, network ports, etc.) or do they all fail? Maybe rabbitmq should be considered after all, you could share your rabbitmq settings from the different openstack services and I will collect mine to compare. And then also the rabbitmq config (policies, vhosts, queues).
Regards, Eugen
Zitat von Albert Braden <ozzzo@yahoo.com>:
What's the recommended method for rebooting controllers? Do we need to use the "remove from cluster" and "add to cluster" procedures or is there a better way?
https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-host...
On Friday, May 12, 2023, 03:04:26 PM EDT, Albert Braden <ozzzo@yahoo.com> wrote:
We use keepalived and exabgp to manage failover for haproxy. That works but it takes a few minutes, and during those few minutes customers experience impact. We tell them to not build/delete VMs during patching, but they still do, and then complain about the failures.
We're planning to experiment with adding a "manual" haproxy failover to our patching automation, but I'm wondering if there is anything on the controller that needs to be failed over or disabled before rebooting the KVM. I looked at the "remove from cluster" and "add to cluster" procedures but that seems unnecessarily cumbersome for rebooting the KVM. On Friday, May 12, 2023, 03:42:42 AM EDT, Eugen Block <eblock@nde.ag> wrote:
Hi Albert,
how is your haproxy placement controlled, something like pacemaker or similar? I would always do a failover when I'm aware of interruptions (maintenance window), that should speed things up for clients. We have a pacemaker controlled HA control plane, it takes more time until pacemaker realizes that the resource is gone if I just rebooted a server without failing over. I have no benchmarks though. There's always a risk of losing a couple of requests during the failover but we didn't have complaints yet, I believe most of the components try to resend the lost messages. In one of our customer's cluster with many resources (they also use terraform) I haven't seen issues during a regular maintenance window. When they had a DNS outage a few months back it resulted in a mess, manual cleaning was necessary, but the regular failovers seem to work just fine. And I don't see rabbitmq issues either after rebooting a server, usually the haproxy (and virtual IP) failover suffice to prevent interruptions.
Regards, Eugen
Zitat von Satish Patel <satish.txt@gmail.com>:
Are you running your stack on top of the kvm virtual machine? How many controller nodes do you have? mostly rabbitMQ causing issues if you
restart
controller nodes.
On Thu, May 11, 2023 at 8:34 AM Albert Braden <ozzzo@yahoo.com> wrote:
We have our haproxy and controller nodes on KVM hosts. When those KVM hosts are restarted, customers who are building or deleting VMs see impact. VMs may go into error status, fail to get DNS records, fail to delete, etc. The obvious reason is because traffic that is being routed to the haproxy on the restarting KVM is lost. If we manually fail over haproxy before restarting the KVM, will that be sufficient to stop traffic being lost, or do we also need to do something with the controller?
Controllers and haproxy are on KVM. We have 3 controllers. We used to have RMQ issues during KVM reboot but that stopped after we switched to durable queues. We expected to stop seeing customer impact after fixing the RMQ issues, but we still see it. On Thursday, May 11, 2023, 11:14:17 PM EDT, Satish Patel <satish.txt@gmail.com> wrote: Are you running your stack on top of the kvm virtual machine? How many controller nodes do you have? mostly rabbitMQ causing issues if you restart controller nodes. On Thu, May 11, 2023 at 8:34 AM Albert Braden <ozzzo@yahoo.com> wrote: We have our haproxy and controller nodes on KVM hosts. When those KVM hosts are restarted, customers who are building or deleting VMs see impact. VMs may go into error status, fail to get DNS records, fail to delete, etc. The obvious reason is because traffic that is being routed to the haproxy on the restarting KVM is lost. If we manually fail over haproxy before restarting the KVM, will that be sufficient to stop traffic being lost, or do we also need to do something with the controller?
participants (3)
-
Albert Braden
-
Eugen Block
-
Satish Patel