Open Stack

We have rebooted the controller many times one by one without removing it
from the cluster and things survived. We have 200 compute nodes in the
cluster and the cluster is quite chatty too. If they are not, that means
you have something wrong either in your configuration of the openstack
version you are running. In my experience RabbitMQ is the biggest culprit
when it comes to reboot controller nodes.

This is the two bit I have configure for RabbitMQ

# /etc/kolla/config/global.conf
[oslo_messaging_rabbit]
kombu_reconnect_delay=0.5
rabbit_transient_queues_ttl=60

# /etc/kolla/global.yml
om_enable_rabbitmq_high_availability: True


I encountered one bug in amqp during ocata release which caused lots of
issues but they are all fixed. I highly doubt it's related to HAProxy.

On Thu, May 18, 2023 at 8:23 AM Albert Braden <ozzzo at yahoo.com> wrote:

> There must be a way to stop traffic from being sent to a controller, so
> that it can be rebooted in an orderly fashion. If that's not possible, then
> reducing the period of disruption with network settings would be my second
> choice.
>
> Can someone from the kolla team give advice about this? What is the
> recommended method for rebooting a kolla-ansible controller in an orderly
> fashion? Do I need to use the "remove from cluster" and "add to cluster"
> procedures, or is there a better way?
> On Wednesday, May 17, 2023, 07:25:34 AM EDT, Eugen Block <eblock at nde.ag>
> wrote:
>
>
> Hi,
> I found this [1] reference, it recommends to reduce the kernel option
> for tcp_retries to reduce the impact of a service interruption:
>
> # /etc/kolla/globals.yml
> haproxy_host_ipv4_tcp_retries2: 6
>
> Apparently, this option was introduced in Victoria [2], it states:
>
> > Added a new haproxy configuration variable,
> > haproxy_host_ipv4_tcp_retries2, which allows users to modify this
> > kernel option. This option sets maximum number of times a TCP packet
> > is retransmitted in established state before giving up. The default
> > kernel value is 15, which corresponds to a duration of approximately
> > between 13 to 30 minutes, depending on the retransmission timeout.
> > This variable can be used to mitigate an issue with stuck
> > connections in case of VIP failover, see bug 1917068 for details.
>
> It reads like exactly what you're describing. If I remember correctly,
> you're still on Train? In that case you'll probably have to configure
> that setting manually (scripted maybe), it is this value:
> /proc/sys/net/ipv4/tcp_retries2
> The solution in [3] even talks about setting it to 3 for HA deployments.
>
> # sysctl -a | grep net.ipv4.tcp_retries2
> net.ipv4.tcp_retries2 = 15
>
> Regards,
> Eugen
>
> [1]
>
> https://docs.openstack.org/kolla-ansible/latest/reference/high-availability/haproxy-guide.html
> [2] https://docs.openstack.org/releasenotes/kolla-ansible/victoria.html
> [3] https://access.redhat.com/solutions/726753
>
> Zitat von Albert Braden <ozzzo at yahoo.com>:
>
> > Before we switched to durable queues we were seeing RMQ issues after
> > a restart. Now RMQ is fine after restart, but operations in progress
> > will fail. VMs will fail to build, or not get DNS records. Volumes
> > don't get attached or detached. It looks like haproxy is the issue
> > now; connections continue going to the down node. I think we can fix
> > that by failing over haproxy before rebooting.
> >
> > The problem is, I'm not sure that haproxy is the only issue. All 3
> > controllers are doing stuff, and when I reboot one, whatever it is
> > doing is likely to fail. Is there an orderly way to stop work from
> > being done on a controller without ruining work that is already in
> > progress, besides removing it from the cluster? Would "kolla-ansible
> > stop" do it?
> >      On Tuesday, May 16, 2023, 02:23:59 PM EDT, Eugen Block
> > <eblock at nde.ag> wrote:
> >
> >  Hi Albert,
> >
> > sorry, I'm swamped with different stuff right now. I just took a
> > glance at the docs you mentioned and it seems way too much for
> > something simple as a controller restart to actually remove hosts,
> > that should definitely not be necessary.
> > I'm not familiar with kolla or exabgp, but can you describe what
> > exactly takes that long to failover? Maybe that could be improved? And
> > can you limit the failing requests to a specific service (volumes,
> > network ports, etc.) or do they all fail? Maybe rabbitmq should be
> > considered after all, you could share your rabbitmq settings from the
> > different openstack services and I will collect mine to compare. And
> > then also the rabbitmq config (policies, vhosts, queues).
> >
> > Regards,
> > Eugen
> >
> > Zitat von Albert Braden <ozzzo at yahoo.com>:
> >
> >> What's the recommended method for rebooting controllers? Do we need
> >> to use the "remove from cluster" and "add to cluster" procedures or
> >> is there a better way?
> >>
> >>
> https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hosts.html
> >>       On Friday, May 12, 2023, 03:04:26 PM EDT, Albert Braden
> >> <ozzzo at yahoo.com> wrote:
> >>
> >>   We use keepalived and exabgp to manage failover for haproxy. That
> >> works but it takes a few minutes, and during those few minutes
> >> customers experience impact. We tell them to not build/delete VMs
> >> during patching, but they still do, and then complain about the
> >> failures.
> >>
> >> We're planning to experiment with adding a "manual" haproxy failover
> >> to our patching automation, but I'm wondering if there is anything
> >> on the controller that needs to be failed over or disabled before
> >> rebooting the KVM. I looked at the "remove from cluster" and "add to
> >> cluster" procedures but that seems unnecessarily cumbersome for
> >> rebooting the KVM.
> >>       On Friday, May 12, 2023, 03:42:42 AM EDT, Eugen Block
> >> <eblock at nde.ag> wrote:
> >>
> >>   Hi Albert,
> >>
> >> how is your haproxy placement controlled, something like pacemaker or
> >> similar? I would always do a failover when I'm aware of interruptions
> >> (maintenance window), that should speed things up for clients. We have
> >> a pacemaker controlled HA control plane, it takes more time until
> >> pacemaker realizes that the resource is gone if I just rebooted a
> >> server without failing over. I have no benchmarks though. There's
> >> always a risk of losing a couple of requests during the failover but
> >> we didn't have complaints yet, I believe most of the components try to
> >> resend the lost messages. In one of our customer's cluster with many
> >> resources (they also use terraform) I haven't seen issues during a
> >> regular maintenance window. When they had a DNS outage a few months
> >> back it resulted in a mess, manual cleaning was necessary, but the
> >> regular failovers seem to work just fine.
> >> And I don't see rabbitmq issues either after rebooting a server,
> >> usually the haproxy (and virtual IP) failover suffice to prevent
> >> interruptions.
> >>
> >> Regards,
> >> Eugen
> >>
> >> Zitat von Satish Patel <satish.txt at gmail.com>:
> >>
> >>> Are you running your stack on top of the kvm virtual machine? How many
> >>> controller nodes do you have? mostly rabbitMQ causing issues if you
> restart
> >>> controller nodes.
> >>>
> >>> On Thu, May 11, 2023 at 8:34 AM Albert Braden <ozzzo at yahoo.com> wrote:
> >>>
> >>>> We have our haproxy and controller nodes on KVM hosts. When those KVM
> >>>> hosts are restarted, customers who are building or deleting VMs
> >>>> see impact.
> >>>> VMs may go into error status, fail to get DNS records, fail to
> >>>> delete, etc.
> >>>> The obvious reason is because traffic that is being routed to the
> haproxy
> >>>> on the restarting KVM is lost. If we manually fail over haproxy before
> >>>> restarting the KVM, will that be sufficient to stop traffic being
> lost, or
> >>>> do we also need to do something with the controller?
> >>>>
> >>>>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230518/21d3b162/attachment-0001.htm>