Hi Tiago, Have you seen this? https://bugs.launchpad.net/nova/+bug/1969592 Mohammed On Fri, May 6, 2022 at 3:56 PM Tiago Pires <tiagohp@gmail.com> wrote:
Hi all,
I was checking the mail list history and this thread https://mail.openvswitch.org/pipermail/ovs-discuss/2018-March/046438.html caught my attention about raft ovsdb clustering. In my setup (OVN 20.03 and Openstack Ussuri) on the ovn-controller we have configured the ovn-remote="tcp:10.2X.4X.4:6642,tcp:10.2X.4X.68:6642,tcp:10.2X.4X.132:6642" with the 3 OVN central member that they are in cluster mode. Also on the neutron ML2 side: [ovn] ovn_native_dhcp = True ovn_nb_connection = tcp:10.2X.4X.4:6641,tcp:10.2X.4X.68:6641,tcp:10.2X.4X.132:6641 ovn_sb_connection = tcp:10.2X.4X.4:6642,tcp:10.2X.4X.68:6642,tcp:10.2X.4X.132:6642
We are experiencing an issue with Neutron when the OVN leader decide to take a snapshot and by design another member became leader(more less every 8 minutes): 2022-05-05T16:57:42.135Z|17401|raft|INFO|Transferring leadership to write a snapshot.
ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound 4a03 Name: OVN_Southbound Cluster ID: ca74 (ca744caf-40cd-4751-a2f2-86e35ad6541c) Server ID: 4a03 (4a0328dc-e9a4-495e-a4f1-0a0340fc6d19) Address: tcp:10.2X.4X.132:6644 Status: cluster member Role: leader Term: 1912 Leader: self Vote: self
Election timer: 10000 Log: [497643, 498261] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->3d6c ->4ef0 <-3d6c <-4ef0 Servers: 4a03 (4a03 at tcp:10.2X.4X.132:6644) (self) next_index=497874 match_index=498260 3d6c (3d6c at tcp:10.2X.4X.68:6644) next_index=498261 match_index=498260 4ef0 (4ef0 at tcp:10.2X.4X.4:6644) next_index=498261 match_index=498260
As I understood the tcp connections from the Neutron (NB) and ovn-controllers (SB) to OVN Central are established only with the leader:
#OVN central leader $ netstat -nap | grep 6642| more
tcp 0 0 0.0.0.0:6642 0.0.0.0:* LISTEN - tcp 0 0 10.2X.4X.132:6642 10.24.40.17:47278 ESTABLISHED - tcp 0 0 10.2X.4X.132:6642 10.24.40.76:36240 ESTABLISHED - tcp 0 0 10.2X.4X.132:6642 10.2X.4X.17:47280 ESTABLISHED - tcp 0 0 10.2X.4X.132:6642 10.2X.4X.6:43102 ESTABLISHED - tcp 0 0 10.2X.4X.132:6642 10.2X.4X.75:58890 ESTABLISHED - tcp 0 0 10.2X.4X.132:6642 10.2X.4X.6:43108 ESTABLISHED - tcp 0 0 10.2X.4X.132:6642 10.2X.4X.17:47142 ESTABLISHED - tcp 0 0 10.2X.4X.132:6642 10.2X.4X.71:48808 ESTABLISHED - tcp 0 0 10.2X.4X.132:6642 10.2X.4X.17:47096 ESTABLISHED - #OVN follower 2
$ netstat -nap | grep 6642
tcp 0 0 0.0.0.0:6642 0.0.0.0:* LISTEN - tcp 0 0 10.2X.4X.4:6642 10.2X.4X.76:57256 ESTABLISHED - tcp 0 0 10.2X.4X.4:6642 10.2X.4X.134:54026 ESTABLISHED - tcp 0 0 10.2X.4X.4:6642 10.2X.4X.10:34962 ESTABLISHED - tcp 0 0 10.2X.4X.4:6642 10.2X.4X.6:49238 ESTABLISHED - tcp 0 0 10.2X.4X.4:6642 10.2X.4X.135:59972 ESTABLISHED - tcp 0 0 10.2X.4X.4:6642 10.2X.4X.75:40162 ESTABLISHED - tcp 0 0 10.2X.4X.4:39566 10.2X.4X.132:6642 ESTABLISHED - #OVN follower 3
netstat -nap | grep 6642
tcp 0 0 0.0.0.0:6642 0.0.0.0:* LISTEN - tcp 0 0 10.2X.4X.68:6642 10.2X.4X.70:40750 ESTABLISHED - tcp 0 0 10.2X.4X.68:6642 10.2X.4X.11:49718 ESTABLISHED - tcp 0 0 10.2X.4X.68:45632 10.2X.4X.132:6642 ESTABLISHED - tcp 0 0 10.2X.4X.68:6642 10.2X.4X.16:44816 ESTABLISHED - tcp 0 0 10.2X.4X.68:6642 10.2X.4X.7:45216 ESTABLISHED
The issue that we are experiencing is on the neutron-server that disconnects when there is the ovn leader change (due snapshot like each 8 minutes) and reconnects to the next leader. It breaks the Openstack API when someone is trying to create a VM at the same time. First, is my current configuration correct? Should the leader change and break the neutron side? Or is there some missing configuration? I was wondering if it is possible to use a LB with VIP and this VIP balance the connections to the ovn central members and I would reconfigure on the neutron side only with the VIP and also on the ovs-controllers. Does that make sense?
Thank you.
Regards,
Tiago Pires
-- Mohammed Naser VEXXHOST, Inc.