<div dir="ltr"><div>Hi Mohammed,</div><div><br></div><div>It seems a little bit like our issue. </div><div><br></div><div>Thank you.</div><div><br></div><div>Tiago Pires</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Em sex., 6 de mai. de 2022 às 18:21, Mohammed Naser <<a href="mailto:mnaser@vexxhost.com">mnaser@vexxhost.com</a>> escreveu:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Tiago,<br>
<br>
Have you seen this?<br>
<br>
<a href="https://bugs.launchpad.net/nova/+bug/1969592" rel="noreferrer" target="_blank">https://bugs.launchpad.net/nova/+bug/1969592</a><br>
<br>
Mohammed<br>
<br>
On Fri, May 6, 2022 at 3:56 PM Tiago Pires <<a href="mailto:tiagohp@gmail.com" target="_blank">tiagohp@gmail.com</a>> wrote:<br>
><br>
> Hi all,<br>
><br>
> I was checking the mail list history and this thread <a href="https://mail.openvswitch.org/pipermail/ovs-discuss/2018-March/046438.html" rel="noreferrer" target="_blank">https://mail.openvswitch.org/pipermail/ovs-discuss/2018-March/046438.html</a> caught my attention about raft ovsdb clustering.<br>
> In my setup (OVN 20.03 and Openstack Ussuri) on the ovn-controller we have configured the ovn-remote="tcp:10.2X.4X.4:6642,tcp:10.2X.4X.68:6642,tcp:10.2X.4X.132:6642" with the 3 OVN central member that they are in cluster mode.<br>
> Also on the neutron ML2 side:<br>
> [ovn]<br>
> ovn_native_dhcp = True<br>
> ovn_nb_connection = tcp:10.2X.4X.4:6641,tcp:10.2X.4X.68:6641,tcp:10.2X.4X.132:6641<br>
> ovn_sb_connection = tcp:10.2X.4X.4:6642,tcp:10.2X.4X.68:6642,tcp:10.2X.4X.132:6642<br>
><br>
> We are experiencing an issue with Neutron when the OVN leader decide to take a snapshot and by design another member became leader(more less every 8 minutes):<br>
> 2022-05-05T16:57:42.135Z|17401|raft|INFO|Transferring leadership to write a snapshot.<br>
><br>
> ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound<br>
> 4a03<br>
> Name: OVN_Southbound<br>
> Cluster ID: ca74 (ca744caf-40cd-4751-a2f2-86e35ad6541c)<br>
> Server ID: 4a03 (4a0328dc-e9a4-495e-a4f1-0a0340fc6d19)<br>
> Address: tcp:10.2X.4X.132:6644<br>
> Status: cluster member<br>
> Role: leader<br>
> Term: 1912<br>
> Leader: self<br>
> Vote: self<br>
><br>
> Election timer: 10000<br>
> Log: [497643, 498261]<br>
> Entries not yet committed: 0<br>
> Entries not yet applied: 0<br>
> Connections: ->3d6c ->4ef0 <-3d6c <-4ef0<br>
> Servers:<br>
> 4a03 (4a03 at tcp:10.2X.4X.132:6644) (self) next_index=497874 match_index=498260<br>
> 3d6c (3d6c at tcp:10.2X.4X.68:6644) next_index=498261 match_index=498260<br>
> 4ef0 (4ef0 at tcp:10.2X.4X.4:6644) next_index=498261 match_index=498260<br>
><br>
> As I understood the tcp connections from the Neutron (NB) and ovn-controllers (SB) to OVN Central are established only with the leader:<br>
><br>
> #OVN central leader<br>
> $ netstat -nap | grep 6642| more<br>
><br>
> tcp 0 0 <a href="http://0.0.0.0:6642" rel="noreferrer" target="_blank">0.0.0.0:6642</a> 0.0.0.0:* LISTEN -<br>
> tcp 0 0 10.2X.4X.132:6642 <a href="http://10.24.40.17:47278" rel="noreferrer" target="_blank">10.24.40.17:47278</a> ESTABLISHED -<br>
> tcp 0 0 10.2X.4X.132:6642 <a href="http://10.24.40.76:36240" rel="noreferrer" target="_blank">10.24.40.76:36240</a> ESTABLISHED -<br>
> tcp 0 0 10.2X.4X.132:6642 10.2X.4X.17:47280 ESTABLISHED -<br>
> tcp 0 0 10.2X.4X.132:6642 10.2X.4X.6:43102 ESTABLISHED -<br>
> tcp 0 0 10.2X.4X.132:6642 10.2X.4X.75:58890 ESTABLISHED -<br>
> tcp 0 0 10.2X.4X.132:6642 10.2X.4X.6:43108 ESTABLISHED -<br>
> tcp 0 0 10.2X.4X.132:6642 10.2X.4X.17:47142 ESTABLISHED -<br>
> tcp 0 0 10.2X.4X.132:6642 10.2X.4X.71:48808 ESTABLISHED -<br>
> tcp 0 0 10.2X.4X.132:6642 10.2X.4X.17:47096 ESTABLISHED -<br>
> #OVN follower 2<br>
><br>
> $ netstat -nap | grep 6642<br>
><br>
> tcp 0 0 <a href="http://0.0.0.0:6642" rel="noreferrer" target="_blank">0.0.0.0:6642</a> 0.0.0.0:* LISTEN -<br>
> tcp 0 0 10.2X.4X.4:6642 10.2X.4X.76:57256 ESTABLISHED -<br>
> tcp 0 0 10.2X.4X.4:6642 10.2X.4X.134:54026 ESTABLISHED -<br>
> tcp 0 0 10.2X.4X.4:6642 10.2X.4X.10:34962 ESTABLISHED -<br>
> tcp 0 0 10.2X.4X.4:6642 10.2X.4X.6:49238 ESTABLISHED -<br>
> tcp 0 0 10.2X.4X.4:6642 10.2X.4X.135:59972 ESTABLISHED -<br>
> tcp 0 0 10.2X.4X.4:6642 10.2X.4X.75:40162 ESTABLISHED -<br>
> tcp 0 0 10.2X.4X.4:39566 10.2X.4X.132:6642 ESTABLISHED -<br>
> #OVN follower 3<br>
><br>
> netstat -nap | grep 6642<br>
><br>
> tcp 0 0 <a href="http://0.0.0.0:6642" rel="noreferrer" target="_blank">0.0.0.0:6642</a> 0.0.0.0:* LISTEN -<br>
> tcp 0 0 10.2X.4X.68:6642 10.2X.4X.70:40750 ESTABLISHED -<br>
> tcp 0 0 10.2X.4X.68:6642 10.2X.4X.11:49718 ESTABLISHED -<br>
> tcp 0 0 10.2X.4X.68:45632 10.2X.4X.132:6642 ESTABLISHED -<br>
> tcp 0 0 10.2X.4X.68:6642 10.2X.4X.16:44816 ESTABLISHED -<br>
> tcp 0 0 10.2X.4X.68:6642 10.2X.4X.7:45216 ESTABLISHED<br>
><br>
> The issue that we are experiencing is on the neutron-server that disconnects when there is the ovn leader change (due snapshot like each 8 minutes) and reconnects to the next leader. It breaks the Openstack API when someone is trying to create a VM at the same time.<br>
> First, is my current configuration correct? Should the leader change and break the neutron side? Or is there some missing configuration?<br>
> I was wondering if it is possible to use a LB with VIP and this VIP balance the connections to the ovn central members and I would reconfigure on the neutron side only with the VIP and also on the ovs-controllers. Does that make sense?<br>
><br>
> Thank you.<br>
><br>
> Regards,<br>
><br>
> Tiago Pires<br>
<br>
<br>
<br>
-- <br>
Mohammed Naser<br>
VEXXHOST, Inc.<br>
</blockquote></div></div>