<div dir="ltr"><div>Hi all,</div><div><br></div><div>Thanks Terry.</div><div>As I'm using an old OVS (2.13)/OVN(20.3) version due openstack release (Ussuri), would it be possible to upgrade only the OVN/OVS without upgrading the whole openstack? </div><div>I'm only checking options in this case, how are you guys dealing with this in production? </div><div><br></div><div>Regards,</div><div><br></div><div>Tiago Pires</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Em seg., 9 de mai. de 2022 às 10:45, Terry Wilson <<a href="mailto:twilson@redhat.com">twilson@redhat.com</a>> escreveu:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Sorry, I was on PTO. Jakub is right, w/o using python-ovs 2.17, when<br>
ovs breaks the connection for the leadership clients will re-download<br>
the entire content of their registered tables. With 2.17,<br>
monitor-cond-since/update3 support is added to python-ovs and it<br>
should just download the changes since they reconnected. As long as<br>
the client code handles reconnections, this reconnecting should not be<br>
an issue. It is still possible that there is code that doesn't<br>
properly handle reconnections in general, but I'd start with trying<br>
ovs 2.17. The disonnections will always happen, but they shouldn't<br>
break things.<br>
<br>
On Fri, May 6, 2022 at 5:13 PM Tiago Pires <<a href="mailto:tiagohp@gmail.com" target="_blank">tiagohp@gmail.com</a>> wrote:<br>
><br>
> Hi Mohammed,<br>
><br>
> It seems a little bit like our issue.<br>
><br>
> Thank you.<br>
><br>
> Tiago Pires<br>
><br>
> Em sex., 6 de mai. de 2022 às 18:21, Mohammed Naser <<a href="mailto:mnaser@vexxhost.com" target="_blank">mnaser@vexxhost.com</a>> escreveu:<br>
>><br>
>> Hi Tiago,<br>
>><br>
>> Have you seen this?<br>
>><br>
>> <a href="https://bugs.launchpad.net/nova/+bug/1969592" rel="noreferrer" target="_blank">https://bugs.launchpad.net/nova/+bug/1969592</a><br>
>><br>
>> Mohammed<br>
>><br>
>> On Fri, May 6, 2022 at 3:56 PM Tiago Pires <<a href="mailto:tiagohp@gmail.com" target="_blank">tiagohp@gmail.com</a>> wrote:<br>
>> ><br>
>> > Hi all,<br>
>> ><br>
>> > I was checking the mail list history and this thread <a href="https://mail.openvswitch.org/pipermail/ovs-discuss/2018-March/046438.html" rel="noreferrer" target="_blank">https://mail.openvswitch.org/pipermail/ovs-discuss/2018-March/046438.html</a> caught my attention about raft ovsdb clustering.<br>
>> > In my setup (OVN 20.03 and Openstack Ussuri) on the ovn-controller we have configured the ovn-remote="tcp:10.2X.4X.4:6642,tcp:10.2X.4X.68:6642,tcp:10.2X.4X.132:6642" with the 3 OVN central member that they are in cluster mode.<br>
>> > Also on the neutron ML2 side:<br>
>> > [ovn]<br>
>> > ovn_native_dhcp = True<br>
>> > ovn_nb_connection = tcp:10.2X.4X.4:6641,tcp:10.2X.4X.68:6641,tcp:10.2X.4X.132:6641<br>
>> > ovn_sb_connection = tcp:10.2X.4X.4:6642,tcp:10.2X.4X.68:6642,tcp:10.2X.4X.132:6642<br>
>> ><br>
>> > We are experiencing an issue with Neutron when the OVN leader decide to take a snapshot and by design another member became leader(more less every 8 minutes):<br>
>> > 2022-05-05T16:57:42.135Z|17401|raft|INFO|Transferring leadership to write a snapshot.<br>
>> ><br>
>> > ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound<br>
>> > 4a03<br>
>> > Name: OVN_Southbound<br>
>> > Cluster ID: ca74 (ca744caf-40cd-4751-a2f2-86e35ad6541c)<br>
>> > Server ID: 4a03 (4a0328dc-e9a4-495e-a4f1-0a0340fc6d19)<br>
>> > Address: tcp:10.2X.4X.132:6644<br>
>> > Status: cluster member<br>
>> > Role: leader<br>
>> > Term: 1912<br>
>> > Leader: self<br>
>> > Vote: self<br>
>> ><br>
>> > Election timer: 10000<br>
>> > Log: [497643, 498261]<br>
>> > Entries not yet committed: 0<br>
>> > Entries not yet applied: 0<br>
>> > Connections: ->3d6c ->4ef0 <-3d6c <-4ef0<br>
>> > Servers:<br>
>> > 4a03 (4a03 at tcp:10.2X.4X.132:6644) (self) next_index=497874 match_index=498260<br>
>> > 3d6c (3d6c at tcp:10.2X.4X.68:6644) next_index=498261 match_index=498260<br>
>> > 4ef0 (4ef0 at tcp:10.2X.4X.4:6644) next_index=498261 match_index=498260<br>
>> ><br>
>> > As I understood the tcp connections from the Neutron (NB) and ovn-controllers (SB) to OVN Central are established only with the leader:<br>
>> ><br>
>> > #OVN central leader<br>
>> > $ netstat -nap | grep 6642| more<br>
>> ><br>
>> > tcp 0 0 <a href="http://0.0.0.0:6642" rel="noreferrer" target="_blank">0.0.0.0:6642</a> 0.0.0.0:* LISTEN -<br>
>> > tcp 0 0 10.2X.4X.132:6642 <a href="http://10.24.40.17:47278" rel="noreferrer" target="_blank">10.24.40.17:47278</a> ESTABLISHED -<br>
>> > tcp 0 0 10.2X.4X.132:6642 <a href="http://10.24.40.76:36240" rel="noreferrer" target="_blank">10.24.40.76:36240</a> ESTABLISHED -<br>
>> > tcp 0 0 10.2X.4X.132:6642 10.2X.4X.17:47280 ESTABLISHED -<br>
>> > tcp 0 0 10.2X.4X.132:6642 10.2X.4X.6:43102 ESTABLISHED -<br>
>> > tcp 0 0 10.2X.4X.132:6642 10.2X.4X.75:58890 ESTABLISHED -<br>
>> > tcp 0 0 10.2X.4X.132:6642 10.2X.4X.6:43108 ESTABLISHED -<br>
>> > tcp 0 0 10.2X.4X.132:6642 10.2X.4X.17:47142 ESTABLISHED -<br>
>> > tcp 0 0 10.2X.4X.132:6642 10.2X.4X.71:48808 ESTABLISHED -<br>
>> > tcp 0 0 10.2X.4X.132:6642 10.2X.4X.17:47096 ESTABLISHED -<br>
>> > #OVN follower 2<br>
>> ><br>
>> > $ netstat -nap | grep 6642<br>
>> ><br>
>> > tcp 0 0 <a href="http://0.0.0.0:6642" rel="noreferrer" target="_blank">0.0.0.0:6642</a> 0.0.0.0:* LISTEN -<br>
>> > tcp 0 0 10.2X.4X.4:6642 10.2X.4X.76:57256 ESTABLISHED -<br>
>> > tcp 0 0 10.2X.4X.4:6642 10.2X.4X.134:54026 ESTABLISHED -<br>
>> > tcp 0 0 10.2X.4X.4:6642 10.2X.4X.10:34962 ESTABLISHED -<br>
>> > tcp 0 0 10.2X.4X.4:6642 10.2X.4X.6:49238 ESTABLISHED -<br>
>> > tcp 0 0 10.2X.4X.4:6642 10.2X.4X.135:59972 ESTABLISHED -<br>
>> > tcp 0 0 10.2X.4X.4:6642 10.2X.4X.75:40162 ESTABLISHED -<br>
>> > tcp 0 0 10.2X.4X.4:39566 10.2X.4X.132:6642 ESTABLISHED -<br>
>> > #OVN follower 3<br>
>> ><br>
>> > netstat -nap | grep 6642<br>
>> ><br>
>> > tcp 0 0 <a href="http://0.0.0.0:6642" rel="noreferrer" target="_blank">0.0.0.0:6642</a> 0.0.0.0:* LISTEN -<br>
>> > tcp 0 0 10.2X.4X.68:6642 10.2X.4X.70:40750 ESTABLISHED -<br>
>> > tcp 0 0 10.2X.4X.68:6642 10.2X.4X.11:49718 ESTABLISHED -<br>
>> > tcp 0 0 10.2X.4X.68:45632 10.2X.4X.132:6642 ESTABLISHED -<br>
>> > tcp 0 0 10.2X.4X.68:6642 10.2X.4X.16:44816 ESTABLISHED -<br>
>> > tcp 0 0 10.2X.4X.68:6642 10.2X.4X.7:45216 ESTABLISHED<br>
>> ><br>
>> > The issue that we are experiencing is on the neutron-server that disconnects when there is the ovn leader change (due snapshot like each 8 minutes) and reconnects to the next leader. It breaks the Openstack API when someone is trying to create a VM at the same time.<br>
>> > First, is my current configuration correct? Should the leader change and break the neutron side? Or is there some missing configuration?<br>
>> > I was wondering if it is possible to use a LB with VIP and this VIP balance the connections to the ovn central members and I would reconfigure on the neutron side only with the VIP and also on the ovs-controllers. Does that make sense?<br>
>> ><br>
>> > Thank you.<br>
>> ><br>
>> > Regards,<br>
>> ><br>
>> > Tiago Pires<br>
>><br>
>><br>
>><br>
>> --<br>
>> Mohammed Naser<br>
>> VEXXHOST, Inc.<br>
<br>
</blockquote></div></div>