<div dir="ltr"><ul><li>Any chance to revert back switches + server? That would indicate that MTU was the issue.</li><li>Dont ping the iscsi bay, ping between the controllers in Openstack, they are the ones running mariadb/galera.</li><li>Since the icmp packets are small, it might not trigger the MTU issues. Can you try something like "ping -s 8972 -M do -c 4 $mariadb_host_2" from $
mariadb_host_1?</li><li>What is your network setup on the servers? Two ports in a bond? Did you change both physical interface MTU + bond interface itself?</li></ul><div><br></div><div>4 minutes to create a 20GB empty volume seems too long to me. For an actual 20GB image, it's going to depend on the speed of the backing storage tech.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Feb 5, 2022 at 1:51 AM Franck VEDEL <<a href="mailto:franck.vedel@univ-grenoble-alpes.fr">franck.vedel@univ-grenoble-alpes.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div>Thanks for your help.</div><div><br></div><div><br></div><div><blockquote type="cite"><div dir="ltr"><ul><li>What was the starting value for MTU?</li></ul></div></blockquote>1500<br><blockquote type="cite"><div dir="ltr"><ul><li>What was the starting value changed to for MTU?<br></li></ul></div></blockquote>9000<br><blockquote type="cite"><div dir="ltr"><ul><li>Can ping between all your controllers?</li></ul></div></blockquote>yes, all container starts except nova-conductor, nova-scheduler, maraidb</div><div><br></div><div><br><blockquote type="cite"><div dir="ltr"><ul><li>Do you just have two controllers running mariadb?</li></ul></div></blockquote>yes<br><blockquote type="cite"><div dir="ltr"><ul><li>How did you change MTU?</li></ul></div></blockquote><div><br></div>On the 3 servers:<br><pre><code>nmcli connection modify team0-port1 802-3-ethernet.mtu 9000
nmcli connection modify team1-port2 802-3-ethernet.mtu 9000</code></pre><pre><code><pre><code>nmcli connection modify type team0 team.runner lack ethernet.mtu 9000</code></pre><pre>nmcli con down team0</pre><pre><pre>nmcli con down team1</pre><div><br></div></pre></code></pre><div><br></div><blockquote type="cite"><div dir="ltr"><ul><li>Was the change reverted at the network level as well (switches need to be configured higher or at the same MTU value then the servers)</li></ul></div></blockquote><div>I didn’t change Mtu on network (switches) , but ping -s 10.0.5.117 (iscsi bay) was working from serv3.</div></div><div><br></div><div>I changed the value of the mtu because the creation of the volumes takes a lot of time I find (4 minutes for 20G, which is too long for what I want to do, the patience of the students decreases with the years)</div><div><br></div><div>Franck</div>
<div><br><blockquote type="cite"><div>Le 4 févr. 2022 à 23:12, Laurent Dumont <<a href="mailto:laurentfdumont@gmail.com" target="_blank">laurentfdumont@gmail.com</a>> a écrit :</div><br><div><div dir="ltr"><ul><li>What was the starting value for MTU?</li><li>What was the starting value changed to for MTU?<br></li><li>Can ping between all your controllers?</li><li>Do you just have two controllers running mariadb?</li><li>How did you change MTU?</li><li>Was the change reverted at the network level as well (switches need to be configured higher or at the same MTU value then the servers)</li></ul><div><a>4567 seems to be the port for galera (clustering for mariadb)</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 4, 2022 at 11:52 AM Franck VEDEL <<a href="mailto:franck.vedel@univ-grenoble-alpes.fr" target="_blank">franck.vedel@univ-grenoble-alpes.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div>Hello,</div><div>I am in an emergency situation, quite catastrophic situation because I do not know what to do.</div><div><br></div><div>I have an Openstack cluster with 3 servers (serv1, serv2, serv3). He was doing so well…</div><div><br></div><div><br></div><div>A network admin came to me and told me to change an MTU on the cards. I knew it shouldn't be done...I shouldn't have done it.</div><div>I did it.</div><div>Of course, it didn't work as expected. I went back to my starting configuration and there I have a big problem with mariadb which is set up on serv1 and serv2.</div><div><br></div><div>Here are my errors:</div><div><br></div><div><br></div><div><div>2022-02-04 17:40:36 0 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)</div><div><span style="white-space:pre-wrap"> </span> at gcomm/src/pc.cpp:connect():160</div><div>2022-02-04 17:40:36 0 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():209: Failed to open backend connection: -110 (Connection timed out)</div><div>2022-02-04 17:40:36 0 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1475: Failed to open channel 'openstack' at '<a>gcomm://10.0.5.109:4567,10.0.5.110:4567':</a> -110 (Connection timed out)</div><div>2022-02-04 17:40:36 0 [ERROR] WSREP: gcs connect failed: Connection timed out</div><div>2022-02-04 17:40:36 0 [ERROR] WSREP: wsrep::connect(<a>gcomm://10.0.5.109:4567,10.0.5.110:4567</a>) failed: 7</div><div>2022-02-04 17:40:36 0 [ERROR] Aborting</div></div><div><br></div><div><br></div><div><br></div><div><br></div><div>I do not know what to do. My installation is done with kolla-ansible, mariadb docker restarts every 30 seconds.</div><div><br></div><div>Can the "kolla-ansible reconfigure mariadb" command be a solution?</div><div>Could the command "kolla-ansible mariadb recovery" be a solution?</div><div><br></div><div>Thanks in advance if you can help me.</div><div><br></div></div><div><br></div><div><br></div><div>
<div dir="auto" style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div dir="auto" style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div dir="auto" style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div dir="auto" style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div><span style="color:rgb(31,73,125);font-family:Arial,sans-serif;font-size:10pt">Franck</span></div></div></div><div style="margin:0cm 0cm 0.0001pt"><br></div><div style="font-family:Calibri,sans-serif;font-size:11pt;margin:0cm 0cm 0.0001pt"></div></div></div>
</div>
<br></div></blockquote></div>
</div></blockquote></div><br></div></blockquote></div>