<div dir="ltr">Salut Franck!<div><br></div><div>Can you share the output of docker exec -it rabbitmq rabbitmqctl cluster_status?</div><div><br></div><div>Can you "nc -v" from one of the compute nodes towards the controller nodes?</div><div><br></div><div>Laurent</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Nov 11, 2022 at 2:47 PM Franck VEDEL <<a href="mailto:franck.vedel@univ-grenoble-alpes.fr">franck.vedel@univ-grenoble-alpes.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;">Thanks for your help Erik.<div>All is fine with NTP.</div><div>Exactly the same result with " docker exec -it rabbitmq rabbitmqctl cluster_status" on the 2 nodes.</div><div><br></div><div>I Will try this: </div><div><blockquote type="cite"><div dir="ltr"><div class="gmail_quote"><div>On both controllers do: </div><div>docker rm rabbitmq</div><div>docker volume rm rabbitmq</div><div><br></div><div>Then kolla-ansible --tags rabbitmq deploy</div></div></div></blockquote></div><div><div dir="ltr"><div class="gmail_quote"><div><br></div><div><br></div></div></div></div><div> <br><div>
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div><span style="color:rgb(31,73,125);font-family:Arial,sans-serif;font-size:10pt">Franck VEDEL</span></div></div></div><div style="margin:0cm 0cm 0.0001pt"><b><font color="#0096ff"><font face="Arial Black, sans-serif" size="2"><span style="font-weight:normal">Dép. Réseaux </span></font><font face="Arial Black">Informatiques</font><font face="Arial Black, sans-serif" size="2"><span style="font-weight:normal"> & Télécoms</span></font></font></b></div><div style="font-family:Calibri,sans-serif;font-size:11pt;margin:0cm 0cm 0.0001pt"><b><span style="font-size:10pt;font-family:"Arial Black",sans-serif"><font color="#005493">IUT1 - Univ GRENOBLE Alpes</font></span></b></div><div style="font-family:Calibri,sans-serif;font-size:11pt;margin:0cm 0cm 0.0001pt"><b><span style="font-size:10pt;font-family:"Arial Black",sans-serif"><font color="#0433ff"><b style="font-family:Calibri,sans-serif;font-size:14.6667px"><span style="font-size:10pt;font-family:"Arial Black",sans-serif"><font color="#0433ff">0476824462</font></span></b></font></span></b></div><div style="font-family:Calibri,sans-serif;font-size:11pt;margin:0cm 0cm 0.0001pt"><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(31,73,125)">Stages, Alternance, Emploi.</span><span style="color:rgb(31,73,125)"><u></u><u></u></span></div><div style="font-family:Calibri,sans-serif;font-size:11pt;margin:0cm 0cm 0.0001pt"><br></div></div></div></div><br>
</div>
<div><br><blockquote type="cite"><div>Le 11 nov. 2022 à 16:59, Erik McCormick <<a href="mailto:emccormick@cirrusseven.com" target="_blank">emccormick@cirrusseven.com</a>> a écrit :</div><br><div><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Nov 11, 2022 at 3:05 AM Franck VEDEL <<a href="mailto:franck.vedel@univ-grenoble-alpes.fr" target="_blank">franck.vedel@univ-grenoble-alpes.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><br></div>Thanks for your help, really.<div>My cluster: 2 controllers nodes, OVS, L3-HA.</div><div>All nodes had to be rebooted </div><div>All is working for example with external networks (so dhcp on external networks). </div><div>There are no dead containers, all seems ok.</div><div><br></div><div>I try to create a new instance on a L3 network. No ERROR in neutron*.log.</div><div>The only error is nova-api.log:</div><div><br></div><div>Example:</div><div><font color="#e32400">2022-11-11 08:45:54.452 42 ERROR oslo.messaging._drivers.impl_rabbit [-] [8b6fd776-f096-4c8a-927e-88225a3adb43] AMQP server on <a href="http://10.0.5.109:5672/" target="_blank">10.0.5.109:5672</a> is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error></font></div><div><br></div><div>But on the first node (10.0.5.109 on the internal network) « netstat -atnp |wc-l » ———>>> 505 connections</div><div><br></div></div></blockquote><div>Sounds to me like Rabbit is broken. This could also be an issue with NTP which I asked about earlier. Did you confirm your systems are all correctly synced to the same time source?</div><div><br></div><div>You can check the status of rabbit on each control node with:</div><div><br></div><div> docker exec -it rabbitmq rabbitmqctl cluster_status</div><div><br></div><div> Output should show the same on both of your controllers. If not, restart your rabbit containers. If they won't come back properly, you could destroy and redeploy just those two containers l</div><div><br></div><div>On both controllers do: </div><div>docker rm rabbitmq</div><div>docker volume rm rabbitmq</div><div><br></div><div>Then kolla-ansible --tags rabbitmq deploy</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div></div><div>So…. if I backup /etc/kolla, my glance images, my configuration files…</div><div>if a do « koll-ansible destroy », is next step « kolla-ansible bootstraps…. » and preaches, and deploy,</div><div>or directly deploy ? </div><div><br></div><div>What’s the difference with cleanup-containers ? </div><div><br></div></div></blockquote><div>You don't need to bootstrap again. That just installs prerequisites which won't get removed from the destroy. Just go right to doing kolla-ansible deploy again. Do remember this will give you a brand new Openstack with nothing preserved from before. </div><div><br></div><div>cleanup-containers alone may leave behind some docker tweaks that Neutron needs. It probably doesn't matter if you're going to just redeploy the same configuration though so go ahead and use that instead.</div><div><br></div><div>-Erik</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div></div><div><br></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div><div><blockquote type="cite">
</blockquote></div><br></div></div></div></blockquote></div></div>
</div></blockquote></div><br></div></div></blockquote></div>