<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi Eddie,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This is the process that I use to reset RMQ when it fails. RMQ messages are ephemeral; losing your old RMQ messages doesn’t ruin the cluster.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">On master: <o:p></o:p></p>
<p class="MsoNormal">service rabbitmq-server stop <o:p></o:p></p>
<p class="MsoNormal">ps auxw|grep rabbit <o:p></o:p></p>
<p class="MsoNormal">(kill any rabbit processes) <o:p></o:p></p>
<p class="MsoNormal">rm -rf /var/lib/rabbitmq/mnesia/* <o:p></o:p></p>
<p class="MsoNormal">service rabbitmq-server start <o:p></o:p></p>
<p class="MsoNormal">rabbitmqctl add_user admin <RMQ pwd from transport_url in nova.conf>
<o:p></o:p></p>
<p class="MsoNormal">rabbitmqctl set_user_tags admin administrator <o:p></o:p></p>
<p class="MsoNormal">rabbitmqctl set_permissions -p / admin ".*" ".*" ".*" <o:p></o:p></p>
<p class="MsoNormal">rabbitmqctl add_user openstack <RMQ pwd from transport_url in nova.conf>
<o:p></o:p></p>
<p class="MsoNormal">rabbitmqctl set_permissions -p / openstack ".*" ".*" ".*" <o:p>
</o:p></p>
<p class="MsoNormal">rabbitmqctl set_policy ha-all "" '{"ha-mode":"all"}' <o:p></o:p></p>
<p class="MsoNormal">rabbitmqctl list_policies <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">on slaves: <o:p></o:p></p>
<p class="MsoNormal">rabbitmqctl stop_app <o:p></o:p></p>
<p class="MsoNormal">If RMQ fails to reset on a slave, or fails to start after resetting, then:
<o:p></o:p></p>
<p class="MsoNormal">service rabbitmq-server stop <o:p></o:p></p>
<p class="MsoNormal">ps auxw|grep rabbit <o:p></o:p></p>
<p class="MsoNormal">(kill any rabbit processes) <o:p></o:p></p>
<p class="MsoNormal">rm -rf /var/lib/rabbitmq/mnesia/* <o:p></o:p></p>
<p class="MsoNormal">service rabbitmq-server start <o:p></o:p></p>
<p class="MsoNormal">rabbitmqctl stop_app <o:p></o:p></p>
<p class="MsoNormal">rabbitmqctl reset <o:p></o:p></p>
<p class="MsoNormal">rabbitmqctl start_app <o:p></o:p></p>
<p class="MsoNormal">rabbitmqctl stop_app <o:p></o:p></p>
<p class="MsoNormal">rabbitmqctl join_cluster rabbit@<master> <o:p></o:p></p>
<p class="MsoNormal">rabbitmqctl start_app <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>From:</b> Eddie Yen <missile0407@gmail.com> <br>
<b>Sent:</b> Wednesday, February 5, 2020 3:33 AM<br>
<b>To:</b> openstack-discuss <openstack-discuss@lists.openstack.org><br>
<b>Subject:</b> Re: [kolla] All services stats DOWN after re-launch whole cluster.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Today I tried to recovery RabbitMQ back, but still not useful, even delete everything<o:p></o:p></p>
<div>
<p class="MsoNormal">about data and configs for RabbitMQ then re-deploy (without destroy).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">And I found that the /etc/hosts on every nodes all been flushed, the hostname<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">resolve data created by kolla-ansible are gone. Checked and found that the MAAS<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">just enabled manage_etc_hosts config in /etc/cloud/cloud.cfg.d/ which caused <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">/etc/hosts been reset everytime when boot.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Not sure it was a root cause or not but unfortunately I already reset whole RabbitMQ<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">data, so only I can do is destroy and deploy again. Fortunately this cluster was just<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">beginning so no VM launch, and no do complex setup yet.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I think the issue may solved, although still need a time to investigate. Based on this<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">experience, need to notice about this may going to happen if using MAAS to deploy<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">the OS.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">-Eddie<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Eddie Yen <<a href="mailto:missile0407@gmail.com">missile0407@gmail.com</a>>
<span style="font-family:"MS Gothic"">於</span> 2020<span style="font-family:"MS Gothic"">年</span>2<span style="font-family:"MS Gothic"">月</span>4<span style="font-family:"MS Gothic"">日</span>
<span style="font-family:"MS Gothic"">週二</span> <span style="font-family:"MS Gothic"">
下午</span>9:45<span style="font-family:"MS Gothic"">寫道：</span><o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">Hi Erik,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I'm already checked NIC link and no issue found. Pinging the nodes each other on each interfaces is OK.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">And I'm not check docker logs about rabbitmq sbecause it works normally. I'll check that out later.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">-Eddie<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Erik McCormick <<a href="mailto:emccormick@cirrusseven.com" target="_blank">emccormick@cirrusseven.com</a>>
<span style="font-family:"MS Gothic"">於</span> 2020<span style="font-family:"MS Gothic"">年</span>2<span style="font-family:"MS Gothic"">月</span>4<span style="font-family:"MS Gothic"">日</span>
<span style="font-family:"MS Gothic"">週二</span> <span style="font-family:"MS Gothic"">
下午</span>9:19<span style="font-family:"MS Gothic"">寫道：</span><o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">⁹<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Feb 4, 2020, 7:20 AM Eddie Yen <<a href="mailto:missile0407@gmail.com" target="_blank">missile0407@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">Hi everyone,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We have the Kolla Openstack site, which is 3 HCI (Controller+Compute) + 3 Storage (Ceph OSD)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">site without internet. We did the shutdown few days ago since CNY holidays. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Today we re-launch whole cluster back. First we met the issue that MariaDB containers keep<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">restarting, and we fixed by using mariadb_recovery command.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">After that we check the status of each services, and found that all services shown at<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Admin > System > System Information are DOWN. Strange is no MariaDB, AMQP connection,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">or other error found when check the downed service log.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We tried reboot each servers but the situation still a same. Then we found the RabbitMQ log not<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">updating, the last log still stayed at the date we shutdown. Logged in to RabbitMQ container and<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">type "rabbitmqctl status" shows connection refused, and tried access its web manager from <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><VIP>:15672 on browser just gave us "503 Service unavailable" message. Also no port 5672<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">listening.<o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</blockquote>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Any chance you have a NIC that didn't come up? What is in the log of the container itself? (ie. docker log rabbitmq). </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
</div>
</div>
<div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">I searched this issue on the internet but only few information about this. One of solution is delete<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">some files in mnesia folder, another is remove rabbitmq container and its volume then re-deploy.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">But both are not sure. Does anyone know how to solve it?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Many thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Eddie.<o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">-Erik<o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</body>
</html>