<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
code
{mso-style-priority:99;
font-family:"Courier New";}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Can you verify if the servers where rabbitmq instances are running are reachable from other nodes?<o:p></o:p></span></p>
<pre># rabbit_hosts=rabbit1:5672,rabbit2:5672,rabbit3:5672<o:p></o:p></pre>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Are the above hosts reachable?
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">If yes, check if rabbitmq is listening on correct ports:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">netstat –nap | grep rabbit<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Also, did you set up correct credentials and access for “openstack” user on all rabbitmq hosts?<o:p></o:p></span></p>
<pre># rabbitmqctl add_user openstack <code><i>RABBIT_PASS</i></code><o:p></o:p></pre>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""># rabbitmqctl set_permissions openstack ".*" ".*" ".*"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-Kaustubh<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Yura Poltoratskiy [mailto:yurapoltora@gmail.com]
<br>
<b>Sent:</b> Thursday, March 24, 2016 9:37 AM<br>
<b>To:</b> openstack@lists.openstack.org<br>
<b>Subject:</b> [Openstack] AMQP server is unreachable / AMQP server closed the connection. Check login credentials<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hello,<o:p></o:p></p>
</div>
<p class="MsoNormal">I am a new to OpenStack. I am deploying from scratch and my architecture as such:<o:p></o:p></p>
</div>
<p class="MsoNormal">2 servers - computes<o:p></o:p></p>
</div>
<p class="MsoNormal">3 servers - controllers<o:p></o:p></p>
</div>
<p class="MsoNormal">3 servers - SAN<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">1 server - network<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">On every controller node I have created lxc contrainer for every service, so I have 3 containers for every service, and each container live on bare server.
<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Operation system - Ubuntu 14.04 with cloud-archive:liberty on each server/container and there are no firewall enabled.<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Messages server is RabbitMQ, with I have deployed step-by-steb from this guide:
<a href="http://docs.openstack.org/ha-guide/controller-ha-rabbitmq.html#rabbitmq-services" target="_blank">
http://docs.openstack.org/ha-guide/controller-ha-rabbitmq.html#rabbitmq-services</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">My first try to launch instance was failed. From the log I assume that the problem is near the messages server. Here is some log.<br>
<br>
<br>
nova-compute.log:<br>
<br>
2016-03-24 12:00:06.833 11111 ERROR oslo.messaging._drivers.impl_rabbit [req-8e5b0e51-ed48-48cf-9182-81b437246bcf - - - - -] AM<br>
QP server on rabbitmq01:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.<br>
2016-03-24 12:00:07.848 11111 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on rabbitmq01:5672<br>
<br>
or<br>
<br>
2016-03-24 12:05:23.910 11111 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server rabbitmq01:5672 closed the connection.
<br>
Check login credentials: Socket closed<br>
<br>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">neutron-server.log:<br>
<br>
2016-03-24 12:27:40.140 25737 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on rabbitmq01:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.<br>
2016-03-24 12:27:41.156 25737 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on rabbitmq01:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.<br>
2016-03-24 12:27:42.202 25737 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on rabbitmq02:5672<br>
2016-03-24 12:27:48.786 25737 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer<br>
2016-03-24 12:28:03.580 25737 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on rabbitmq01:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.<br>
2016-03-24 12:28:04.606 25737 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on rabbitmq01:5672<br>
<br>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><a href="mailto:rabbit@rabbitmq01.log">rabbit@rabbitmq01.log</a>:<br>
<br>
=INFO REPORT==== 24-Mar-2016::14:18:01 ===<br>
accepting AMQP connection <0.2831.0> (<a href="http://10.1.1.101:50740" target="_blank">10.1.1.101:50740</a> ->
<a href="http://10.1.1.41:5672" target="_blank">10.1.1.41:5672</a>)<br>
<br>
=INFO REPORT==== 24-Mar-2016::14:18:01 ===<br>
accepting AMQP connection <0.2828.0> (<a href="http://10.1.1.101:50738" target="_blank">10.1.1.101:50738</a> ->
<a href="http://10.1.1.41:5672" target="_blank">10.1.1.41:5672</a>)<br>
<br>
=INFO REPORT==== 24-Mar-2016::14:18:01 ===<br>
accepting AMQP connection <0.2825.0> (<a href="http://10.1.1.101:50736" target="_blank">10.1.1.101:50736</a> ->
<a href="http://10.1.1.41:5672" target="_blank">10.1.1.41:5672</a>)<br>
<br>
=INFO REPORT==== 24-Mar-2016::14:18:01 ===<br>
accepting AMQP connection <0.2834.0> (<a href="http://10.1.1.101:50742" target="_blank">10.1.1.101:50742</a> ->
<a href="http://10.1.1.41:5672" target="_blank">10.1.1.41:5672</a>)<br>
<br>
=INFO REPORT==== 24-Mar-2016::14:18:01 ===<br>
Mirrored queue 'reply_6a400f775e434bafb05db9b8b1976c29' in vhost '/': Adding mirror on node rabbit@rabbitmq02: <7323.17821.0><br>
<br>
=INFO REPORT==== 24-Mar-2016::14:18:01 ===<br>
Mirrored queue 'reply_6a400f775e434bafb05db9b8b1976c29' in vhost '/': Adding mirror on node rabbit@rabbitmq03: <7324.18726.0><br>
<br>
=INFO REPORT==== 24-Mar-2016::14:18:01 ===<br>
Mirrored queue 'compute_fanout_aa803659695f48a2ab997e6bb8bf28bf' in vhost '/': Adding mirror on node rabbit@rabbitmq02: <7323.17827.0><br>
<br>
=INFO REPORT==== 24-Mar-2016::14:18:01 ===<br>
Mirrored queue 'compute_fanout_aa803659695f48a2ab997e6bb8bf28bf' in vhost '/': Adding mirror on node rabbit@rabbitmq03: <7324.18732.0><br>
<br>
=ERROR REPORT==== 24-Mar-2016::14:19:41 ===<br>
closing AMQP connection <0.2825.0> (<a href="http://10.1.1.101:50736" target="_blank">10.1.1.101:50736</a> ->
<a href="http://10.1.1.41:5672" target="_blank">10.1.1.41:5672</a>):<br>
Missed heartbeats from client, timeout: 20s<br>
<br>
=ERROR REPORT==== 24-Mar-2016::14:19:41 ===<br>
closing AMQP connection <0.2834.0> (<a href="http://10.1.1.101:50742" target="_blank">10.1.1.101:50742</a> ->
<a href="http://10.1.1.41:5672" target="_blank">10.1.1.41:5672</a>):<br>
Missed heartbeats from client, timeout: 20s<br>
<br>
=ERROR REPORT==== 24-Mar-2016::14:20:01 ===<br>
closing AMQP connection <0.2828.0> (<a href="http://10.1.1.101:50738" target="_blank">10.1.1.101:50738</a> ->
<a href="http://10.1.1.41:5672" target="_blank">10.1.1.41:5672</a>):<br>
Missed heartbeats from client, timeout: 20s<br>
<br>
=ERROR REPORT==== 24-Mar-2016::14:20:01 ===<br>
closing AMQP connection <0.2831.0> (<a href="http://10.1.1.101:50740" target="_blank">10.1.1.101:50740</a> ->
<a href="http://10.1.1.41:5672" target="_blank">10.1.1.41:5672</a>):<br>
Missed heartbeats from client, timeout: 20s<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">I reinstalled rabbitmq-server on dedicated server without success.
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Someone help me pls to fix the problem.
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I could provide any additional info.<o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>