<div dir="ltr"><div>Hi,</div><div><br></div><div>We have a 5 node openstack rdo newton setup with one controller node.</div><div><br></div><div>Compute service status is getting fluctuated for all nodes. For few seconds its showing up and then it's getting down in nova-service-list.</div><div><br></div><div>Here is the snap of it.</div><div><br></div><div><div>[root@controller-internal ~]# nova service-list</div><div>+-----+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+</div><div>| Id  | Binary           | Host                | Zone     | Status  | State | Updated_at                 | Disabled Reason |</div><div>+-----+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+</div><div>| 5   | nova-compute     | redhat-compute1     | compute1 | enabled | up    | 2017-11-20T10:43:35.000000 | -               |</div><div>| 12  | nova-conductor   | controller-internal | internal | enabled | up    | 2017-11-20T10:43:51.000000 | -               |</div><div>| 15  | nova-consoleauth | controller-internal | internal | enabled | up    | 2017-11-20T10:43:54.000000 | -               |</div><div>| 16  | nova-cert        | controller-internal | internal | enabled | up    | 2017-11-20T10:43:54.000000 | -               |</div><div>| 17  | nova-scheduler   | controller-internal | internal | enabled | up    | 2017-11-20T10:43:58.000000 | -               |</div><div>| 18  | nova-compute     | redhat-compute2     | compute2 | enabled | down  | 2017-11-20T10:40:47.000000 | -               |</div><div>| 19  | nova-compute     | redhat-compute3     | compute3 | enabled | down  | 2017-11-20T10:41:24.000000 | -               |</div><div>| 20  | nova-compute     | redhat-compute4     | compute4 | enabled | down  | 2017-11-19T17:11:26.000000 | -               |</div><div>| 21  | nova-compute     | redhat-compute5     | compute5 | enabled | down  | 2017-11-20T10:40:19.000000 | -               |</div></div><div>-----------------------------------------------------------------------------</div><div><br></div><div>From rabbitmq logs I can see few reply exchanges are missing and also its showing heartbeat_timeout error.</div><div>----------------------------------------------------------------------------------------------------------</div><div><br></div><div><div style="font-size:12.8px">=ERROR REPORT==== 20-Nov-2017::05:26:24 ===</div><div style="font-size:12.8px">connection <0.4177.0>, channel 1 - soft error:</div><div style="font-size:12.8px">{amqp_error,not_found,</div><div style="font-size:12.8px">            "no exchange 'reply_<wbr>6cd8b245e6b84bc3a80cfc07da243b<wbr>79' in vhost '/'",</div><div style="font-size:12.8px">            'exchange.declare'}</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">=ERROR REPORT==== 20-Nov-2017::05:26:25 ===</div><div style="font-size:12.8px">closing AMQP connection <0.2725.0> (<a href="http://10.0.0.2:59658/" target="_blank">10.0.0.2:59658</a> -> <a href="http://10.0.0.2:5672/" target="_blank">10.0.0.2:5672</a>):</div><div style="font-size:12.8px">{heartbeat_timeout,running}</div><div class="gmail-yj6qo gmail-ajU" style="margin:2px 0px 0px"><div id="gmail-:t1" class="gmail-ajR" tabindex="0" style="font-size:12.8px"><span style="font-size:small">----------------------------------------------------------------------------------------------------------</span></div><div id="gmail-:t1" class="gmail-ajR" tabindex="0" style="font-size:12.8px"><span style="font-size:small"><br></span></div><div id="gmail-:t1" class="gmail-ajR" tabindex="0">I can see this exchange is available in rabbitmq. Don't know how it couldn't connect to it. Telnet to mq port is working fine from compute node. No firewall enabled.</div><div id="gmail-:t1" class="gmail-ajR" tabindex="0"><br></div><div id="gmail-:t1" class="gmail-ajR" tabindex="0"><div id="gmail-:t1" class="gmail-ajR" tabindex="0">[root@controller-internal ~]# rabbitmqctl list_queues |grep reply_6cd8b245e6b84bc3a80cfc07da243b79</div><div id="gmail-:t1" class="gmail-ajR" tabindex="0">reply_6cd8b245e6b84bc3a80cfc07da243b79  0</div><div id="gmail-:t1" class="gmail-ajR" tabindex="0">[root@controller-internal ~]#</div><div><br></div></div><div id="gmail-:t1" class="gmail-ajR" tabindex="0" style="font-size:12.8px">Ant suggestions or troubleshooting steps are welcome.</div><div id="gmail-:t1" class="gmail-ajR" tabindex="0" style="font-size:12.8px"><img class="gmail-ajT" src="https://ssl.gstatic.com/ui/v1/icons/mail/images/cleardot.gif"><br></div></div></div><div><br></div><br clear="all"><div><div class="gmail_signature"><div dir="ltr">Thanks & Regards,<div>Shyam Biradar,</div><div>Email: <a href="mailto:shyambiradarsggsit@gmail.com" target="_blank">shyambiradarsggsit@gmail.com</a>,</div><div>Contact: +91 8600266938.</div></div></div></div>
</div>