<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hello,<div class=""><br class=""></div><div class="">I'm having some problems with my OpenStack installation (Icehouse) made following the installation guide provided on the OpenStack website.</div><div class="">I'm using Neutron (<a href="http://docs.openstack.org/icehouse/install-guide/install/yum/content/ch_overview.html#architecture_example-architectures" class="">Figure 1.2</a>) with the following specificities :</div><div class=""> - The Network node and the Controller node are the same node</div><div class=""> - 2 network interfaces (Management and External use the same one)</div><div class=""> - The Controller/Network node has only 1 ethernet port so this port is "VLAN tagged" in order to be connected to both network</div><div class=""><br class=""></div><div class="">The rest of the installation is similar to the one done in the guide.</div><div class=""><br class=""></div><div class="">Everything seems to be working fine except :</div><div class=""> - the nova-compute service of my compute nodes will randomly "crash" with the following log message repeating itself over and over (the service appears active but I'm forced to restart it for the controller to be able to use it):</div><div class=""><blockquote type="cite" class="">nova-compute[10376]: Exception RuntimeError: 'maximum recursion depth exceeded while calling a Python object' in <bound method GreenSocket.__del__ of <eventlet.greenio.GreenSocket object at 0x7f02b47894d0>> ignored</blockquote><br class=""></div><div class=""> - nova commands can be really really slow (not always, appears to be random to me) :</div><div class=""><div class=""></div><blockquote type="cite" class=""> nova --timing list</blockquote><blockquote type="cite" class=""><div class="">+-------------------------------------------------------------------------------+----------------+</div><div class="">| url | seconds |</div><div class="">+-------------------------------------------------------------------------------+----------------+</div><div class="">| POST <a href="http://controller:35357/v2.0/tokens" class="">http://controller:35357/v2.0/tokens</a> | 0.424832105637 |</div><div class="">| GET <a href="http://controller:8774/v2/566965d25fae409fbe1fb589a1066cfb/servers/detail" class="">http://controller:8774/v2/566965d25fae409fbe1fb589a1066cfb/servers/detail</a> | 25.0711970329 |</div><div class="">| Total | 25.4960291386 |</div><div class="">+-------------------------------------------------------------------------------+----------------+</div></blockquote></div><div class=""></div><br class=""><blockquote type="cite" class=""><div class="">nova --timing show demo-instance1</div><div class="">+-------------------------------------------------------------------------------------------------------------+----------------+<br class="">| url | seconds |<br class="">+-------------------------------------------------------------------------------------------------------------+----------------+<br class="">| POST <a href="http://controller:35357/v2.0/tokens" class="">http://controller:35357/v2.0/tokens</a> | 0.344960927963 |<br class="">| GET <a href="http://controller:8774/v2/566965d25fae409fbe1fb589a1066cfb/servers" class="">http://controller:8774/v2/566965d25fae409fbe1fb589a1066cfb/servers</a> | 0.366063117981 |<br class="">| GET <a href="http://controller:8774/v2/566965d25fae409fbe1fb589a1066cfb/servers/5d5f04d8-edfb-4689-86a1-982647fd4e67" class="">http://controller:8774/v2/566965d25fae409fbe1fb589a1066cfb/servers/5d5f04d8-edfb-4689-86a1-982647fd4e67</a> | 1.37827086449 |<br class="">| GET <a href="http://controller:8774/v2/566965d25fae409fbe1fb589a1066cfb/flavors/1" class="">http://controller:8774/v2/566965d25fae409fbe1fb589a1066cfb/flavors/1</a> | 0.7764108181 |<br class="">| GET <a href="http://controller:8774/v2/566965d25fae409fbe1fb589a1066cfb/images/0bfcb0a3-d631-48ba-b158-9379e67fbc9e" class="">http://controller:8774/v2/566965d25fae409fbe1fb589a1066cfb/images/0bfcb0a3-d631-48ba-b158-9379e67fbc9e</a> | 28.1774730682 |<br class="">| Total | 31.0431787968 |<br class="">+-------------------------------------------------------------------------------------------------------------+----------------+</div></blockquote><div class=""><br class=""></div><div class="">These very same commands might take half a second some other times.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Parsing the logs I found this error in almost every service of every node repeating every now and then :</div><div class=""><blockquote type="cite" class="">2015-04-29 16:17:26.707 28642 INFO oslo.messaging._drivers.impl_qpid [-] Connected to AMQP server on controller:5672<br class="">2015-04-29 17:34:23.957 28642 ERROR oslo.messaging._drivers.impl_qpid [-] Failed to consume message from queue: heartbeat timeout<br class="">2015-04-29 17:34:23.957 28642 TRACE oslo.messaging._drivers.impl_qpid Traceback (most recent call last):<br class="">2015-04-29 17:34:23.957 28642 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py", line 546, in ensure<br class="">2015-04-29 17:34:23.957 28642 TRACE oslo.messaging._drivers.impl_qpid return method(*args, **kwargs)<br class="">2015-04-29 17:34:23.957 28642 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py", line 599, in _consume<br class="">2015-04-29 17:34:23.957 28642 TRACE oslo.messaging._drivers.impl_qpid nxt_receiver = self.session.next_receiver(timeout=timeout)<br class="">2015-04-29 17:34:23.957 28642 TRACE oslo.messaging._drivers.impl_qpid File "<string>", line 6, in next_receiver<br class="">2015-04-29 17:34:23.957 28642 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 689, in next_receiver<br class="">2015-04-29 17:34:23.957 28642 TRACE oslo.messaging._drivers.impl_qpid if self._ecwait(lambda: self.incoming, timeout):<br class="">2015-04-29 17:34:23.957 28642 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait<br class="">2015-04-29 17:34:23.957 28642 TRACE oslo.messaging._drivers.impl_qpid result = self._ewait(lambda: self.closed or predicate(), timeout)<br class="">2015-04-29 17:34:23.957 28642 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 595, in _ewait<br class="">2015-04-29 17:34:23.957 28642 TRACE oslo.messaging._drivers.impl_qpid result = self.connection._ewait(lambda: self.error or predicate(), timeout)<br class="">2015-04-29 17:34:23.957 28642 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 234, in _ewait<br class="">2015-04-29 17:34:23.957 28642 TRACE oslo.messaging._drivers.impl_qpid self.check_error()<br class="">2015-04-29 17:34:23.957 28642 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 227, in check_error<br class="">2015-04-29 17:34:23.957 28642 TRACE oslo.messaging._drivers.impl_qpid raise e<br class="">2015-04-29 17:34:23.957 28642 TRACE oslo.messaging._drivers.impl_qpid HeartbeatTimeout: heartbeat timeout<br class="">2015-04-29 17:34:23.957 28642 TRACE oslo.messaging._drivers.impl_qpid <br class="">2015-04-29 17:34:23.978 28642 INFO oslo.messaging._drivers.impl_qpid [-] Connected to AMQP server on controller:5672<br class="">2015-04-29 17:34:47.068 28642 ERROR oslo.messaging._drivers.impl_qpid [-] Failed to publish message to topic 'conductor': heartbeat timeout<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid Traceback (most recent call last):<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py", line 546, in ensure<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid return method(*args, **kwargs)<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py", line 619, in _publisher_send<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid publisher = cls(self.conf, self.session, topic)<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py", line 407, in __init__<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid super(TopicPublisher, self).__init__(conf, session, node_name)<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py", line 337, in __init__<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid self.reconnect(session)<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py", line 341, in reconnect<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid self.sender = session.sender(self.address)<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid File "<string>", line 6, in sender<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 621, in sender<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid sender._ewait(lambda: sender.linked)<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 831, in _ewait<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid result = self.session._ewait(lambda: self.error or predicate(), timeout)<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 595, in _ewait<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid result = self.connection._ewait(lambda: self.error or predicate(), timeout)<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 234, in _ewait<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid self.check_error()<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 227, in check_error<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid raise e<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid HeartbeatTimeout: heartbeat timeout<br class="">2015-04-29 17:34:47.068 28642 TRACE oslo.messaging._drivers.impl_qpid <br class="">2015-04-29 17:34:47.141 28642 INFO oslo.messaging._drivers.impl_qpid [-] Connected to AMQP server on controller:5672</blockquote><br class=""></div><div class="">I don't know if these problems are (all) related but if anyone has any idea how I could solve them I would be really grateful.</div><div class=""><br class=""></div><div class="">Thanks and regards,</div><div class="">Thomas</div>
</body></html>