<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi all,<div class=""><br class=""></div><div class="">After trying to upgrade last piece of release Stein to Train which were all “python-oslo*” & “python2-oslo*” packages along with some others,</div><div class="">nova computes status changed to “DOWN”, and from nova-compute.log below logs </div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue";" class=""> ERROR oslo.messaging._drivers.impl_rabbit [req-de95c598-8e72-46ad-aba6-1b26893ccad7 - - - - -] [739dc403-021e-4bba-a31a-3d86f3d7a908] AMQP server on controller5:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: IOError: Server unexpectedly closed connection</div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue";" class="">2020-06-19 17:40:20.681 5518 INFO oslo.messaging._drivers.impl_rabbit [req-de95c598-8e72-46ad-aba6-1b26893ccad7 - - - - -] [739dc403-021e-4bba-a31a-3d86f3d7a908] Reconnected to AMQP server on controller5:5672 via [amqp] client with port 32894</div></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue";" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue";" class="">This happened every two minutes or so. </div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue";" class="">Connection between rabbitmq servers all looks fine and tested on port accessibility.</div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue";" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue";" class="">Logs in rabbitmq got really nasty and after downgrading below packages(downgrade to Stein) all got to normal</div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue";" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue";" class="">#############################</div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue";" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue";" class=""><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">openvswitch.x86_64</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python-openvswitch.x86_64</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python-pycadf-common.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-automaton.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-barbicanclient.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-castellan.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-cliff.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-debtcollector.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-django-horizon.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-etcd3gw.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-eventlet.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-futurist.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-kombu.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-os-brick.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-os-client-config.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-os-ken.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-os-vif.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-os-win.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-osprofiler.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-ovsdbapp.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-pycadf.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-qpid-proton.x86_64</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-simplejson.x86_64</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-stevedore.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-swiftclient.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-taskflow.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-tenacity.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">python2-tooz.noarch</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">qpid-proton-c.x86_64</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">##########################</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">I’m not sure completely but suspecting in “kombu”.</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">Also after rabbitmq got in high load and high ratio of failure “cinder” services got to DOWN too.</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">Any idea will be great, thanks</div></div></body></html>