[nova][oslo][rabbitmq][compute]
Hi all, After trying to upgrade last piece of release Stein to Train which were all “python-oslo*” & “python2-oslo*” packages along with some others, nova computes status changed to “DOWN”, and from nova-compute.log below logs ERROR oslo.messaging._drivers.impl_rabbit [req-de95c598-8e72-46ad-aba6-1b26893ccad7 - - - - -] [739dc403-021e-4bba-a31a-3d86f3d7a908] AMQP server on controller5:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: IOError: Server unexpectedly closed connection 2020-06-19 17:40:20.681 5518 INFO oslo.messaging._drivers.impl_rabbit [req-de95c598-8e72-46ad-aba6-1b26893ccad7 - - - - -] [739dc403-021e-4bba-a31a-3d86f3d7a908] Reconnected to AMQP server on controller5:5672 via [amqp] client with port 32894 This happened every two minutes or so. Connection between rabbitmq servers all looks fine and tested on port accessibility. Logs in rabbitmq got really nasty and after downgrading below packages(downgrade to Stein) all got to normal ############################# openvswitch.x86_64 python-openvswitch.x86_64 python-pycadf-common.noarch python2-automaton.noarch python2-barbicanclient.noarch python2-castellan.noarch python2-cliff.noarch python2-debtcollector.noarch python2-django-horizon.noarch python2-etcd3gw.noarch python2-eventlet.noarch python2-futurist.noarch python2-kombu.noarch python2-os-brick.noarch python2-os-client-config.noarch python2-os-ken.noarch python2-os-vif.noarch python2-os-win.noarch python2-osprofiler.noarch python2-ovsdbapp.noarch python2-pycadf.noarch python2-qpid-proton.x86_64 python2-simplejson.x86_64 python2-stevedore.noarch python2-swiftclient.noarch python2-taskflow.noarch python2-tenacity.noarch python2-tooz.noarch qpid-proton-c.x86_64 ########################## I’m not sure completely but suspecting in “kombu”. Also after rabbitmq got in high load and high ratio of failure “cinder” services got to DOWN too. Any idea will be great, thanks
participants (1)
-
Amjad Kotobi