Hi Koalas, I am not sure but is this normal ? In rabbitmq.yml is this +++ b/ansible/rabbitmq.yml @@ -0,0 +1,115 @@ +--- +# For RabbitMQ we need to be careful about restarting services, to avoid losing quorum. +- name: Apply role rabbitmq + gather_facts: false + hosts: + - rabbitmq + - '&enable_rabbitmq_True' But I can see during the upgrade restart every node at the same moment and then ther is some try to do it better ... I mean is that restart before OK ? Thanks TASK [rabbitmq : Enable all stable feature flags] ***************************************************************************************************************************************************************** ok: [controller1] ok: [controller0] ok: [controller2] TASK [rabbitmq : Check rabbitmq containers] *********************************************************************************************************************************************************************** changed: [controller0] => (item={'key': 'rabbitmq', 'value': {'container_name': 'rabbitmq', 'group': 'rabbitmq', 'enabled': True, 'image': 'dockerhub.ultimum.io/kolla-dev/debian-rabbitmq:KB-2023.1', 'bootstrap_environment': {'KOLLA_BOOTSTRAP': None, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'volumes': ['/etc/kolla/rabbitmq/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'rabbitmq:/var/lib/rabbitmq/', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'healthcheck': {'interval': '30', 'retries': '3', 'start_period': '5', 'test': ['CMD-SHELL', 'healthcheck_rabbitmq'], 'timeout': '30'}, 'haproxy': {'rabbitmq_management': {'enabled': 'yes', 'mode': 'http', 'port': '15672', 'host_group': 'rabbitmq'}, 'rabbitmq_outward_management': {'enabled': False, 'mode': 'http', 'port': '15674', 'host_group': 'outward-rabbitmq'}, 'rabbitmq_outward_external': {'enabled': False, 'mode': 'tcp', 'external': True, 'port': '5674', 'host_group': 'outward-rabbitmq', 'frontend_tcp_extra': ['timeout client 1h'], 'backend_tcp_extra': ['timeout server 1h']}}}}) changed: [controller2] => (item={'key': 'rabbitmq', 'value': {'container_name': 'rabbitmq', 'group': 'rabbitmq', 'enabled': True, 'image': 'dockerhub.ultimum.io/kolla-dev/debian-rabbitmq:KB-2023.1', 'bootstrap_environment': {'KOLLA_BOOTSTRAP': None, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'volumes': ['/etc/kolla/rabbitmq/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'rabbitmq:/var/lib/rabbitmq/', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'healthcheck': {'interval': '30', 'retries': '3', 'start_period': '5', 'test': ['CMD-SHELL', 'healthcheck_rabbitmq'], 'timeout': '30'}, 'haproxy': {'rabbitmq_management': {'enabled': 'yes', 'mode': 'http', 'port': '15672', 'host_group': 'rabbitmq'}, 'rabbitmq_outward_management': {'enabled': False, 'mode': 'http', 'port': '15674', 'host_group': 'outward-rabbitmq'}, 'rabbitmq_outward_external': {'enabled': False, 'mode': 'tcp', 'external': True, 'port': '5674', 'host_group': 'outward-rabbitmq', 'frontend_tcp_extra': ['timeout client 1h'], 'backend_tcp_extra': ['timeout server 1h']}}}}) changed: [controller1] => (item={'key': 'rabbitmq', 'value': {'container_name': 'rabbitmq', 'group': 'rabbitmq', 'enabled': True, 'image': 'dockerhub.ultimum.io/kolla-dev/debian-rabbitmq:KB-2023.1', 'bootstrap_environment': {'KOLLA_BOOTSTRAP': None, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'volumes': ['/etc/kolla/rabbitmq/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'rabbitmq:/var/lib/rabbitmq/', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'healthcheck': {'interval': '30', 'retries': '3', 'start_period': '5', 'test': ['CMD-SHELL', 'healthcheck_rabbitmq'], 'timeout': '30'}, 'haproxy': {'rabbitmq_management': {'enabled': 'yes', 'mode': 'http', 'port': '15672', 'host_group': 'rabbitmq'}, 'rabbitmq_outward_management': {'enabled': False, 'mode': 'http', 'port': '15674', 'host_group': 'outward-rabbitmq'}, 'rabbitmq_outward_external': {'enabled': False, 'mode': 'tcp', 'external': True, 'port': '5674', 'host_group': 'outward-rabbitmq', 'frontend_tcp_extra': ['timeout client 1h'], 'backend_tcp_extra': ['timeout server 1h']}}}}) RUNNING HANDLER [rabbitmq : Restart rabbitmq container] *********************************************************************************************************************************************************** [WARNING]: Failure using method (v2_playbook_on_handler_task_start) in callback plugin (<ansible.plugins.callback.ara_default.CallbackModule object at 0x7f66542e0100>): 'NoneType' object is not subscriptable changed: [controller0] changed: [controller1] changed: [controller2] PLAY [Restart rabbitmq services] ********************************************************************************************************************************************************************************** TASK [rabbitmq : Get info on RabbitMQ container] ****************************************************************************************************************************************************************** ok: [controller0] TASK [rabbitmq : Put RabbitMQ node into maintenance mode] ********************************************************************************************************************************************************* changed: [controller0] TASK [rabbitmq : Restart rabbitmq container] ********************************************************************************************************************************************************************** changed: [controller0] TASK [rabbitmq : Waiting for rabbitmq to start] ******************************************************************************************************************************************************************* changed: [controller0] PLAY [Restart rabbitmq services] ********************************************************************************************************************************************************************************** TASK [rabbitmq : Get info on RabbitMQ container] ****************************************************************************************************************************************************************** ok: [controller1] TASK [rabbitmq : Put RabbitMQ node into maintenance mode] ********************************************************************************************************************************************************* changed: [controller1] TASK [rabbitmq : Restart rabbitmq container] ********************************************************************************************************************************************************************** changed: [controller1] TASK [rabbitmq : Waiting for rabbitmq to start] ******************************************************************************************************************************************************************* changed: [controller1] PLAY [Restart rabbitmq services] ********************************************************************************************************************************************************************************** TASK [rabbitmq : Get info on RabbitMQ container] ****************************************************************************************************************************************************************** ok: [controller2] TASK [rabbitmq : Put RabbitMQ node into maintenance mode] ********************************************************************************************************************************************************* changed: [controller2] TASK [rabbitmq : Restart rabbitmq container] ********************************************************************************************************************************************************************** changed: [controller2] TASK [rabbitmq : Waiting for rabbitmq to start] ******************************************************************************************************************************************************************* changed: [controller2] PLAY [Apply rabbitmq post-configuration] ************************************************************************************************************************************************************************** TASK [Include rabbitmq post-deploy.yml] *************************************************************************************************************************************************************************** TASK [rabbitmq : Enable all stable feature flags] ************************* Michal Arbet Openstack Engineer Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic +420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io <https://ultimum.io/>* LinkedIn <https://www.linkedin.com/company/ultimum-technologies> | Twitter <https://twitter.com/ultimumtech> | Facebook <https://www.facebook.com/ultimumtechnologies/timeline>