Hi Koalas,

I am not sure but is this normal ? 
In rabbitmq.yml is this 

+++ b/ansible/rabbitmq.yml
@@ -0,0 +1,115 @@
+---
+# For RabbitMQ we need to be careful about restarting services, to avoid losing quorum.
+- name: Apply role rabbitmq
+  gather_facts: false
+  hosts:
+    - rabbitmq
+    - '&enable_rabbitmq_True'


But I can see during the upgrade restart every node at the same moment and then ther is some try to do it better ...

I mean is that restart before OK ? 

Thanks

TASK [rabbitmq : Enable all stable feature flags] *****************************************************************************************************************************************************************
ok: [controller1]
ok: [controller0]
ok: [controller2]

TASK [rabbitmq : Check rabbitmq containers] ***********************************************************************************************************************************************************************
changed: [controller0] => (item={'key': 'rabbitmq', 'value': {'container_name': 'rabbitmq', 'group': 'rabbitmq', 'enabled': True, 'image': 'dockerhub.ultimum.io/kolla-dev/debian-rabbitmq:KB-2023.1', 'bootstrap_environment': {'KOLLA_BOOTSTRAP': None, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'volumes': ['/etc/kolla/rabbitmq/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'rabbitmq:/var/lib/rabbitmq/', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'healthcheck': {'interval': '30', 'retries': '3', 'start_period': '5', 'test': ['CMD-SHELL', 'healthcheck_rabbitmq'], 'timeout': '30'}, 'haproxy': {'rabbitmq_management': {'enabled': 'yes', 'mode': 'http', 'port': '15672', 'host_group': 'rabbitmq'}, 'rabbitmq_outward_management': {'enabled': False, 'mode': 'http', 'port': '15674', 'host_group': 'outward-rabbitmq'}, 'rabbitmq_outward_external': {'enabled': False, 'mode': 'tcp', 'external': True, 'port': '5674', 'host_group': 'outward-rabbitmq', 'frontend_tcp_extra': ['timeout client 1h'], 'backend_tcp_extra': ['timeout server 1h']}}}})
changed: [controller2] => (item={'key': 'rabbitmq', 'value': {'container_name': 'rabbitmq', 'group': 'rabbitmq', 'enabled': True, 'image': 'dockerhub.ultimum.io/kolla-dev/debian-rabbitmq:KB-2023.1', 'bootstrap_environment': {'KOLLA_BOOTSTRAP': None, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'volumes': ['/etc/kolla/rabbitmq/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'rabbitmq:/var/lib/rabbitmq/', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'healthcheck': {'interval': '30', 'retries': '3', 'start_period': '5', 'test': ['CMD-SHELL', 'healthcheck_rabbitmq'], 'timeout': '30'}, 'haproxy': {'rabbitmq_management': {'enabled': 'yes', 'mode': 'http', 'port': '15672', 'host_group': 'rabbitmq'}, 'rabbitmq_outward_management': {'enabled': False, 'mode': 'http', 'port': '15674', 'host_group': 'outward-rabbitmq'}, 'rabbitmq_outward_external': {'enabled': False, 'mode': 'tcp', 'external': True, 'port': '5674', 'host_group': 'outward-rabbitmq', 'frontend_tcp_extra': ['timeout client 1h'], 'backend_tcp_extra': ['timeout server 1h']}}}})
changed: [controller1] => (item={'key': 'rabbitmq', 'value': {'container_name': 'rabbitmq', 'group': 'rabbitmq', 'enabled': True, 'image': 'dockerhub.ultimum.io/kolla-dev/debian-rabbitmq:KB-2023.1', 'bootstrap_environment': {'KOLLA_BOOTSTRAP': None, 'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'RABBITMQ_CLUSTER_COOKIE': 'ultimum', 'RABBITMQ_LOG_DIR': '/var/log/kolla/rabbitmq'}, 'volumes': ['/etc/kolla/rabbitmq/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'rabbitmq:/var/lib/rabbitmq/', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'healthcheck': {'interval': '30', 'retries': '3', 'start_period': '5', 'test': ['CMD-SHELL', 'healthcheck_rabbitmq'], 'timeout': '30'}, 'haproxy': {'rabbitmq_management': {'enabled': 'yes', 'mode': 'http', 'port': '15672', 'host_group': 'rabbitmq'}, 'rabbitmq_outward_management': {'enabled': False, 'mode': 'http', 'port': '15674', 'host_group': 'outward-rabbitmq'}, 'rabbitmq_outward_external': {'enabled': False, 'mode': 'tcp', 'external': True, 'port': '5674', 'host_group': 'outward-rabbitmq', 'frontend_tcp_extra': ['timeout client 1h'], 'backend_tcp_extra': ['timeout server 1h']}}}})

RUNNING HANDLER [rabbitmq : Restart rabbitmq container] ***********************************************************************************************************************************************************
[WARNING]: Failure using method (v2_playbook_on_handler_task_start) in callback plugin (<ansible.plugins.callback.ara_default.CallbackModule object at 0x7f66542e0100>): 'NoneType' object is not subscriptable
changed: [controller0]
changed: [controller1]
changed: [controller2]

PLAY [Restart rabbitmq services] **********************************************************************************************************************************************************************************

TASK [rabbitmq : Get info on RabbitMQ container] ******************************************************************************************************************************************************************
ok: [controller0]

TASK [rabbitmq : Put RabbitMQ node into maintenance mode] *********************************************************************************************************************************************************
changed: [controller0]

TASK [rabbitmq : Restart rabbitmq container] **********************************************************************************************************************************************************************
changed: [controller0]

TASK [rabbitmq : Waiting for rabbitmq to start] *******************************************************************************************************************************************************************
changed: [controller0]

PLAY [Restart rabbitmq services] **********************************************************************************************************************************************************************************

TASK [rabbitmq : Get info on RabbitMQ container] ******************************************************************************************************************************************************************
ok: [controller1]

TASK [rabbitmq : Put RabbitMQ node into maintenance mode] *********************************************************************************************************************************************************
changed: [controller1]

TASK [rabbitmq : Restart rabbitmq container] **********************************************************************************************************************************************************************
changed: [controller1]

TASK [rabbitmq : Waiting for rabbitmq to start] *******************************************************************************************************************************************************************
changed: [controller1]

PLAY [Restart rabbitmq services] **********************************************************************************************************************************************************************************

TASK [rabbitmq : Get info on RabbitMQ container] ******************************************************************************************************************************************************************
ok: [controller2]

TASK [rabbitmq : Put RabbitMQ node into maintenance mode] *********************************************************************************************************************************************************
changed: [controller2]

TASK [rabbitmq : Restart rabbitmq container] **********************************************************************************************************************************************************************
changed: [controller2]

TASK [rabbitmq : Waiting for rabbitmq to start] *******************************************************************************************************************************************************************
changed: [controller2]

PLAY [Apply rabbitmq post-configuration] **************************************************************************************************************************************************************************

TASK [Include rabbitmq post-deploy.yml] ***************************************************************************************************************************************************************************

TASK [rabbitmq : Enable all stable feature flags] *************************
Michal Arbet
Openstack Engineer

Ultimum Technologies a.s.
Na Poříčí 1047/26, 11000 Praha 1
Czech Republic

+420 604 228 897 
michal.arbet@ultimum.io
https://ultimum.io