[kolla]RabbitMQ HA policies define
Folks, I have deployed 3 node kolla controllers and notice rabbitMQ doesn't have any HA policy for queues. when i rebooted one of the nodes and found some messages got lost. Any reason kolla doesn't use HA policies by default? If i want to add policies then where should i specify them or any doc you can point me to define policies.
you will need add "{"vhost": "/", "name": "ha-all", "pattern": "^(?!(amq\.)|(.*_fanout_)|(reply_)).*", "apply-to": "all", "definition": {"ha-mode":"all"}, "priority":0}{% if project_name == 'outward_rabbitmq' %}" to kolla-ansible/ansible/roles/rabbitmq/templates/definitions.json.j2 Nguyen Huu Khoi On Fri, Feb 10, 2023 at 10:47 AM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I have deployed 3 node kolla controllers and notice rabbitMQ doesn't have any HA policy for queues. when i rebooted one of the nodes and found some messages got lost.
Any reason kolla doesn't use HA policies by default?
If i want to add policies then where should i specify them or any doc you can point me to define policies.
Thank you!! Does this change survive the upgrade of kolla? Why is this change not part of /etc/kolla/config/ style? why don't we have this change default in kolla because HA is always good. On Fri, Feb 10, 2023 at 12:01 AM Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
you will need add
"{"vhost": "/", "name": "ha-all", "pattern": "^(?!(amq\.)|(.*_fanout_)|(reply_)).*", "apply-to": "all", "definition": {"ha-mode":"all"}, "priority":0}{% if project_name == 'outward_rabbitmq' %}" to
kolla-ansible/ansible/roles/rabbitmq/templates/definitions.json.j2
Nguyen Huu Khoi
On Fri, Feb 10, 2023 at 10:47 AM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I have deployed 3 node kolla controllers and notice rabbitMQ doesn't have any HA policy for queues. when i rebooted one of the nodes and found some messages got lost.
Any reason kolla doesn't use HA policies by default?
If i want to add policies then where should i specify them or any doc you can point me to define policies.
Hi Nguyễn, I have tried your config but getting syntax errors, what version of rabbit are you running? i have created file and added your config: /etc/rabbitmq/rabbitmq_definitions.jso Getting the following error and I am running Ubuntu 22.04 and Zed release TASK [rabbitmq : Copying over definitions.json] ********************************************************************************************************************************** An exception occurred during task execution. To see the full traceback, use -vvv. The error was: } failed: [ctrl1] (item=/etc/kolla/config/rabbitmq/definitions.json) => {"ansible_loop_var": "item", "changed": false, "item": "/etc/kolla/config/rabbitmq/definitions.json", "msg": "AnsibleError: template error while templating string: Unexpected end of template. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'. The innermost block that needs to be closed is 'if'.. String: {\n \"vhosts\": [\n {\"name\": \"/\"}{% if project_name == 'outward_rabbitmq' %},\n {\"name\": \"{{ murano_agent_rabbitmq_vhost }}\"}\n {% endif %}\n ],\n \"users\": [\n {\"name\": \"{{ role_rabbitmq_user }}\", \"password\": \"{{ role_rabbitmq_password }}\", \"tags\": \"administrator\"}{% if role_rabbitmq_monitoring_user is defined and role_rabbitmq_monitoring_user %},\n {\"name\": \"{{ role_rabbitmq_monitoring_user }}\", \"password\": \"{{ role_rabbitmq_monitoring_password }}\", \"tags\": \"monitoring\"}{% endif %}{% if project_name == 'outward_rabbitmq' %},\n {\"name\": \"{{ murano_agent_rabbitmq_user }}\", \"password\": \"{{ murano_agent_rabbitmq_password }}\", \"tags\": \"management\"}\n {% endif %}\n ],\n \"permissions\": [\n {\"user\": \"{{ role_rabbitmq_user }}\", \"vhost\": \"/\", \"configure\": \".*\", \"write\": \".*\", \"read\": \".*\"}{% if role_rabbitmq_monitoring_user is defined and role_rabbitmq_monitoring_user %},\n {\"user\": \"{{ role_rabbitmq_monitoring_user }}\", \"vhost\": \"/\", \"configure\": \"^$\", \"write\": \"^$\", \"read\": \".*\"}{% endif %}{% if project_name == 'outward_rabbitmq' %},\n {\"user\": \"{{ murano_agent_rabbitmq_user }}\", \"vhost\": \"{{ murano_agent_rabbitmq_vhost }}\", \"configure\": \".*\", \"write\": \".*\", \"read\": \".*\"}\n {% endif %}\n ],\n \"policies\":[\"{\"vhost\": \"/\", \"name\": \"ha-all\", \"pattern\": \"^(?!(amq\\.)|(.*_fanout_)|(reply_)).*\", \"apply-to\": \"all\", \"definition\": {\"ha-mode\":\"all\"}, \"priority\":0}{% if project_name == 'outward_rabbitmq' %}\"]\n}\n"} An exception occurred during task execution. To see the full traceback, use -vvv. The error was: } failed: [ctrl2] (item=/etc/kolla/config/rabbitmq/definitions.json) => {"ansible_loop_var": "item", "changed": false, "item": "/etc/kolla/config/rabbitmq/definitions.json", "msg": "AnsibleError: template error while templating string: Unexpected end of template. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'. The innermost block that needs to be closed is 'if'.. String: {\n \"vhosts\": [\n {\"name\": \"/\"}{% if project_name == 'outward_rabbitmq' %},\n {\"name\": \"{{ murano_agent_rabbitmq_vhost }}\"}\n {% endif %}\n ],\n \"users\": [\n {\"name\": \"{{ role_rabbitmq_user }}\", \"password\": \"{{ role_rabbitmq_password }}\", \"tags\": \"administrator\"}{% if role_rabbitmq_monitoring_user is defined and role_rabbitmq_monitoring_user %},\n {\"name\": \"{{ role_rabbitmq_monitoring_user }}\", \"password\": \"{{ role_rabbitmq_monitoring_password }}\", \"tags\": \"monitoring\"}{% endif %}{% if project_name == 'outward_rabbitmq' %},\n {\"name\": \"{{ murano_agent_rabbitmq_user }}\", \"password\": \"{{ murano_agent_rabbitmq_password }}\", \"tags\": \"management\"}\n {% endif %}\n ],\n \"permissions\": [\n {\"user\": \"{{ role_rabbitmq_user }}\", \"vhost\": \"/\", \"configure\": \".*\", \"write\": \".*\", \"read\": \".*\"}{% if role_rabbitmq_monitoring_user is defined and role_rabbitmq_monitoring_user %},\n {\"user\": \"{{ role_rabbitmq_monitoring_user }}\", \"vhost\": \"/\", \"configure\": \"^$\", \"write\": \"^$\", \"read\": \".*\"}{% endif %}{% if project_name == 'outward_rabbitmq' %},\n {\"user\": \"{{ murano_agent_rabbitmq_user }}\", \"vhost\": \"{{ murano_agent_rabbitmq_vhost }}\", \"configure\": \".*\", \"write\": \".*\", \"read\": \".*\"}\n {% endif %}\n ],\n \"policies\":[\"{\"vhost\": \"/\", \"name\": \"ha-all\", \"pattern\": \"^(?!(amq\\.)|(.*_fanout_)|(reply_)).*\", \"apply-to\": \"all\", \"definition\": {\"ha-mode\":\"all\"}, \"priority\":0}{% if project_name == 'outward_rabbitmq' %}\"]\n}\n"} An exception occurred during task execution. To see the full traceback, use -vvv. The error was: } failed: [ctrl3] (item=/etc/kolla/config/rabbitmq/definitions.json) => {"ansible_loop_var": "item", "changed": false, "item": "/etc/kolla/config/rabbitmq/definitions.json", "msg": "AnsibleError: template error while templating string: Unexpected end of template. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'. The innermost block that needs to be closed is 'if'.. String: {\n \"vhosts\": [\n {\"name\": \"/\"}{% if project_name == 'outward_rabbitmq' %},\n {\"name\": \"{{ murano_agent_rabbitmq_vhost }}\"}\n {% endif %}\n ],\n \"users\": [\n {\"name\": \"{{ role_rabbitmq_user }}\", \"password\": \"{{ role_rabbitmq_password }}\", \"tags\": \"administrator\"}{% if role_rabbitmq_monitoring_user is defined and role_rabbitmq_monitoring_user %},\n {\"name\": \"{{ role_rabbitmq_monitoring_user }}\", \"password\": \"{{ role_rabbitmq_monitoring_password }}\", \"tags\": \"monitoring\"}{% endif %}{% if project_name == 'outward_rabbitmq' %},\n {\"name\": \"{{ murano_agent_rabbitmq_user }}\", \"password\": \"{{ murano_agent_rabbitmq_password }}\", \"tags\": \"management\"}\n {% endif %}\n ],\n \"permissions\": [\n {\"user\": \"{{ role_rabbitmq_user }}\", \"vhost\": \"/\", \"configure\": \".*\", \"write\": \".*\", \"read\": \".*\"}{% if role_rabbitmq_monitoring_user is defined and role_rabbitmq_monitoring_user %},\n {\"user\": \"{{ role_rabbitmq_monitoring_user }}\", \"vhost\": \"/\", \"configure\": \"^$\", \"write\": \"^$\", \"read\": \".*\"}{% endif %}{% if project_name == 'outward_rabbitmq' %},\n {\"user\": \"{{ murano_agent_rabbitmq_user }}\", \"vhost\": \"{{ murano_agent_rabbitmq_vhost }}\", \"configure\": \".*\", \"write\": \".*\", \"read\": \".*\"}\n {% endif %}\n ],\n \"policies\":[\"{\"vhost\": \"/\", \"name\": \"ha-all\", \"pattern\": \"^(?!(amq\\.)|(.*_fanout_)|(reply_)).*\", \"apply-to\": \"all\", \"definition\": {\"ha-mode\":\"all\"}, \"priority\":0}{% if project_name == 'outward_rabbitmq' %}\"]\n}\n"} PLAY RECAP *********************************************************************************************************************************************************************** ctrl1 : ok=12 changed=0 unreachable=0 failed=1 skipped=3 rescued=0 ignored=0 ctrl2 : ok=12 changed=0 unreachable=0 failed=1 skipped=3 rescued=0 ignored=0 ctrl3 : ok=11 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 localhost : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 On Fri, Feb 10, 2023 at 12:01 AM Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
you will need add
"{"vhost": "/", "name": "ha-all", "pattern": "^(?!(amq\.)|(.*_fanout_)|(reply_)).*", "apply-to": "all", "definition": {"ha-mode":"all"}, "priority":0}{% if project_name == 'outward_rabbitmq' %}" to
kolla-ansible/ansible/roles/rabbitmq/templates/definitions.json.j2
Nguyen Huu Khoi
On Fri, Feb 10, 2023 at 10:47 AM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I have deployed 3 node kolla controllers and notice rabbitMQ doesn't have any HA policy for queues. when i rebooted one of the nodes and found some messages got lost.
Any reason kolla doesn't use HA policies by default?
If i want to add policies then where should i specify them or any doc you can point me to define policies.
This is my config file /etc/kolla/config/rabbitmq/definitions.json https://paste.opendev.org/show/b5V1jGCqM3QAs54bxcTC/ On Sat, Feb 11, 2023 at 12:10 PM Satish Patel <satish.txt@gmail.com> wrote:
Hi Nguyễn,
I have tried your config but getting syntax errors, what version of rabbit are you running? i have created file and added your config: /etc/rabbitmq/rabbitmq_definitions.jso
Getting the following error and I am running Ubuntu 22.04 and Zed release
TASK [rabbitmq : Copying over definitions.json] ********************************************************************************************************************************** An exception occurred during task execution. To see the full traceback, use -vvv. The error was: } failed: [ctrl1] (item=/etc/kolla/config/rabbitmq/definitions.json) => {"ansible_loop_var": "item", "changed": false, "item": "/etc/kolla/config/rabbitmq/definitions.json", "msg": "AnsibleError: template error while templating string: Unexpected end of template. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'. The innermost block that needs to be closed is 'if'.. String: {\n \"vhosts\": [\n {\"name\": \"/\"}{% if project_name == 'outward_rabbitmq' %},\n {\"name\": \"{{ murano_agent_rabbitmq_vhost }}\"}\n {% endif %}\n ],\n \"users\": [\n {\"name\": \"{{ role_rabbitmq_user }}\", \"password\": \"{{ role_rabbitmq_password }}\", \"tags\": \"administrator\"}{% if role_rabbitmq_monitoring_user is defined and role_rabbitmq_monitoring_user %},\n {\"name\": \"{{ role_rabbitmq_monitoring_user }}\", \"password\": \"{{ role_rabbitmq_monitoring_password }}\", \"tags\": \"monitoring\"}{% endif %}{% if project_name == 'outward_rabbitmq' %},\n {\"name\": \"{{ murano_agent_rabbitmq_user }}\", \"password\": \"{{ murano_agent_rabbitmq_password }}\", \"tags\": \"management\"}\n {% endif %}\n ],\n \"permissions\": [\n {\"user\": \"{{ role_rabbitmq_user }}\", \"vhost\": \"/\", \"configure\": \".*\", \"write\": \".*\", \"read\": \".*\"}{% if role_rabbitmq_monitoring_user is defined and role_rabbitmq_monitoring_user %},\n {\"user\": \"{{ role_rabbitmq_monitoring_user }}\", \"vhost\": \"/\", \"configure\": \"^$\", \"write\": \"^$\", \"read\": \".*\"}{% endif %}{% if project_name == 'outward_rabbitmq' %},\n {\"user\": \"{{ murano_agent_rabbitmq_user }}\", \"vhost\": \"{{ murano_agent_rabbitmq_vhost }}\", \"configure\": \".*\", \"write\": \".*\", \"read\": \".*\"}\n {% endif %}\n ],\n \"policies\":[\"{\"vhost\": \"/\", \"name\": \"ha-all\", \"pattern\": \"^(?!(amq\\.)|(.*_fanout_)|(reply_)).*\", \"apply-to\": \"all\", \"definition\": {\"ha-mode\":\"all\"}, \"priority\":0}{% if project_name == 'outward_rabbitmq' %}\"]\n}\n"} An exception occurred during task execution. To see the full traceback, use -vvv. The error was: } failed: [ctrl2] (item=/etc/kolla/config/rabbitmq/definitions.json) => {"ansible_loop_var": "item", "changed": false, "item": "/etc/kolla/config/rabbitmq/definitions.json", "msg": "AnsibleError: template error while templating string: Unexpected end of template. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'. The innermost block that needs to be closed is 'if'.. String: {\n \"vhosts\": [\n {\"name\": \"/\"}{% if project_name == 'outward_rabbitmq' %},\n {\"name\": \"{{ murano_agent_rabbitmq_vhost }}\"}\n {% endif %}\n ],\n \"users\": [\n {\"name\": \"{{ role_rabbitmq_user }}\", \"password\": \"{{ role_rabbitmq_password }}\", \"tags\": \"administrator\"}{% if role_rabbitmq_monitoring_user is defined and role_rabbitmq_monitoring_user %},\n {\"name\": \"{{ role_rabbitmq_monitoring_user }}\", \"password\": \"{{ role_rabbitmq_monitoring_password }}\", \"tags\": \"monitoring\"}{% endif %}{% if project_name == 'outward_rabbitmq' %},\n {\"name\": \"{{ murano_agent_rabbitmq_user }}\", \"password\": \"{{ murano_agent_rabbitmq_password }}\", \"tags\": \"management\"}\n {% endif %}\n ],\n \"permissions\": [\n {\"user\": \"{{ role_rabbitmq_user }}\", \"vhost\": \"/\", \"configure\": \".*\", \"write\": \".*\", \"read\": \".*\"}{% if role_rabbitmq_monitoring_user is defined and role_rabbitmq_monitoring_user %},\n {\"user\": \"{{ role_rabbitmq_monitoring_user }}\", \"vhost\": \"/\", \"configure\": \"^$\", \"write\": \"^$\", \"read\": \".*\"}{% endif %}{% if project_name == 'outward_rabbitmq' %},\n {\"user\": \"{{ murano_agent_rabbitmq_user }}\", \"vhost\": \"{{ murano_agent_rabbitmq_vhost }}\", \"configure\": \".*\", \"write\": \".*\", \"read\": \".*\"}\n {% endif %}\n ],\n \"policies\":[\"{\"vhost\": \"/\", \"name\": \"ha-all\", \"pattern\": \"^(?!(amq\\.)|(.*_fanout_)|(reply_)).*\", \"apply-to\": \"all\", \"definition\": {\"ha-mode\":\"all\"}, \"priority\":0}{% if project_name == 'outward_rabbitmq' %}\"]\n}\n"} An exception occurred during task execution. To see the full traceback, use -vvv. The error was: } failed: [ctrl3] (item=/etc/kolla/config/rabbitmq/definitions.json) => {"ansible_loop_var": "item", "changed": false, "item": "/etc/kolla/config/rabbitmq/definitions.json", "msg": "AnsibleError: template error while templating string: Unexpected end of template. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'. The innermost block that needs to be closed is 'if'.. String: {\n \"vhosts\": [\n {\"name\": \"/\"}{% if project_name == 'outward_rabbitmq' %},\n {\"name\": \"{{ murano_agent_rabbitmq_vhost }}\"}\n {% endif %}\n ],\n \"users\": [\n {\"name\": \"{{ role_rabbitmq_user }}\", \"password\": \"{{ role_rabbitmq_password }}\", \"tags\": \"administrator\"}{% if role_rabbitmq_monitoring_user is defined and role_rabbitmq_monitoring_user %},\n {\"name\": \"{{ role_rabbitmq_monitoring_user }}\", \"password\": \"{{ role_rabbitmq_monitoring_password }}\", \"tags\": \"monitoring\"}{% endif %}{% if project_name == 'outward_rabbitmq' %},\n {\"name\": \"{{ murano_agent_rabbitmq_user }}\", \"password\": \"{{ murano_agent_rabbitmq_password }}\", \"tags\": \"management\"}\n {% endif %}\n ],\n \"permissions\": [\n {\"user\": \"{{ role_rabbitmq_user }}\", \"vhost\": \"/\", \"configure\": \".*\", \"write\": \".*\", \"read\": \".*\"}{% if role_rabbitmq_monitoring_user is defined and role_rabbitmq_monitoring_user %},\n {\"user\": \"{{ role_rabbitmq_monitoring_user }}\", \"vhost\": \"/\", \"configure\": \"^$\", \"write\": \"^$\", \"read\": \".*\"}{% endif %}{% if project_name == 'outward_rabbitmq' %},\n {\"user\": \"{{ murano_agent_rabbitmq_user }}\", \"vhost\": \"{{ murano_agent_rabbitmq_vhost }}\", \"configure\": \".*\", \"write\": \".*\", \"read\": \".*\"}\n {% endif %}\n ],\n \"policies\":[\"{\"vhost\": \"/\", \"name\": \"ha-all\", \"pattern\": \"^(?!(amq\\.)|(.*_fanout_)|(reply_)).*\", \"apply-to\": \"all\", \"definition\": {\"ha-mode\":\"all\"}, \"priority\":0}{% if project_name == 'outward_rabbitmq' %}\"]\n}\n"}
PLAY RECAP *********************************************************************************************************************************************************************** ctrl1 : ok=12 changed=0 unreachable=0 failed=1 skipped=3 rescued=0 ignored=0 ctrl2 : ok=12 changed=0 unreachable=0 failed=1 skipped=3 rescued=0 ignored=0 ctrl3 : ok=11 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 localhost : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
On Fri, Feb 10, 2023 at 12:01 AM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
you will need add
"{"vhost": "/", "name": "ha-all", "pattern": "^(?!(amq\.)|(.*_fanout_)|(reply_)).*", "apply-to": "all", "definition": {"ha-mode":"all"}, "priority":0}{% if project_name == 'outward_rabbitmq' %}" to
kolla-ansible/ansible/roles/rabbitmq/templates/definitions.json.j2
Nguyen Huu Khoi
On Fri, Feb 10, 2023 at 10:47 AM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I have deployed 3 node kolla controllers and notice rabbitMQ doesn't have any HA policy for queues. when i rebooted one of the nodes and found some messages got lost.
Any reason kolla doesn't use HA policies by default?
If i want to add policies then where should i specify them or any doc you can point me to define policies.
Finally figured it out (venv-kolla) root@ctrl1:/etc/kolla# cat /etc/kolla/config/rabbitmq/definitions.json { "vhosts": [{ "name": "/" }], "users": [{ "name": "openstack", "password": "3eQJwZ2IqFz4mM0SCEC5CzG3DB", "tags": "administrator" }, { "name": "monitoring", "password": "81nCTxycXxBAlr0mHCEqkK5y4s", "tags": "monitoring" } ], "permissions": [{ "user": "openstack", "vhost": "/", "configure": ".*", "write": ".*", "read": ".*" }, { "user": "monitoring", "vhost": "/", "configure": "^$", "write": "^$", "read": ".*" } ], "policies": [{ "vhost": "/", "name": "ha-all", "pattern": "^(?!(amq\\.)|(.*_fanout_)|(reply_)).*", "apply-to": "all", "definition": { "ha-mode": "all" }, "priority": 0 }] } On Sat, Feb 11, 2023 at 12:22 PM Satish Patel <satish.txt@gmail.com> wrote:
This is my config file /etc/kolla/config/rabbitmq/definitions.json
https://paste.opendev.org/show/b5V1jGCqM3QAs54bxcTC/
On Sat, Feb 11, 2023 at 12:10 PM Satish Patel <satish.txt@gmail.com> wrote:
Hi Nguyễn,
I have tried your config but getting syntax errors, what version of rabbit are you running? i have created file and added your config: /etc/rabbitmq/rabbitmq_definitions.jso
Getting the following error and I am running Ubuntu 22.04 and Zed release
TASK [rabbitmq : Copying over definitions.json] ********************************************************************************************************************************** An exception occurred during task execution. To see the full traceback, use -vvv. The error was: } failed: [ctrl1] (item=/etc/kolla/config/rabbitmq/definitions.json) => {"ansible_loop_var": "item", "changed": false, "item": "/etc/kolla/config/rabbitmq/definitions.json", "msg": "AnsibleError: template error while templating string: Unexpected end of template. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'. The innermost block that needs to be closed is 'if'.. String: {\n \"vhosts\": [\n {\"name\": \"/\"}{% if project_name == 'outward_rabbitmq' %},\n {\"name\": \"{{ murano_agent_rabbitmq_vhost }}\"}\n {% endif %}\n ],\n \"users\": [\n {\"name\": \"{{ role_rabbitmq_user }}\", \"password\": \"{{ role_rabbitmq_password }}\", \"tags\": \"administrator\"}{% if role_rabbitmq_monitoring_user is defined and role_rabbitmq_monitoring_user %},\n {\"name\": \"{{ role_rabbitmq_monitoring_user }}\", \"password\": \"{{ role_rabbitmq_monitoring_password }}\", \"tags\": \"monitoring\"}{% endif %}{% if project_name == 'outward_rabbitmq' %},\n {\"name\": \"{{ murano_agent_rabbitmq_user }}\", \"password\": \"{{ murano_agent_rabbitmq_password }}\", \"tags\": \"management\"}\n {% endif %}\n ],\n \"permissions\": [\n {\"user\": \"{{ role_rabbitmq_user }}\", \"vhost\": \"/\", \"configure\": \".*\", \"write\": \".*\", \"read\": \".*\"}{% if role_rabbitmq_monitoring_user is defined and role_rabbitmq_monitoring_user %},\n {\"user\": \"{{ role_rabbitmq_monitoring_user }}\", \"vhost\": \"/\", \"configure\": \"^$\", \"write\": \"^$\", \"read\": \".*\"}{% endif %}{% if project_name == 'outward_rabbitmq' %},\n {\"user\": \"{{ murano_agent_rabbitmq_user }}\", \"vhost\": \"{{ murano_agent_rabbitmq_vhost }}\", \"configure\": \".*\", \"write\": \".*\", \"read\": \".*\"}\n {% endif %}\n ],\n \"policies\":[\"{\"vhost\": \"/\", \"name\": \"ha-all\", \"pattern\": \"^(?!(amq\\.)|(.*_fanout_)|(reply_)).*\", \"apply-to\": \"all\", \"definition\": {\"ha-mode\":\"all\"}, \"priority\":0}{% if project_name == 'outward_rabbitmq' %}\"]\n}\n"} An exception occurred during task execution. To see the full traceback, use -vvv. The error was: } failed: [ctrl2] (item=/etc/kolla/config/rabbitmq/definitions.json) => {"ansible_loop_var": "item", "changed": false, "item": "/etc/kolla/config/rabbitmq/definitions.json", "msg": "AnsibleError: template error while templating string: Unexpected end of template. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'. The innermost block that needs to be closed is 'if'.. String: {\n \"vhosts\": [\n {\"name\": \"/\"}{% if project_name == 'outward_rabbitmq' %},\n {\"name\": \"{{ murano_agent_rabbitmq_vhost }}\"}\n {% endif %}\n ],\n \"users\": [\n {\"name\": \"{{ role_rabbitmq_user }}\", \"password\": \"{{ role_rabbitmq_password }}\", \"tags\": \"administrator\"}{% if role_rabbitmq_monitoring_user is defined and role_rabbitmq_monitoring_user %},\n {\"name\": \"{{ role_rabbitmq_monitoring_user }}\", \"password\": \"{{ role_rabbitmq_monitoring_password }}\", \"tags\": \"monitoring\"}{% endif %}{% if project_name == 'outward_rabbitmq' %},\n {\"name\": \"{{ murano_agent_rabbitmq_user }}\", \"password\": \"{{ murano_agent_rabbitmq_password }}\", \"tags\": \"management\"}\n {% endif %}\n ],\n \"permissions\": [\n {\"user\": \"{{ role_rabbitmq_user }}\", \"vhost\": \"/\", \"configure\": \".*\", \"write\": \".*\", \"read\": \".*\"}{% if role_rabbitmq_monitoring_user is defined and role_rabbitmq_monitoring_user %},\n {\"user\": \"{{ role_rabbitmq_monitoring_user }}\", \"vhost\": \"/\", \"configure\": \"^$\", \"write\": \"^$\", \"read\": \".*\"}{% endif %}{% if project_name == 'outward_rabbitmq' %},\n {\"user\": \"{{ murano_agent_rabbitmq_user }}\", \"vhost\": \"{{ murano_agent_rabbitmq_vhost }}\", \"configure\": \".*\", \"write\": \".*\", \"read\": \".*\"}\n {% endif %}\n ],\n \"policies\":[\"{\"vhost\": \"/\", \"name\": \"ha-all\", \"pattern\": \"^(?!(amq\\.)|(.*_fanout_)|(reply_)).*\", \"apply-to\": \"all\", \"definition\": {\"ha-mode\":\"all\"}, \"priority\":0}{% if project_name == 'outward_rabbitmq' %}\"]\n}\n"} An exception occurred during task execution. To see the full traceback, use -vvv. The error was: } failed: [ctrl3] (item=/etc/kolla/config/rabbitmq/definitions.json) => {"ansible_loop_var": "item", "changed": false, "item": "/etc/kolla/config/rabbitmq/definitions.json", "msg": "AnsibleError: template error while templating string: Unexpected end of template. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'. The innermost block that needs to be closed is 'if'.. String: {\n \"vhosts\": [\n {\"name\": \"/\"}{% if project_name == 'outward_rabbitmq' %},\n {\"name\": \"{{ murano_agent_rabbitmq_vhost }}\"}\n {% endif %}\n ],\n \"users\": [\n {\"name\": \"{{ role_rabbitmq_user }}\", \"password\": \"{{ role_rabbitmq_password }}\", \"tags\": \"administrator\"}{% if role_rabbitmq_monitoring_user is defined and role_rabbitmq_monitoring_user %},\n {\"name\": \"{{ role_rabbitmq_monitoring_user }}\", \"password\": \"{{ role_rabbitmq_monitoring_password }}\", \"tags\": \"monitoring\"}{% endif %}{% if project_name == 'outward_rabbitmq' %},\n {\"name\": \"{{ murano_agent_rabbitmq_user }}\", \"password\": \"{{ murano_agent_rabbitmq_password }}\", \"tags\": \"management\"}\n {% endif %}\n ],\n \"permissions\": [\n {\"user\": \"{{ role_rabbitmq_user }}\", \"vhost\": \"/\", \"configure\": \".*\", \"write\": \".*\", \"read\": \".*\"}{% if role_rabbitmq_monitoring_user is defined and role_rabbitmq_monitoring_user %},\n {\"user\": \"{{ role_rabbitmq_monitoring_user }}\", \"vhost\": \"/\", \"configure\": \"^$\", \"write\": \"^$\", \"read\": \".*\"}{% endif %}{% if project_name == 'outward_rabbitmq' %},\n {\"user\": \"{{ murano_agent_rabbitmq_user }}\", \"vhost\": \"{{ murano_agent_rabbitmq_vhost }}\", \"configure\": \".*\", \"write\": \".*\", \"read\": \".*\"}\n {% endif %}\n ],\n \"policies\":[\"{\"vhost\": \"/\", \"name\": \"ha-all\", \"pattern\": \"^(?!(amq\\.)|(.*_fanout_)|(reply_)).*\", \"apply-to\": \"all\", \"definition\": {\"ha-mode\":\"all\"}, \"priority\":0}{% if project_name == 'outward_rabbitmq' %}\"]\n}\n"}
PLAY RECAP *********************************************************************************************************************************************************************** ctrl1 : ok=12 changed=0 unreachable=0 failed=1 skipped=3 rescued=0 ignored=0 ctrl2 : ok=12 changed=0 unreachable=0 failed=1 skipped=3 rescued=0 ignored=0 ctrl3 : ok=11 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 localhost : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
On Fri, Feb 10, 2023 at 12:01 AM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
you will need add
"{"vhost": "/", "name": "ha-all", "pattern": "^(?!(amq\.)|(.*_fanout_)|(reply_)).*", "apply-to": "all", "definition": {"ha-mode":"all"}, "priority":0}{% if project_name == 'outward_rabbitmq' %}" to
kolla-ansible/ansible/roles/rabbitmq/templates/definitions.json.j2
Nguyen Huu Khoi
On Fri, Feb 10, 2023 at 10:47 AM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I have deployed 3 node kolla controllers and notice rabbitMQ doesn't have any HA policy for queues. when i rebooted one of the nodes and found some messages got lost.
Any reason kolla doesn't use HA policies by default?
If i want to add policies then where should i specify them or any doc you can point me to define policies.
This has been discussed at great length; here are a couple of links that lead to interesting threads: https://lists.openstack.org/pipermail/openstack-discuss/2021-November/026074... https://lists.openstack.org/pipermail/openstack-discuss/2020-August/016362.h... There used to be a useful document here: https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit That link now points to the “Large Scale SIG” home page. Does anyone know where the useful RMQ document is? I assume that it is buried somewhere under “Large Scale Journey.” The bottom line is that, by default, kolla-ansible enables replication but not durable-queues, which is an unstable configuration. I believe that this may have been fixed in a newer version; we are on Train. https://lists.openstack.org/pipermail/openstack-discuss/2020-August/016524.h... On Thursday, February 9, 2023, 10:48:11 PM EST, Satish Patel <satish.txt@gmail.com> wrote: Folks, I have deployed 3 node kolla controllers and notice rabbitMQ doesn't have any HA policy for queues. when i rebooted one of the nodes and found some messages got lost. Any reason kolla doesn't use HA policies by default? If i want to add policies then where should i specify them or any doc you can point me to define policies.
Hi Albert, the Large Scale Sig documentation for rabbit is at: https://docs.openstack.org/large-scale/journey/configure/rabbitmq.html -- Felix Huettner From: Albert Braden <ozzzo@yahoo.com> Sent: Friday, February 10, 2023 5:16 PM To: OpenStack Discuss <openstack-discuss@lists.openstack.org>; Satish Patel <satish.txt@gmail.com> Subject: Re: [kolla]RabbitMQ HA policies define This has been discussed at great length; here are a couple of links that lead to interesting threads: https://lists.openstack.org/pipermail/openstack-discuss/2021-November/026074... https://lists.openstack.org/pipermail/openstack-discuss/2020-August/016362.h... There used to be a useful document here: https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit That link now points to the “Large Scale SIG” home page. Does anyone know where the useful RMQ document is? I assume that it is buried somewhere under “Large Scale Journey.” The bottom line is that, by default, kolla-ansible enables replication but not durable-queues, which is an unstable configuration. I believe that this may have been fixed in a newer version; we are on Train. https://lists.openstack.org/pipermail/openstack-discuss/2020-August/016524.h... On Thursday, February 9, 2023, 10:48:11 PM EST, Satish Patel <satish.txt@gmail.com<mailto:satish.txt@gmail.com>> wrote: Folks, I have deployed 3 node kolla controllers and notice rabbitMQ doesn't have any HA policy for queues. when i rebooted one of the nodes and found some messages got lost. Any reason kolla doesn't use HA policies by default? If i want to add policies then where should i specify them or any doc you can point me to define policies. Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hier<https://www.datenschutz.schwarz>.
Found it: https://docs.openstack.org/large-scale/journey/configure/rabbitmq.html On Friday, February 10, 2023, 11:23:12 AM EST, Albert Braden <ozzzo@yahoo.com> wrote: This has been discussed at great length; here are a couple of links that lead to interesting threads: https://lists.openstack.org/pipermail/openstack-discuss/2021-November/026074... https://lists.openstack.org/pipermail/openstack-discuss/2020-August/016362.h... There used to be a useful document here: https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit That link now points to the “Large Scale SIG” home page. Does anyone know where the useful RMQ document is? I assume that it is buried somewhere under “Large Scale Journey.” The bottom line is that, by default, kolla-ansible enables replication but not durable-queues, which is an unstable configuration. I believe that this may have been fixed in a newer version; we are on Train. https://lists.openstack.org/pipermail/openstack-discuss/2020-August/016524.h... On Thursday, February 9, 2023, 10:48:11 PM EST, Satish Patel <satish.txt@gmail.com> wrote: Folks, I have deployed 3 node kolla controllers and notice rabbitMQ doesn't have any HA policy for queues. when i rebooted one of the nodes and found some messages got lost. Any reason kolla doesn't use HA policies by default? If i want to add policies then where should i specify them or any doc you can point me to define policies.
Hi, I went through that doc because it always shows up first in google search :) My question was why the kolla default doesn't come with HA policies? I can surely add policy by myself and adjust requirements. I have also noticed without HA if any node fails then cause strange behavior in the entire cluster because of missing queues/data etc. Personally I hate HA because the majority of messages are stateless. In my experience HA is a little stable when it comes down to node failure but with performance big no. On Fri, Feb 10, 2023 at 11:15 AM Albert Braden <ozzzo@yahoo.com> wrote:
This has been discussed at great length; here are a couple of links that lead to interesting threads:
https://lists.openstack.org/pipermail/openstack-discuss/2021-November/026074...
https://lists.openstack.org/pipermail/openstack-discuss/2020-August/016362.h...
There used to be a useful document here: https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit
That link now points to the “Large Scale SIG” home page. Does anyone know where the useful RMQ document is? I assume that it is buried somewhere under “Large Scale Journey.”
The bottom line is that, by default, kolla-ansible enables replication but not durable-queues, which is an unstable configuration. I believe that this may have been fixed in a newer version; we are on Train.
https://lists.openstack.org/pipermail/openstack-discuss/2020-August/016524.h... On Thursday, February 9, 2023, 10:48:11 PM EST, Satish Patel < satish.txt@gmail.com> wrote:
Folks,
I have deployed 3 node kolla controllers and notice rabbitMQ doesn't have any HA policy for queues. when i rebooted one of the nodes and found some messages got lost.
Any reason kolla doesn't use HA policies by default?
If i want to add policies then where should i specify them or any doc you can point me to define policies.
IIRC the plan is to switch to Quorum Queues: https://blog.rabbitmq.com/posts/2020/04/rabbitmq-gets-an-ha-upgrade/ I'm an operator; hopefully the devs can tell us more about these plans. On Friday, February 10, 2023, 11:54:15 AM EST, Satish Patel <satish.txt@gmail.com> wrote: Hi, I went through that doc because it always shows up first in google search :) My question was why the kolla default doesn't come with HA policies? I can surely add policy by myself and adjust requirements. I have also noticed without HA if any node fails then cause strange behavior in the entire cluster because of missing queues/data etc. Personally I hate HA because the majority of messages are stateless. In my experience HA is a little stable when it comes down to node failure but with performance big no. On Fri, Feb 10, 2023 at 11:15 AM Albert Braden <ozzzo@yahoo.com> wrote: This has been discussed at great length; here are a couple of links that lead to interesting threads: https://lists.openstack.org/pipermail/openstack-discuss/2021-November/026074... https://lists.openstack.org/pipermail/openstack-discuss/2020-August/016362.h... There used to be a useful document here: https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit That link now points to the “Large Scale SIG” home page. Does anyone know where the useful RMQ document is? I assume that it is buried somewhere under “Large Scale Journey.” The bottom line is that, by default, kolla-ansible enables replication but not durable-queues, which is an unstable configuration. I believe that this may have been fixed in a newer version; we are on Train. https://lists.openstack.org/pipermail/openstack-discuss/2020-August/016524.h... On Thursday, February 9, 2023, 10:48:11 PM EST, Satish Patel <satish.txt@gmail.com> wrote: Folks, I have deployed 3 node kolla controllers and notice rabbitMQ doesn't have any HA policy for queues. when i rebooted one of the nodes and found some messages got lost. Any reason kolla doesn't use HA policies by default? If i want to add policies then where should i specify them or any doc you can point me to define policies.
Hi, in this change kolla-ansible removed setting the HA policies per default (along with why). TL;DR: The option `amqp_durable_queues` was not enabled on all openstack services, therefor HA queues do not make any sense (and actually actively make things worse). -- Felix Huettner From: Satish Patel <satish.txt@gmail.com> Sent: Friday, February 10, 2023 5:47 PM To: ozzzo@yahoo.com Cc: OpenStack Discuss <openstack-discuss@lists.openstack.org> Subject: Re: [kolla]RabbitMQ HA policies define Hi, I went through that doc because it always shows up first in google search :) My question was why the kolla default doesn't come with HA policies? I can surely add policy by myself and adjust requirements. I have also noticed without HA if any node fails then cause strange behavior in the entire cluster because of missing queues/data etc. Personally I hate HA because the majority of messages are stateless. In my experience HA is a little stable when it comes down to node failure but with performance big no. On Fri, Feb 10, 2023 at 11:15 AM Albert Braden <ozzzo@yahoo.com<mailto:ozzzo@yahoo.com>> wrote: This has been discussed at great length; here are a couple of links that lead to interesting threads: https://lists.openstack.org/pipermail/openstack-discuss/2021-November/026074.html<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.openstack.org%2Fpipermail%2Fopenstack-discuss%2F2021-November%2F026074.html&data=05%7C01%7C%7C60f6a1fa3ee74910b82808db0b875351%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C638116448067688419%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2F2ewiZjNMTiVbO13shuiQBGgvHNWSAxw6cNCZ8GN2zo%3D&reserved=0> https://lists.openstack.org/pipermail/openstack-discuss/2020-August/016362.html<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.openstack.org%2Fpipermail%2Fopenstack-discuss%2F2020-August%2F016362.html&data=05%7C01%7C%7C60f6a1fa3ee74910b82808db0b875351%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C638116448067688419%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Nivuk0mlqy%2F%2By8de3%2FiadRhB10XhSCgOeGszwdjRx0Q%3D&reserved=0> There used to be a useful document here: https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.openstack.org%2Fwiki%2FLarge_Scale_Configuration_Rabbit&data=05%7C01%7C%7C60f6a1fa3ee74910b82808db0b875351%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C638116448067688419%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=U9w8zVCx1C%2BJxBIB5iP8nTxoI4FDHnv76ezQmOgYubw%3D&reserved=0> That link now points to the "Large Scale SIG" home page. Does anyone know where the useful RMQ document is? I assume that it is buried somewhere under "Large Scale Journey." The bottom line is that, by default, kolla-ansible enables replication but not durable-queues, which is an unstable configuration. I believe that this may have been fixed in a newer version; we are on Train. https://lists.openstack.org/pipermail/openstack-discuss/2020-August/016524.html<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.openstack.org%2Fpipermail%2Fopenstack-discuss%2F2020-August%2F016524.html&data=05%7C01%7C%7C60f6a1fa3ee74910b82808db0b875351%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C638116448067688419%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=q%2FVgkhGeq0q59W9h5%2BCqKCt7lbgMh1RdIMbxE%2F0LBW4%3D&reserved=0> On Thursday, February 9, 2023, 10:48:11 PM EST, Satish Patel <satish.txt@gmail.com<mailto:satish.txt@gmail.com>> wrote: Folks, I have deployed 3 node kolla controllers and notice rabbitMQ doesn't have any HA policy for queues. when i rebooted one of the nodes and found some messages got lost. Any reason kolla doesn't use HA policies by default? If i want to add policies then where should i specify them or any doc you can point me to define policies. Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hier<https://www.datenschutz.schwarz>.
participants (4)
-
Albert Braden
-
Felix Hüttner
-
Nguyễn Hữu Khôi
-
Satish Patel