[Kolla] FAILED - RETRYING: wait for MariaDB to be available via HAProxy (10 retries left).
Mark, Thanks for the tip. Now I get a failure (as below) during deploy (I have all-in-one) config. I am on stein. It appears to me that this has been presumably fixed . But I still running into this error. I do not see anything explicit in the HAProxy/MariaDB logs. TASK [mariadb : wait for MariaDB to be available via HAProxy] ************************************************************************************************** FAILED - RETRYING: wait for MariaDB to be available via HAProxy (10 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (9 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (8 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (7 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (6 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (5 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (4 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (3 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (2 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (1 retries left). fatal: [localhost]: FAILED! => {"attempts": 10, "changed": false, "elapsed": 60, "msg": "Timeout when waiting for search string MariaDB in 10.1.0.250:3306"} NO MORE HOSTS LEFT ********************************************************************************************************************************************* PLAY RECAP ***************************************************************************************************************************************************** localhost : ok=76 changed=0 unreachable=0 failed=1 skipped=74 rescued=0 ignored=0 Command failed ansible-playbook -i ../../all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e kolla_action=deploy /home/cloud-user/kolla-ansible/ansible/site.yml --RamaK -----Original Message----- From: Mark Goddard [mailto:mark@stackhpc.com] Sent: Monday, November 25, 2019 11:08 AM To: Bhupathi, Ramakrishna <C-Ramakrishna.Bhupathi@charter.com> Cc: openstack-discuss@lists.openstack.org Subject: Re: [Kolla] RabbitMQ failure during deploy (Openstack with Kolla) On Mon, 25 Nov 2019 at 13:54, Bhupathi, Ramakrishna <C-Ramakrishna.Bhupathi@charter.com> wrote:
Hey , I am evaluating Openstack with Kolla (have the latest ) and following the steps. I see that rabbitmq fails to start up and (keeps restarting). Essentially the script fails in deploy. I have a single node all in one config.
Can someone tell me what the cause for this failure?
RUNNING HANDLER [rabbitmq : Waiting for rabbitmq to start on first node] ********************************************************************** ******************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "docker exec rabbitmq rabbitmqctl wait /var/lib/rabbitmq/mnesia/rabbitmq.pid", "delta": "0:00:07.732346", "end": "2019-11-25 13:33:12.102342", "msg": "non-zero return code", "rc": 137, "start": "2019-11-25 13:33:04.369996", "stderr": "", "stderr_lines": [], "stdout": "Waiting for 'rabbit@kolla-ubuntu'\npid is 6", "stdout_lines": ["Waiting for 'rabbit@kolla-ubuntu'", "pid is 6"]}
RUNNING HANDLER [rabbitmq : Restart rabbitmq container (rest of nodes)] ********************************************************************** *******************************************
NO MORE HOSTS LEFT ********************************************************************** ********************************************************************** **************************
PLAY RECAP ********************************************************************** ********************************************************************** **********************************
localhost : ok=95 changed=2 unreachable=0 failed=1 skipped=78 rescued=0 ignored=0
Command failed ansible-playbook -i ./all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e kolla_action=deploy /home/ubuntu/venv/share/kolla-ansible/ansible/site.yml
Accessing the container logs for rabbitmq .. this is what I see
BOOT FAILED
===========
Error description:
{could_not_start,rabbitmq_management,
{rabbitmq_management,
{bad_return,
{{rabbit_mgmt_app,start,[normal,[]]},
{'EXIT',
{{could_not_start_listener,
[{port,15672}],
{shutdown,
{failed_to_start_child,ranch_acceptors_sup,
{listen_error,rabbit_web_dispatch_sup_15672,eaddrinuse}}}},
Hi RamaK, here is your issue. RabbitMQ management fails to start up due to the port already being in use (eaddrinuse). Perhaps RabbitMQ is running on the host already?
{gen_server,call,
[rabbit_web_dispatch_registry,
{add,rabbit_mgmt,
[{port,15672}],
#Fun<rabbit_web_dispatch.0.49789095>,
--RamaK
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
On Tue, 26 Nov 2019 at 14:12, Bhupathi, Ramakrishna <C-Ramakrishna.Bhupathi@charter.com> wrote:
Mark, Thanks for the tip.
Now I get a failure (as below) during deploy (I have all-in-one) config. I am on stein. It appears to me that this has been presumably fixed . But I still running into this error.
I do not see anything explicit in the HAProxy/MariaDB logs.
Are both the haproxy and mariadb containers up and not restarting? Does your api_interface have the VIP activated? Is haproxy listening on 10.1.0.250:3306 (check ss)?
TASK [mariadb : wait for MariaDB to be available via HAProxy] ************************************************************************************************** FAILED - RETRYING: wait for MariaDB to be available via HAProxy (10 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (9 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (8 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (7 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (6 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (5 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (4 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (3 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (2 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (1 retries left). fatal: [localhost]: FAILED! => {"attempts": 10, "changed": false, "elapsed": 60, "msg": "Timeout when waiting for search string MariaDB in 10.1.0.250:3306"}
NO MORE HOSTS LEFT *********************************************************************************************************************************************
PLAY RECAP ***************************************************************************************************************************************************** localhost : ok=76 changed=0 unreachable=0 failed=1 skipped=74 rescued=0 ignored=0
Command failed ansible-playbook -i ../../all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e kolla_action=deploy /home/cloud-user/kolla-ansible/ansible/site.yml
--RamaK
-----Original Message----- From: Mark Goddard [mailto:mark@stackhpc.com] Sent: Monday, November 25, 2019 11:08 AM To: Bhupathi, Ramakrishna <C-Ramakrishna.Bhupathi@charter.com> Cc: openstack-discuss@lists.openstack.org Subject: Re: [Kolla] RabbitMQ failure during deploy (Openstack with Kolla)
On Mon, 25 Nov 2019 at 13:54, Bhupathi, Ramakrishna <C-Ramakrishna.Bhupathi@charter.com> wrote:
Hey , I am evaluating Openstack with Kolla (have the latest ) and following the steps. I see that rabbitmq fails to start up and (keeps restarting). Essentially the script fails in deploy. I have a single node all in one config.
Can someone tell me what the cause for this failure?
RUNNING HANDLER [rabbitmq : Waiting for rabbitmq to start on first node] ********************************************************************** ******************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "docker exec rabbitmq rabbitmqctl wait /var/lib/rabbitmq/mnesia/rabbitmq.pid", "delta": "0:00:07.732346", "end": "2019-11-25 13:33:12.102342", "msg": "non-zero return code", "rc": 137, "start": "2019-11-25 13:33:04.369996", "stderr": "", "stderr_lines": [], "stdout": "Waiting for 'rabbit@kolla-ubuntu'\npid is 6", "stdout_lines": ["Waiting for 'rabbit@kolla-ubuntu'", "pid is 6"]}
RUNNING HANDLER [rabbitmq : Restart rabbitmq container (rest of nodes)] ********************************************************************** *******************************************
NO MORE HOSTS LEFT ********************************************************************** ********************************************************************** **************************
PLAY RECAP ********************************************************************** ********************************************************************** **********************************
localhost : ok=95 changed=2 unreachable=0 failed=1 skipped=78 rescued=0 ignored=0
Command failed ansible-playbook -i ./all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e kolla_action=deploy /home/ubuntu/venv/share/kolla-ansible/ansible/site.yml
Accessing the container logs for rabbitmq .. this is what I see
BOOT FAILED
===========
Error description:
{could_not_start,rabbitmq_management,
{rabbitmq_management,
{bad_return,
{{rabbit_mgmt_app,start,[normal,[]]},
{'EXIT',
{{could_not_start_listener,
[{port,15672}],
{shutdown,
{failed_to_start_child,ranch_acceptors_sup,
{listen_error,rabbit_web_dispatch_sup_15672,eaddrinuse}}}},
Hi RamaK, here is your issue. RabbitMQ management fails to start up due to the port already being in use (eaddrinuse). Perhaps RabbitMQ is running on the host already?
{gen_server,call,
[rabbit_web_dispatch_registry,
{add,rabbit_mgmt,
[{port,15672}],
#Fun<rabbit_web_dispatch.0.49789095>,
--RamaK
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
On Tue, Nov 26, 2019, at 7:04 AM, Mark Goddard wrote:
On Tue, 26 Nov 2019 at 14:12, Bhupathi, Ramakrishna <C-Ramakrishna.Bhupathi@charter.com> wrote:
Mark, Thanks for the tip.
Now I get a failure (as below) during deploy (I have all-in-one) config. I am on stein. It appears to me that this has been presumably fixed . But I still running into this error.
I do not see anything explicit in the HAProxy/MariaDB logs.
Are both the haproxy and mariadb containers up and not restarting? Does your api_interface have the VIP activated? Is haproxy listening on 10.1.0.250:3306 (check ss)?
We ran into mariadb startup slowness recently in Zuul CI jobs. Basically they generate timezone data tables on first startup which can be very slow depending on your disk IO. Not sure if this is the issue, but the workaround is to set an env var to disable the timezone data table generation (if you don't use those tables). https://github.com/docker-library/mariadb/issues/261 https://github.com/docker-library/mariadb/issues/262
Thanks Mark. The deploy worked after a reboot of my server. Appreciate your help. --RamaK -----Original Message----- From: Mark Goddard [mailto:mark@stackhpc.com] Sent: Tuesday, November 26, 2019 10:05 AM To: Bhupathi, Ramakrishna <C-Ramakrishna.Bhupathi@charter.com> Cc: openstack-discuss@lists.openstack.org Subject: Re: [Kolla] FAILED - RETRYING: wait for MariaDB to be available via HAProxy (10 retries left). On Tue, 26 Nov 2019 at 14:12, Bhupathi, Ramakrishna <C-Ramakrishna.Bhupathi@charter.com> wrote:
Mark, Thanks for the tip.
Now I get a failure (as below) during deploy (I have all-in-one) config. I am on stein. It appears to me that this has been presumably fixed . But I still running into this error.
I do not see anything explicit in the HAProxy/MariaDB logs.
Are both the haproxy and mariadb containers up and not restarting? Does your api_interface have the VIP activated? Is haproxy listening on 10.1.0.250:3306 (check ss)?
TASK [mariadb : wait for MariaDB to be available via HAProxy] ********************************************************************** **************************** FAILED - RETRYING: wait for MariaDB to be available via HAProxy (10 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (9 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (8 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (7 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (6 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (5 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (4 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (3 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (2 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (1 retries left). fatal: [localhost]: FAILED! => {"attempts": 10, "changed": false, "elapsed": 60, "msg": "Timeout when waiting for search string MariaDB in 10.1.0.250:3306"}
NO MORE HOSTS LEFT ********************************************************************** ********************************************************************** *
PLAY RECAP ***************************************************************************************************************************************************** localhost : ok=76 changed=0 unreachable=0 failed=1 skipped=74 rescued=0 ignored=0
Command failed ansible-playbook -i ../../all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e kolla_action=deploy /home/cloud-user/kolla-ansible/ansible/site.yml
--RamaK
-----Original Message----- From: Mark Goddard [mailto:mark@stackhpc.com] Sent: Monday, November 25, 2019 11:08 AM To: Bhupathi, Ramakrishna <C-Ramakrishna.Bhupathi@charter.com> Cc: openstack-discuss@lists.openstack.org Subject: Re: [Kolla] RabbitMQ failure during deploy (Openstack with Kolla)
On Mon, 25 Nov 2019 at 13:54, Bhupathi, Ramakrishna <C-Ramakrishna.Bhupathi@charter.com> wrote:
Hey , I am evaluating Openstack with Kolla (have the latest ) and following the steps. I see that rabbitmq fails to start up and (keeps restarting). Essentially the script fails in deploy. I have a single node all in one config.
Can someone tell me what the cause for this failure?
RUNNING HANDLER [rabbitmq : Waiting for rabbitmq to start on first node] ******************************************************************** ** ******************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "docker exec rabbitmq rabbitmqctl wait /var/lib/rabbitmq/mnesia/rabbitmq.pid", "delta": "0:00:07.732346", "end": "2019-11-25 13:33:12.102342", "msg": "non-zero return code", "rc": 137, "start": "2019-11-25 13:33:04.369996", "stderr": "", "stderr_lines": [], "stdout": "Waiting for 'rabbit@kolla-ubuntu'\npid is 6", "stdout_lines": ["Waiting for 'rabbit@kolla-ubuntu'", "pid is 6"]}
RUNNING HANDLER [rabbitmq : Restart rabbitmq container (rest of nodes)] ******************************************************************** ** *******************************************
NO MORE HOSTS LEFT ******************************************************************** ** ******************************************************************** ** **************************
PLAY RECAP ******************************************************************** ** ******************************************************************** ** **********************************
localhost : ok=95 changed=2 unreachable=0 failed=1 skipped=78 rescued=0 ignored=0
Command failed ansible-playbook -i ./all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e kolla_action=deploy /home/ubuntu/venv/share/kolla-ansible/ansible/site.yml
Accessing the container logs for rabbitmq .. this is what I see
BOOT FAILED
===========
Error description:
{could_not_start,rabbitmq_management,
{rabbitmq_management,
{bad_return,
{{rabbit_mgmt_app,start,[normal,[]]},
{'EXIT',
{{could_not_start_listener,
[{port,15672}],
{shutdown,
{failed_to_start_child,ranch_acceptors_sup,
{listen_error,rabbit_web_dispatch_sup_15672,eaddrinuse}}}},
Hi RamaK, here is your issue. RabbitMQ management fails to start up due to the port already being in use (eaddrinuse). Perhaps RabbitMQ is running on the host already?
{gen_server,call,
[rabbit_web_dispatch_registry,
{add,rabbit_mgmt,
[{port,15672}],
#Fun<rabbit_web_dispatch.0.49789095>,
--RamaK
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
participants (3)
-
Bhupathi, Ramakrishna
-
Clark Boylan
-
Mark Goddard