Hello, I'm facing a problem with kolla-toolbox while deploying zed-ubunu-jammy-aarch64 on a Raspberry Pi cluster using kolla-ansible. I 've built the images myself (on Raspbbery Pi) using kolla-build because images for Ubuntu-aarch64 were missing in standard repos (for Debian/aarch64, stuff installs from quay.io and works fine, but wanted to try out Ubuntu). The error can be seen at the very end of a fragment of the ansible log pasted below. As seen, images in my local repo are reachable (fluentd & kolla-toolbox restart succesfully). But in task "Initializing kolla-toolbox container" three hosts fail with the notification "Container xyz is restarting, wait until the container is running", while the fourth one gives "unable to upgrade to tcp, received 409". Interestingly, in a preceding task "Ensuring config directories have correct owner and permission" (the very first part of the log below), kolla-toolbox check is skipped. I can provide more data, create a bug report in appropriate place, etc. But please instruct me what and where. Best regards, Darek TASK [common : Ensuring config directories have correct owner and permission] ************************************************************************************************ ok: [ost64] => (item={'key': 'fluentd', 'value': {'container_name': 'fluentd', 'group': 'fluentd', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/fluentd:zed-ubuntu-jammy-aarch64', 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS'}, 'volumes': ['/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/', 'fluentd_data:/var/lib/fluentd/data/'], 'dimensions': {}}}) ok: [ost61] => (item={'key': 'fluentd', 'value': {'container_name': 'fluentd', 'group': 'fluentd', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/fluentd:zed-ubuntu-jammy-aarch64', 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS'}, 'volumes': ['/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/', 'fluentd_data:/var/lib/fluentd/data/'], 'dimensions': {}}}) skipping: [ost64] => (item={'key': 'kolla-toolbox', 'value': {'container_name': 'kolla_toolbox', 'group': 'kolla-toolbox', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/kolla-toolbox:zed-ubuntu-jammy-aarch64', 'environment': {'ANSIBLE_NOCOLOR': '1', 'ANSIBLE_LIBRARY': '/usr/share/ansible'}, 'privileged': True, 'volumes': ['/etc/kolla/kolla-toolbox/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', '/dev/:/dev/', '/run/:/run/:shared', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) skipping: [ost61] => (item={'key': 'kolla-toolbox', 'value': {'container_name': 'kolla_toolbox', 'group': 'kolla-toolbox', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/kolla-toolbox:zed-ubuntu-jammy-aarch64', 'environment': {'ANSIBLE_NOCOLOR': '1', 'ANSIBLE_LIBRARY': '/usr/share/ansible'}, 'privileged': True, 'volumes': ['/etc/kolla/kolla-toolbox/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', '/dev/:/dev/', '/run/:/run/:shared', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) ok: [ost62] => (item={'key': 'fluentd', 'value': {'container_name': 'fluentd', 'group': 'fluentd', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/fluentd:zed-ubuntu-jammy-aarch64', 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS'}, 'volumes': ['/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/', 'fluentd_data:/var/lib/fluentd/data/'], 'dimensions': {}}}) skipping: [ost62] => (item={'key': 'kolla-toolbox', 'value': {'container_name': 'kolla_toolbox', 'group': 'kolla-toolbox', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/kolla-toolbox:zed-ubuntu-jammy-aarch64', 'environment': {'ANSIBLE_NOCOLOR': '1', 'ANSIBLE_LIBRARY': '/usr/share/ansible'}, 'privileged': True, 'volumes': ['/etc/kolla/kolla-toolbox/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', '/dev/:/dev/', '/run/:/run/:shared', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) ok: [ost63] => (item={'key': 'fluentd', 'value': {'container_name': 'fluentd', 'group': 'fluentd', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/fluentd:zed-ubuntu-jammy-aarch64', 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS'}, 'volumes': ['/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/', 'fluentd_data:/var/lib/fluentd/data/'], 'dimensions': {}}}) skipping: [ost63] => (item={'key': 'kolla-toolbox', 'value': {'container_name': 'kolla_toolbox', 'group': 'kolla-toolbox', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/kolla-toolbox:zed-ubuntu-jammy-aarch64', 'environment': {'ANSIBLE_NOCOLOR': '1', 'ANSIBLE_LIBRARY': '/usr/share/ansible'}, 'privileged': True, 'volumes': ['/etc/kolla/kolla-toolbox/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', '/dev/:/dev/', '/run/:/run/:shared', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) ok: [ost64] => (item={'key': 'cron', 'value': {'container_name': 'cron', 'group': 'cron', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/cron:zed-ubuntu-jammy-aarch64', 'environment': {'KOLLA_LOGROTATE_SCHEDULE': 'daily'}, 'volumes': ['/etc/kolla/cron/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) ok: [ost61] => (item={'key': 'cron', 'value': {'container_name': 'cron', 'group': 'cron', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/cron:zed-ubuntu-jammy-aarch64', 'environment': {'KOLLA_LOGROTATE_SCHEDULE': 'daily'}, 'volumes': ['/etc/kolla/cron/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) ok: [ost62] => (item={'key': 'cron', 'value': {'container_name': 'cron', 'group': 'cron', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/cron:zed-ubuntu-jammy-aarch64', 'environment': {'KOLLA_LOGROTATE_SCHEDULE': 'daily'}, 'volumes': ['/etc/kolla/cron/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) ok: [ost63] => (item={'key': 'cron', 'value': {'container_name': 'cron', 'group': 'cron', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/cron:zed-ubuntu-jammy-aarch64', 'environment': {'KOLLA_LOGROTATE_SCHEDULE': 'daily'}, 'volumes': ['/etc/kolla/cron/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) TASK [common : Copy rabbitmq-env.conf to kolla toolbox] ********************************************************************************************************************** changed: [ost61] => (item=/home/ubuntu/labs/ostack/kolla-ansible-zed/share/kolla-ansible/ansible/roles/common/templates/rabbitmq-env.conf.j2) changed: [ost64] => (item=/home/ubuntu/labs/ostack/kolla-ansible-zed/share/kolla-ansible/ansible/roles/common/templates/rabbitmq-env.conf.j2) changed: [ost62] => (item=/home/ubuntu/labs/ostack/kolla-ansible-zed/share/kolla-ansible/ansible/roles/common/templates/rabbitmq-env.conf.j2) changed: [ost63] => (item=/home/ubuntu/labs/ostack/kolla-ansible-zed/share/kolla-ansible/ansible/roles/common/templates/rabbitmq-env.conf.j2) TASK [common : Copy rabbitmq erl_inetrc to kolla toolbox] ******************************************************************************************************************** changed: [ost64] => (item=/home/ubuntu/labs/ostack/kolla-ansible-zed/share/kolla-ansible/ansible/roles/common/templates/erl_inetrc.j2) changed: [ost61] => (item=/home/ubuntu/labs/ostack/kolla-ansible-zed/share/kolla-ansible/ansible/roles/common/templates/erl_inetrc.j2) changed: [ost62] => (item=/home/ubuntu/labs/ostack/kolla-ansible-zed/share/kolla-ansible/ansible/roles/common/templates/erl_inetrc.j2) changed: [ost63] => (item=/home/ubuntu/labs/ostack/kolla-ansible-zed/share/kolla-ansible/ansible/roles/common/templates/erl_inetrc.j2) TASK [common : Check common containers] ************************************************************************************************************************************** changed: [ost62] => (item={'key': 'fluentd', 'value': {'container_name': 'fluentd', 'group': 'fluentd', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/fluentd:zed-ubuntu-jammy-aarch64', 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS'}, 'volumes': ['/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/', 'fluentd_data:/var/lib/fluentd/data/'], 'dimensions': {}}}) changed: [ost61] => (item={'key': 'fluentd', 'value': {'container_name': 'fluentd', 'group': 'fluentd', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/fluentd:zed-ubuntu-jammy-aarch64', 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS'}, 'volumes': ['/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/', 'fluentd_data:/var/lib/fluentd/data/'], 'dimensions': {}}}) changed: [ost64] => (item={'key': 'fluentd', 'value': {'container_name': 'fluentd', 'group': 'fluentd', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/fluentd:zed-ubuntu-jammy-aarch64', 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS'}, 'volumes': ['/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/', 'fluentd_data:/var/lib/fluentd/data/'], 'dimensions': {}}}) changed: [ost63] => (item={'key': 'fluentd', 'value': {'container_name': 'fluentd', 'group': 'fluentd', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/fluentd:zed-ubuntu-jammy-aarch64', 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS'}, 'volumes': ['/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/', 'fluentd_data:/var/lib/fluentd/data/'], 'dimensions': {}}}) changed: [ost64] => (item={'key': 'kolla-toolbox', 'value': {'container_name': 'kolla_toolbox', 'group': 'kolla-toolbox', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/kolla-toolbox:zed-ubuntu-jammy-aarch64', 'environment': {'ANSIBLE_NOCOLOR': '1', 'ANSIBLE_LIBRARY': '/usr/share/ansible'}, 'privileged': True, 'volumes': ['/etc/kolla/kolla-toolbox/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', '/dev/:/dev/', '/run/:/run/:shared', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) changed: [ost61] => (item={'key': 'kolla-toolbox', 'value': {'container_name': 'kolla_toolbox', 'group': 'kolla-toolbox', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/kolla-toolbox:zed-ubuntu-jammy-aarch64', 'environment': {'ANSIBLE_NOCOLOR': '1', 'ANSIBLE_LIBRARY': '/usr/share/ansible'}, 'privileged': True, 'volumes': ['/etc/kolla/kolla-toolbox/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', '/dev/:/dev/', '/run/:/run/:shared', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) changed: [ost62] => (item={'key': 'kolla-toolbox', 'value': {'container_name': 'kolla_toolbox', 'group': 'kolla-toolbox', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/kolla-toolbox:zed-ubuntu-jammy-aarch64', 'environment': {'ANSIBLE_NOCOLOR': '1', 'ANSIBLE_LIBRARY': '/usr/share/ansible'}, 'privileged': True, 'volumes': ['/etc/kolla/kolla-toolbox/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', '/dev/:/dev/', '/run/:/run/:shared', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) changed: [ost63] => (item={'key': 'kolla-toolbox', 'value': {'container_name': 'kolla_toolbox', 'group': 'kolla-toolbox', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/kolla-toolbox:zed-ubuntu-jammy-aarch64', 'environment': {'ANSIBLE_NOCOLOR': '1', 'ANSIBLE_LIBRARY': '/usr/share/ansible'}, 'privileged': True, 'volumes': ['/etc/kolla/kolla-toolbox/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', '/dev/:/dev/', '/run/:/run/:shared', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) changed: [ost61] => (item={'key': 'cron', 'value': {'container_name': 'cron', 'group': 'cron', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/cron:zed-ubuntu-jammy-aarch64', 'environment': {'KOLLA_LOGROTATE_SCHEDULE': 'daily'}, 'volumes': ['/etc/kolla/cron/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) changed: [ost62] => (item={'key': 'cron', 'value': {'container_name': 'cron', 'group': 'cron', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/cron:zed-ubuntu-jammy-aarch64', 'environment': {'KOLLA_LOGROTATE_SCHEDULE': 'daily'}, 'volumes': ['/etc/kolla/cron/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) changed: [ost64] => (item={'key': 'cron', 'value': {'container_name': 'cron', 'group': 'cron', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/cron:zed-ubuntu-jammy-aarch64', 'environment': {'KOLLA_LOGROTATE_SCHEDULE': 'daily'}, 'volumes': ['/etc/kolla/cron/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) changed: [ost63] => (item={'key': 'cron', 'value': {'container_name': 'cron', 'group': 'cron', 'enabled': True, 'image': '192.168.2.59:443/openstack.kolla/cron:zed-ubuntu-jammy-aarch64', 'environment': {'KOLLA_LOGROTATE_SCHEDULE': 'daily'}, 'volumes': ['/etc/kolla/cron/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) TASK [common : Creating log volume] ****************************************************************************************************************************************** ok: [ost64] ok: [ost61] ok: [ost62] ok: [ost63] TASK [common : Link kolla_logs volume to /var/log/kolla] ********************************************************************************************************************* changed: [ost64] changed: [ost61] changed: [ost62] changed: [ost63] TASK [common : Flush handlers] *********************************************************************************************************************************************** RUNNING HANDLER [common : Restart fluentd container] ************************************************************************************************************************* changed: [ost63] changed: [ost64] changed: [ost62] changed: [ost61] RUNNING HANDLER [common : Restart kolla-toolbox container] ******************************************************************************************************************* changed: [ost62] changed: [ost61] changed: [ost63] changed: [ost64] RUNNING HANDLER [common : Initializing toolbox container using normal user] ************************************************************************************************** fatal: [ost62]: FAILED! => {"changed": false, "cmd": ["docker", "exec", "-t", "kolla_toolbox", "ansible", "--version"], "delta": "0:00:00.064846", "end": "2023-11-19 21:46:19.187519", "msg": "non-zero return code", "rc": 1, "start": "2023-11-19 21:46:19.122673", "stderr": "Error response from daemon: Container 34d4dcc029d807cb70061bcab3f21e1980ec2541b6e116764dab22917d1ba602 is restarting, wait until the container is running", "stderr_lines": ["Error response from daemon: Container 34d4dcc029d807cb70061bcab3f21e1980ec2541b6e116764dab22917d1ba602 is restarting, wait until the container is running"], "stdout": "", "stdout_lines": []} fatal: [ost61]: FAILED! => {"changed": false, "cmd": ["docker", "exec", "-t", "kolla_toolbox", "ansible", "--version"], "delta": "0:00:00.170840", "end": "2023-11-19 21:46:19.383867", "msg": "non-zero return code", "rc": 1, "start": "2023-11-19 21:46:19.213027", "stderr": "Error response from daemon: Container fbe3a84809b7b5e23dbc84e896735bcc8c72fbd300b53637061bd305a6bd45eb is restarting, wait until the container is running", "stderr_lines": ["Error response from daemon: Container fbe3a84809b7b5e23dbc84e896735bcc8c72fbd300b53637061bd305a6bd45eb is restarting, wait until the container is running"], "stdout": "", "stdout_lines": []} fatal: [ost63]: FAILED! => {"changed": false, "cmd": ["docker", "exec", "-t", "kolla_toolbox", "ansible", "--version"], "delta": "0:00:00.422087", "end": "2023-11-19 21:46:19.579842", "msg": "non-zero return code", "rc": 1, "start": "2023-11-19 21:46:19.157755", "stderr": "Error response from daemon: Container 2b8c1c77cd75d8780ad358407178d0e8d431e22885fc0f58ae1e08e746928d9d is restarting, wait until the container is running", "stderr_lines": ["Error response from daemon: Container 2b8c1c77cd75d8780ad358407178d0e8d431e22885fc0f58ae1e08e746928d9d is restarting, wait until the container is running"], "stdout": "", "stdout_lines": []} fatal: [ost64]: FAILED! => {"changed": false, "cmd": ["docker", "exec", "-t", "kolla_toolbox", "ansible", "--version"], "delta": "0:00:00.693630", "end": "2023-11-19 21:46:19.859925", "msg": "non-zero return code", "rc": 1, "start": "2023-11-19 21:46:19.166295", "stderr": "unable to upgrade to tcp, received 409", "stderr_lines": ["unable to upgrade to tcp, received 409"], "stdout": "", "stdout_lines": []}