Hi Pierre,
thanks for the quick reply and pointing me in the right direction. I used a central local docker registry in the past. Is this still the latest documentation:
https://docs.openstack.org/kolla-ansible/latest/user/multinode.html
I stopped using it, because I always came across problems where
it was no longer possible to pull any container images. Just today
I had this issue without using a local registry:
TASK [service-images-pull : common | Pull images]
***********************************************************************************************************************************************************************************************************************************************************************************************************************************************************
FAILED - RETRYING: common | Pull images (3 retries left).
FAILED - RETRYING: common | Pull images (3 retries left).
FAILED - RETRYING: common | Pull images (3 retries left).
FAILED - RETRYING: common | Pull images (3 retries left).
FAILED - RETRYING: common | Pull images (3 retries left).
FAILED - RETRYING: common | Pull images (2 retries left).
FAILED - RETRYING: common | Pull images (2 retries left).
FAILED - RETRYING: common | Pull images (2 retries left).
FAILED - RETRYING: common | Pull images (2 retries left).
FAILED - RETRYING: common | Pull images (2 retries left).
FAILED - RETRYING: common | Pull images (1 retries left).
FAILED - RETRYING: common | Pull images (1 retries left).
FAILED - RETRYING: common | Pull images (1 retries left).
FAILED - RETRYING: common | Pull images (1 retries left).
FAILED - RETRYING: common | Pull images (1 retries left).
failed: [gedaopl09] (item=fluentd) => {"ansible_loop_var":
"item", "attempts": 3, "changed": true, "item": {"key": "fluentd",
"value": {"container_name": "fluentd", "dimensions": {},
"enabled": true, "environment": {"KOLLA_CONFIG_STRATEGY":
"COPY_ALWAYS"}, "group": "fluentd", "image":
"quay.io/openstack.kolla/fluentd:yoga-centos-stream8", "volumes":
["/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro",
"/etc/localtime:/etc/localtime:ro", "",
"kolla_logs:/var/log/kolla/",
"fluentd_data:/var/lib/fluentd/data/"]}}, "msg": "'Traceback (most
recent call last):\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/client.py\",
line 268, in _raise_for_status\\n
response.raise_for_status()\\n File
\"/usr/lib/python3.6/site-packages/requests/models.py\", line 940,
in raise_for_status\\n raise HTTPError(http_error_msg,
response=self)\\nrequests.exceptions.HTTPError: 404 Client Error:
Not Found for url:
http+docker://localhost/v1.41/images/create?tag=yoga-centos-stream8&fromImage=quay.io%2Fopenstack.kolla%2Ffluentd\\n\\nDuring
handling of the above exception, another exception
occurred:\\n\\nTraceback (most recent call last):\\n File
\"/tmp/ansible_kolla_docker_payload_w20pu6xy/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\",
line 381, in main\\n File
\"/tmp/ansible_kolla_docker_payload_w20pu6xy/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\",
line 451, in pull_image\\n repository=image, tag=tag,
stream=True\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/image.py\",
line 430, in pull\\n self._raise_for_status(response)\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/client.py\",
line 270, in _raise_for_status\\n raise
create_api_error_from_http_exception(e)\\n File
\"/usr/local/lib/python3.6/site-packages/docker/errors.py\", line
31, in create_api_error_from_http_exception\\n raise cls(e,
response=response,
explanation=explanation)\\ndocker.errors.NotFound: 404 Client
Error for
http+docker://localhost/v1.41/images/create?tag=yoga-centos-stream8&fromImage=quay.io%2Fopenstack.kolla%2Ffluentd:
Not Found (\"manifest for
quay.io/openstack.kolla/fluentd:yoga-centos-stream8 not found:
manifest unknown: manifest unknown\")\\n'"}
failed: [gedaopl11] (item=fluentd) => {"ansible_loop_var":
"item", "attempts": 3, "changed": true, "item": {"key": "fluentd",
"value": {"container_name": "fluentd", "dimensions": {},
"enabled": true, "environment": {"KOLLA_CONFIG_STRATEGY":
"COPY_ALWAYS"}, "group": "fluentd", "image":
"quay.io/openstack.kolla/fluentd:yoga-centos-stream8", "volumes":
["/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro",
"/etc/localtime:/etc/localtime:ro", "",
"kolla_logs:/var/log/kolla/",
"fluentd_data:/var/lib/fluentd/data/"]}}, "msg": "'Traceback (most
recent call last):\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/client.py\",
line 268, in _raise_for_status\\n
response.raise_for_status()\\n File
\"/usr/lib/python3.6/site-packages/requests/models.py\", line 940,
in raise_for_status\\n raise HTTPError(http_error_msg,
response=self)\\nrequests.exceptions.HTTPError: 404 Client Error:
Not Found for url:
http+docker://localhost/v1.41/images/create?tag=yoga-centos-stream8&fromImage=quay.io%2Fopenstack.kolla%2Ffluentd\\n\\nDuring
handling of the above exception, another exception
occurred:\\n\\nTraceback (most recent call last):\\n File
\"/tmp/ansible_kolla_docker_payload_xgz79c_7/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\",
line 381, in main\\n File
\"/tmp/ansible_kolla_docker_payload_xgz79c_7/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\",
line 451, in pull_image\\n repository=image, tag=tag,
stream=True\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/image.py\",
line 430, in pull\\n self._raise_for_status(response)\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/client.py\",
line 270, in _raise_for_status\\n raise
create_api_error_from_http_exception(e)\\n File
\"/usr/local/lib/python3.6/site-packages/docker/errors.py\", line
31, in create_api_error_from_http_exception\\n raise cls(e,
response=response,
explanation=explanation)\\ndocker.errors.NotFound: 404 Client
Error for
http+docker://localhost/v1.41/images/create?tag=yoga-centos-stream8&fromImage=quay.io%2Fopenstack.kolla%2Ffluentd:
Not Found (\"manifest for
quay.io/openstack.kolla/fluentd:yoga-centos-stream8 not found:
manifest unknown: manifest unknown\")\\n'"}
failed: [gedaopl04] (item=fluentd) => {"ansible_loop_var":
"item", "attempts": 3, "changed": true, "item": {"key": "fluentd",
"value": {"container_name": "fluentd", "dimensions": {},
"enabled": true, "environment": {"KOLLA_CONFIG_STRATEGY":
"COPY_ALWAYS"}, "group": "fluentd", "image":
"quay.io/openstack.kolla/fluentd:yoga-centos-stream8", "volumes":
["/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro",
"/etc/localtime:/etc/localtime:ro", "",
"kolla_logs:/var/log/kolla/",
"fluentd_data:/var/lib/fluentd/data/"]}}, "msg": "'Traceback (most
recent call last):\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/client.py\",
line 268, in _raise_for_status\\n
response.raise_for_status()\\n File
\"/usr/lib/python3.6/site-packages/requests/models.py\", line 940,
in raise_for_status\\n raise HTTPError(http_error_msg,
response=self)\\nrequests.exceptions.HTTPError: 404 Client Error:
Not Found for url:
http+docker://localhost/v1.41/images/create?tag=yoga-centos-stream8&fromImage=quay.io%2Fopenstack.kolla%2Ffluentd\\n\\nDuring
handling of the above exception, another exception
occurred:\\n\\nTraceback (most recent call last):\\n File
\"/tmp/ansible_kolla_docker_payload_1petvtop/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\",
line 381, in main\\n File
\"/tmp/ansible_kolla_docker_payload_1petvtop/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\",
line 451, in pull_image\\n repository=image, tag=tag,
stream=True\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/image.py\",
line 430, in pull\\n self._raise_for_status(response)\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/client.py\",
line 270, in _raise_for_status\\n raise
create_api_error_from_http_exception(e)\\n File
\"/usr/local/lib/python3.6/site-packages/docker/errors.py\", line
31, in create_api_error_from_http_exception\\n raise cls(e,
response=response,
explanation=explanation)\\ndocker.errors.NotFound: 404 Client
Error for
http+docker://localhost/v1.41/images/create?tag=yoga-centos-stream8&fromImage=quay.io%2Fopenstack.kolla%2Ffluentd:
Not Found (\"manifest for
quay.io/openstack.kolla/fluentd:yoga-centos-stream8 not found:
manifest unknown: manifest unknown\")\\n'"}
failed: [gedaopl07] (item=fluentd) => {"ansible_loop_var":
"item", "attempts": 3, "changed": true, "item": {"key": "fluentd",
"value": {"container_name": "fluentd", "dimensions": {},
"enabled": true, "environment": {"KOLLA_CONFIG_STRATEGY":
"COPY_ALWAYS"}, "group": "fluentd", "image":
"quay.io/openstack.kolla/fluentd:yoga-centos-stream8", "volumes":
["/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro",
"/etc/localtime:/etc/localtime:ro", "",
"kolla_logs:/var/log/kolla/",
"fluentd_data:/var/lib/fluentd/data/"]}}, "msg": "'Traceback (most
recent call last):\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/client.py\",
line 268, in _raise_for_status\\n
response.raise_for_status()\\n File
\"/usr/lib/python3.6/site-packages/requests/models.py\", line 940,
in raise_for_status\\n raise HTTPError(http_error_msg,
response=self)\\nrequests.exceptions.HTTPError: 404 Client Error:
Not Found for url:
http+docker://localhost/v1.41/images/create?tag=yoga-centos-stream8&fromImage=quay.io%2Fopenstack.kolla%2Ffluentd\\n\\nDuring
handling of the above exception, another exception
occurred:\\n\\nTraceback (most recent call last):\\n File
\"/tmp/ansible_kolla_docker_payload_uzj0wg5e/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\",
line 381, in main\\n File
\"/tmp/ansible_kolla_docker_payload_uzj0wg5e/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\",
line 451, in pull_image\\n repository=image, tag=tag,
stream=True\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/image.py\",
line 430, in pull\\n self._raise_for_status(response)\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/client.py\",
line 270, in _raise_for_status\\n raise
create_api_error_from_http_exception(e)\\n File
\"/usr/local/lib/python3.6/site-packages/docker/errors.py\", line
31, in create_api_error_from_http_exception\\n raise cls(e,
response=response,
explanation=explanation)\\ndocker.errors.NotFound: 404 Client
Error for
http+docker://localhost/v1.41/images/create?tag=yoga-centos-stream8&fromImage=quay.io%2Fopenstack.kolla%2Ffluentd:
Not Found (\"manifest for
quay.io/openstack.kolla/fluentd:yoga-centos-stream8 not found:
manifest unknown: manifest unknown\")\\n'"}
failed: [gedaopl10] (item=fluentd) => {"ansible_loop_var":
"item", "attempts": 3, "changed": true, "item": {"key": "fluentd",
"value": {"container_name": "fluentd", "dimensions": {},
"enabled": true, "environment": {"KOLLA_CONFIG_STRATEGY":
"COPY_ALWAYS"}, "group": "fluentd", "image":
"quay.io/openstack.kolla/fluentd:yoga-centos-stream8", "volumes":
["/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro",
"/etc/localtime:/etc/localtime:ro", "",
"kolla_logs:/var/log/kolla/",
"fluentd_data:/var/lib/fluentd/data/"]}}, "msg": "'Traceback (most
recent call last):\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/client.py\",
line 268, in _raise_for_status\\n
response.raise_for_status()\\n File
\"/usr/lib/python3.6/site-packages/requests/models.py\", line 940,
in raise_for_status\\n raise HTTPError(http_error_msg,
response=self)\\nrequests.exceptions.HTTPError: 404 Client Error:
Not Found for url:
http+docker://localhost/v1.41/images/create?tag=yoga-centos-stream8&fromImage=quay.io%2Fopenstack.kolla%2Ffluentd\\n\\nDuring
handling of the above exception, another exception
occurred:\\n\\nTraceback (most recent call last):\\n File
\"/tmp/ansible_kolla_docker_payload_a6kiyuhn/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\",
line 381, in main\\n File
\"/tmp/ansible_kolla_docker_payload_a6kiyuhn/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\",
line 451, in pull_image\\n repository=image, tag=tag,
stream=True\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/image.py\",
line 430, in pull\\n self._raise_for_status(response)\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/client.py\",
line 270, in _raise_for_status\\n raise
create_api_error_from_http_exception(e)\\n File
\"/usr/local/lib/python3.6/site-packages/docker/errors.py\", line
31, in create_api_error_from_http_exception\\n raise cls(e,
response=response,
explanation=explanation)\\ndocker.errors.NotFound: 404 Client
Error for
http+docker://localhost/v1.41/images/create?tag=yoga-centos-stream8&fromImage=quay.io%2Fopenstack.kolla%2Ffluentd:
Not Found (\"manifest for
quay.io/openstack.kolla/fluentd:yoga-centos-stream8 not found:
manifest unknown: manifest unknown\")\\n'"}
FAILED - RETRYING: common | Pull images (3 retries left).
FAILED - RETRYING: common | Pull images (3 retries left).
FAILED - RETRYING: common | Pull images (3 retries left).
FAILED - RETRYING: common | Pull images (3 retries left).
FAILED - RETRYING: common | Pull images (3 retries left).
FAILED - RETRYING: common | Pull images (2 retries left).
FAILED - RETRYING: common | Pull images (2 retries left).
FAILED - RETRYING: common | Pull images (2 retries left).
FAILED - RETRYING: common | Pull images (2 retries left).
FAILED - RETRYING: common | Pull images (2 retries left).
I'm on release Yoga and for some weird reason, I found the
suggestion to change release yoga to master in
/etc/kolla/globals.yaml. And this works. Any ideas why?
Hi Oliver,
The Kolla Ansible documentation describes how to configure Docker to use a proxy: https://docs.openstack.org/kolla-ansible/latest/reference/deployment-and-bootstrapping/bootstrap-servers.html#configurationThis feature was introduced in Wallaby.
Additionally, I suggest you configure a local Docker registry and use it to fetch container images. The same documentation page describes related variables.
Cheers,Pierre Riteau (priteau)
On Fri, 2 Sept 2022 at 13:33, Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi,
I was wondering if there is a way to set a http proxy in kolla-ansible
during deployment / update etc. Today I wanted to pull new containers
and this saturates our internet connection. What we usually do is have
machine that massively download stuff from the internet use a squid
proxy that throttles their bandwidth.
I found the following feature request:
https://bugs.launchpad.net/kolla-ansible/+bug/1829586
And it seems some work had been done, but I can't seem to find any info
if the change really made it into production:
https://review.opendev.org/c/openstack/kolla-ansible/+/692006/
Any help or hints are highly appreciated.
Cheers,
Oliver