<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto">Hi Pierre,<div><br></div><div>Right I messed it up just today. Copy pasted the command from a kolla page about updating and this refers to master. </div><div><br></div><div>One more question about the variables for http proxy. They have to be in /etc/kolla/globals.yml? In case I want to go with the central local docker repo. I have run the steps to install a local docker repo on my seed node. The node from where I run kolla-ansible to deploy Openstack. Do I need to run a bootstrap for all-in-one so that the proxy setting is configured?<br><br><div dir="ltr">Von meinem iPhone gesendet</div><div dir="ltr"><br><blockquote type="cite">Am 02.09.2022 um 18:52 schrieb Pierre Riteau <pierre@stackhpc.com>:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div>Hi Oliver,</div><div><br></div><div>I think you have a mismatch of your kolla-ansible version here. If you look at the fluent image that it is trying to pull, you will see that it has only master-* tags: <a href="https://quay.io/repository/openstack.kolla/fluentd?tab=tags&tag=latest">https://quay.io/repository/openstack.kolla/fluentd?tab=tags&tag=latest</a></div><div><br></div><div>This is because in the development version of kolla-ansible (currently the master branch, which will be released as zed later this year) a new image naming scheme is used.</div><div><br></div><div>You need to switch your kolla-ansible checkout to the stable/yoga branch and reinstall it to fetch the right image, which will be something like centos-source-fluentd:yoga as seen here: <a href="https://quay.io/repository/openstack.kolla/centos-source-fluentd?tab=tags&tag=latest">https://quay.io/repository/openstack.kolla/centos-source-fluentd?tab=tags&tag=latest</a></div><div><br></div><div>Cheers,</div><div>Pierre Riteau (priteau)</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 2 Sept 2022 at 17:49, Oliver Weinmann <<a href="mailto:oliver.weinmann@me.com">oliver.weinmann@me.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div>
<p>Hi Pierre,</p>
<p>thanks for the quick reply and pointing me in the right
direction. I used a central local docker registry in the past. Is
this still the latest documentation:</p>
<p><a href="https://docs.openstack.org/kolla-ansible/latest/user/multinode.html" target="_blank">https://docs.openstack.org/kolla-ansible/latest/user/multinode.html</a><br>
</p>
<p>I stopped using it, because I always came across problems where
it was no longer possible to pull any container images. Just today
I had this issue without using a local registry: <br>
</p>
<p>TASK [service-images-pull : common | Pull images]
***********************************************************************************************************************************************************************************************************************************************************************************************************************************************************<br>
FAILED - RETRYING: common | Pull images (3 retries left).<br>
FAILED - RETRYING: common | Pull images (3 retries left).<br>
FAILED - RETRYING: common | Pull images (3 retries left).<br>
FAILED - RETRYING: common | Pull images (3 retries left).<br>
FAILED - RETRYING: common | Pull images (3 retries left).<br>
FAILED - RETRYING: common | Pull images (2 retries left).<br>
FAILED - RETRYING: common | Pull images (2 retries left).<br>
FAILED - RETRYING: common | Pull images (2 retries left).<br>
FAILED - RETRYING: common | Pull images (2 retries left).<br>
FAILED - RETRYING: common | Pull images (2 retries left).<br>
FAILED - RETRYING: common | Pull images (1 retries left).<br>
FAILED - RETRYING: common | Pull images (1 retries left).<br>
FAILED - RETRYING: common | Pull images (1 retries left).<br>
FAILED - RETRYING: common | Pull images (1 retries left).<br>
FAILED - RETRYING: common | Pull images (1 retries left).<br>
failed: [gedaopl09] (item=fluentd) => {"ansible_loop_var":
"item", "attempts": 3, "changed": true, "item": {"key": "fluentd",
"value": {"container_name": "fluentd", "dimensions": {},
"enabled": true, "environment": {"KOLLA_CONFIG_STRATEGY":
"COPY_ALWAYS"}, "group": "fluentd", "image":
"<a href="http://quay.io/openstack.kolla/fluentd:yoga-centos-stream8" target="_blank">quay.io/openstack.kolla/fluentd:yoga-centos-stream8</a>", "volumes":
["/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro",
"/etc/localtime:/etc/localtime:ro", "",
"kolla_logs:/var/log/kolla/",
"fluentd_<a>data:/var/lib/fluentd/data/</a>"]}}, "msg": "'Traceback (most
recent call last):\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/client.py\",
line 268, in _raise_for_status\\n
response.raise_for_status()\\n File
\"/usr/lib/python3.6/site-packages/requests/models.py\", line 940,
in raise_for_status\\n raise HTTPError(http_error_msg,
response=self)\\nrequests.exceptions.HTTPError: 404 Client Error:
Not Found for url:
http+docker://localhost/v1.41/images/create?tag=yoga-centos-stream8&fromImage=<a href="http://quay.io" target="_blank">quay.io</a>%2Fopenstack.kolla%2Ffluentd\\n\\nDuring
handling of the above exception, another exception
occurred:\\n\\nTraceback (most recent call last):\\n File
\"/tmp/ansible_kolla_docker_payload_w20pu6xy/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\",
line 381, in main\\n File
\"/tmp/ansible_kolla_docker_payload_w20pu6xy/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\",
line 451, in pull_image\\n repository=image, tag=tag,
stream=True\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/image.py\",
line 430, in pull\\n self._raise_for_status(response)\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/client.py\",
line 270, in _raise_for_status\\n raise
create_api_error_from_http_exception(e)\\n File
\"/usr/local/lib/python3.6/site-packages/docker/errors.py\", line
31, in create_api_error_from_http_exception\\n raise cls(e,
response=response,
explanation=explanation)\\ndocker.errors.NotFound: 404 Client
Error for
http+docker://localhost/v1.41/images/create?tag=yoga-centos-stream8&fromImage=<a href="http://quay.io" target="_blank">quay.io</a>%2Fopenstack.kolla%2Ffluentd:
Not Found (\"manifest for
<a href="http://quay.io/openstack.kolla/fluentd:yoga-centos-stream8" target="_blank">quay.io/openstack.kolla/fluentd:yoga-centos-stream8</a> not found:
manifest unknown: manifest unknown\")\\n'"}<br>
failed: [gedaopl11] (item=fluentd) => {"ansible_loop_var":
"item", "attempts": 3, "changed": true, "item": {"key": "fluentd",
"value": {"container_name": "fluentd", "dimensions": {},
"enabled": true, "environment": {"KOLLA_CONFIG_STRATEGY":
"COPY_ALWAYS"}, "group": "fluentd", "image":
"<a href="http://quay.io/openstack.kolla/fluentd:yoga-centos-stream8" target="_blank">quay.io/openstack.kolla/fluentd:yoga-centos-stream8</a>", "volumes":
["/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro",
"/etc/localtime:/etc/localtime:ro", "",
"kolla_logs:/var/log/kolla/",
"fluentd_<a>data:/var/lib/fluentd/data/</a>"]}}, "msg": "'Traceback (most
recent call last):\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/client.py\",
line 268, in _raise_for_status\\n
response.raise_for_status()\\n File
\"/usr/lib/python3.6/site-packages/requests/models.py\", line 940,
in raise_for_status\\n raise HTTPError(http_error_msg,
response=self)\\nrequests.exceptions.HTTPError: 404 Client Error:
Not Found for url:
http+docker://localhost/v1.41/images/create?tag=yoga-centos-stream8&fromImage=<a href="http://quay.io" target="_blank">quay.io</a>%2Fopenstack.kolla%2Ffluentd\\n\\nDuring
handling of the above exception, another exception
occurred:\\n\\nTraceback (most recent call last):\\n File
\"/tmp/ansible_kolla_docker_payload_xgz79c_7/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\",
line 381, in main\\n File
\"/tmp/ansible_kolla_docker_payload_xgz79c_7/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\",
line 451, in pull_image\\n repository=image, tag=tag,
stream=True\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/image.py\",
line 430, in pull\\n self._raise_for_status(response)\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/client.py\",
line 270, in _raise_for_status\\n raise
create_api_error_from_http_exception(e)\\n File
\"/usr/local/lib/python3.6/site-packages/docker/errors.py\", line
31, in create_api_error_from_http_exception\\n raise cls(e,
response=response,
explanation=explanation)\\ndocker.errors.NotFound: 404 Client
Error for
http+docker://localhost/v1.41/images/create?tag=yoga-centos-stream8&fromImage=<a href="http://quay.io" target="_blank">quay.io</a>%2Fopenstack.kolla%2Ffluentd:
Not Found (\"manifest for
<a href="http://quay.io/openstack.kolla/fluentd:yoga-centos-stream8" target="_blank">quay.io/openstack.kolla/fluentd:yoga-centos-stream8</a> not found:
manifest unknown: manifest unknown\")\\n'"}<br>
failed: [gedaopl04] (item=fluentd) => {"ansible_loop_var":
"item", "attempts": 3, "changed": true, "item": {"key": "fluentd",
"value": {"container_name": "fluentd", "dimensions": {},
"enabled": true, "environment": {"KOLLA_CONFIG_STRATEGY":
"COPY_ALWAYS"}, "group": "fluentd", "image":
"<a href="http://quay.io/openstack.kolla/fluentd:yoga-centos-stream8" target="_blank">quay.io/openstack.kolla/fluentd:yoga-centos-stream8</a>", "volumes":
["/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro",
"/etc/localtime:/etc/localtime:ro", "",
"kolla_logs:/var/log/kolla/",
"fluentd_<a>data:/var/lib/fluentd/data/</a>"]}}, "msg": "'Traceback (most
recent call last):\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/client.py\",
line 268, in _raise_for_status\\n
response.raise_for_status()\\n File
\"/usr/lib/python3.6/site-packages/requests/models.py\", line 940,
in raise_for_status\\n raise HTTPError(http_error_msg,
response=self)\\nrequests.exceptions.HTTPError: 404 Client Error:
Not Found for url:
http+docker://localhost/v1.41/images/create?tag=yoga-centos-stream8&fromImage=<a href="http://quay.io" target="_blank">quay.io</a>%2Fopenstack.kolla%2Ffluentd\\n\\nDuring
handling of the above exception, another exception
occurred:\\n\\nTraceback (most recent call last):\\n File
\"/tmp/ansible_kolla_docker_payload_1petvtop/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\",
line 381, in main\\n File
\"/tmp/ansible_kolla_docker_payload_1petvtop/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\",
line 451, in pull_image\\n repository=image, tag=tag,
stream=True\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/image.py\",
line 430, in pull\\n self._raise_for_status(response)\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/client.py\",
line 270, in _raise_for_status\\n raise
create_api_error_from_http_exception(e)\\n File
\"/usr/local/lib/python3.6/site-packages/docker/errors.py\", line
31, in create_api_error_from_http_exception\\n raise cls(e,
response=response,
explanation=explanation)\\ndocker.errors.NotFound: 404 Client
Error for
http+docker://localhost/v1.41/images/create?tag=yoga-centos-stream8&fromImage=<a href="http://quay.io" target="_blank">quay.io</a>%2Fopenstack.kolla%2Ffluentd:
Not Found (\"manifest for
<a href="http://quay.io/openstack.kolla/fluentd:yoga-centos-stream8" target="_blank">quay.io/openstack.kolla/fluentd:yoga-centos-stream8</a> not found:
manifest unknown: manifest unknown\")\\n'"}<br>
failed: [gedaopl07] (item=fluentd) => {"ansible_loop_var":
"item", "attempts": 3, "changed": true, "item": {"key": "fluentd",
"value": {"container_name": "fluentd", "dimensions": {},
"enabled": true, "environment": {"KOLLA_CONFIG_STRATEGY":
"COPY_ALWAYS"}, "group": "fluentd", "image":
"<a href="http://quay.io/openstack.kolla/fluentd:yoga-centos-stream8" target="_blank">quay.io/openstack.kolla/fluentd:yoga-centos-stream8</a>", "volumes":
["/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro",
"/etc/localtime:/etc/localtime:ro", "",
"kolla_logs:/var/log/kolla/",
"fluentd_<a>data:/var/lib/fluentd/data/</a>"]}}, "msg": "'Traceback (most
recent call last):\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/client.py\",
line 268, in _raise_for_status\\n
response.raise_for_status()\\n File
\"/usr/lib/python3.6/site-packages/requests/models.py\", line 940,
in raise_for_status\\n raise HTTPError(http_error_msg,
response=self)\\nrequests.exceptions.HTTPError: 404 Client Error:
Not Found for url:
http+docker://localhost/v1.41/images/create?tag=yoga-centos-stream8&fromImage=<a href="http://quay.io" target="_blank">quay.io</a>%2Fopenstack.kolla%2Ffluentd\\n\\nDuring
handling of the above exception, another exception
occurred:\\n\\nTraceback (most recent call last):\\n File
\"/tmp/ansible_kolla_docker_payload_uzj0wg5e/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\",
line 381, in main\\n File
\"/tmp/ansible_kolla_docker_payload_uzj0wg5e/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\",
line 451, in pull_image\\n repository=image, tag=tag,
stream=True\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/image.py\",
line 430, in pull\\n self._raise_for_status(response)\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/client.py\",
line 270, in _raise_for_status\\n raise
create_api_error_from_http_exception(e)\\n File
\"/usr/local/lib/python3.6/site-packages/docker/errors.py\", line
31, in create_api_error_from_http_exception\\n raise cls(e,
response=response,
explanation=explanation)\\ndocker.errors.NotFound: 404 Client
Error for
http+docker://localhost/v1.41/images/create?tag=yoga-centos-stream8&fromImage=<a href="http://quay.io" target="_blank">quay.io</a>%2Fopenstack.kolla%2Ffluentd:
Not Found (\"manifest for
<a href="http://quay.io/openstack.kolla/fluentd:yoga-centos-stream8" target="_blank">quay.io/openstack.kolla/fluentd:yoga-centos-stream8</a> not found:
manifest unknown: manifest unknown\")\\n'"}<br>
failed: [gedaopl10] (item=fluentd) => {"ansible_loop_var":
"item", "attempts": 3, "changed": true, "item": {"key": "fluentd",
"value": {"container_name": "fluentd", "dimensions": {},
"enabled": true, "environment": {"KOLLA_CONFIG_STRATEGY":
"COPY_ALWAYS"}, "group": "fluentd", "image":
"<a href="http://quay.io/openstack.kolla/fluentd:yoga-centos-stream8" target="_blank">quay.io/openstack.kolla/fluentd:yoga-centos-stream8</a>", "volumes":
["/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro",
"/etc/localtime:/etc/localtime:ro", "",
"kolla_logs:/var/log/kolla/",
"fluentd_<a>data:/var/lib/fluentd/data/</a>"]}}, "msg": "'Traceback (most
recent call last):\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/client.py\",
line 268, in _raise_for_status\\n
response.raise_for_status()\\n File
\"/usr/lib/python3.6/site-packages/requests/models.py\", line 940,
in raise_for_status\\n raise HTTPError(http_error_msg,
response=self)\\nrequests.exceptions.HTTPError: 404 Client Error:
Not Found for url:
http+docker://localhost/v1.41/images/create?tag=yoga-centos-stream8&fromImage=<a href="http://quay.io" target="_blank">quay.io</a>%2Fopenstack.kolla%2Ffluentd\\n\\nDuring
handling of the above exception, another exception
occurred:\\n\\nTraceback (most recent call last):\\n File
\"/tmp/ansible_kolla_docker_payload_a6kiyuhn/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\",
line 381, in main\\n File
\"/tmp/ansible_kolla_docker_payload_a6kiyuhn/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\",
line 451, in pull_image\\n repository=image, tag=tag,
stream=True\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/image.py\",
line 430, in pull\\n self._raise_for_status(response)\\n File
\"/usr/local/lib/python3.6/site-packages/docker/api/client.py\",
line 270, in _raise_for_status\\n raise
create_api_error_from_http_exception(e)\\n File
\"/usr/local/lib/python3.6/site-packages/docker/errors.py\", line
31, in create_api_error_from_http_exception\\n raise cls(e,
response=response,
explanation=explanation)\\ndocker.errors.NotFound: 404 Client
Error for
http+docker://localhost/v1.41/images/create?tag=yoga-centos-stream8&fromImage=<a href="http://quay.io" target="_blank">quay.io</a>%2Fopenstack.kolla%2Ffluentd:
Not Found (\"manifest for
<a href="http://quay.io/openstack.kolla/fluentd:yoga-centos-stream8" target="_blank">quay.io/openstack.kolla/fluentd:yoga-centos-stream8</a> not found:
manifest unknown: manifest unknown\")\\n'"}<br>
FAILED - RETRYING: common | Pull images (3 retries left).<br>
FAILED - RETRYING: common | Pull images (3 retries left).<br>
FAILED - RETRYING: common | Pull images (3 retries left).<br>
FAILED - RETRYING: common | Pull images (3 retries left).<br>
FAILED - RETRYING: common | Pull images (3 retries left).<br>
FAILED - RETRYING: common | Pull images (2 retries left).<br>
FAILED - RETRYING: common | Pull images (2 retries left).<br>
FAILED - RETRYING: common | Pull images (2 retries left).<br>
FAILED - RETRYING: common | Pull images (2 retries left).<br>
FAILED - RETRYING: common | Pull images (2 retries left).<br>
</p>
<p>I'm on release Yoga and for some weird reason, I found the
suggestion to change release yoga to master in
/etc/kolla/globals.yaml. And this works. Any ideas why?<br>
</p>
<div>Am 02.09.2022 um 14:19 schrieb Pierre
Riteau:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">Hi Oliver,
<div><br>
</div>
<div>The Kolla Ansible documentation describes how to
configure Docker to use a proxy: <a href="https://docs.openstack.org/kolla-ansible/latest/reference/deployment-and-bootstrapping/bootstrap-servers.html#configuration" target="_blank">https://docs.openstack.org/kolla-ansible/latest/reference/deployment-and-bootstrapping/bootstrap-servers.html#configuration</a></div>
<div>This feature was introduced in Wallaby.</div>
<div><br>
</div>
<div>Additionally, I suggest you configure a local Docker
registry and use it to fetch container images. The same
documentation page describes related variables.</div>
<div><br>
</div>
<div>Cheers,</div>
<div>Pierre Riteau (priteau)</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, 2 Sept 2022 at
13:33, Oliver Weinmann <<a href="mailto:oliver.weinmann@me.com" target="_blank">oliver.weinmann@me.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
I was wondering if there is a way to set a http proxy in
kolla-ansible <br>
during deployment / update etc. Today I wanted to pull new
containers <br>
and this saturates our internet connection. What we usually
do is have <br>
machine that massively download stuff from the internet use
a squid <br>
proxy that throttles their bandwidth.<br>
<br>
I found the following feature request:<br>
<br>
<a href="https://bugs.launchpad.net/kolla-ansible/+bug/1829586" rel="noreferrer" target="_blank">https://bugs.launchpad.net/kolla-ansible/+bug/1829586</a><br>
<br>
And it seems some work had been done, but I can't seem to
find any info <br>
if the change really made it into production:<br>
<br>
<a href="https://review.opendev.org/c/openstack/kolla-ansible/+/692006/" rel="noreferrer" target="_blank">https://review.opendev.org/c/openstack/kolla-ansible/+/692006/</a><br>
<br>
Any help or hints are highly appreciated.<br>
<br>
Cheers,<br>
<br>
Oliver<br>
<br>
<br>
<br>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote></div></div>
</div></blockquote></div></body></html>