[kolla] local registry setup issue
Folks, I have set up a local registry using the following method as per official doc. docker run -d \ --network host \ --name registry \ --restart=always \ -e REGISTRY_HTTP_ADDR=0.0.0.0:4000 \ -v registry:/var/lib/registry \ registry:2 After that I ran the following command to import all images to the local registry. docker images | grep kolla | grep -v local | awk '{print $1,$2}' | while read -r image tag; do new_image_name=${image#"quay.io/"} docker tag ${image}:${tag} "localhost:4000"/${new_image_name}:${tag} docker push localhost:4000/${new_image_name}:${tag} done Now i can see images in local registry using docker images command (venv-kolla) root@kolla-infra-1:/etc/kolla# docker images | grep localhost localhost:4000/openstack.kolla/ubuntu-source-nova-novncproxy yoga 01dc100dc65a 2 days ago 1.2GB localhost:4000/openstack.kolla/ubuntu-source-horizon yoga a016e1f5b9af 2 days ago 1.1GB localhost:4000/openstack.kolla/ubuntu-source-nova-conductor yoga 381a64a368bd 2 days ago 1.11GB localhost:4000/openstack.kolla/ubuntu-source-nova-api yoga f90a15a09142 2 days ago 1.11GB .... .... I have added it in global.yml docker_registry: 10.73.0.181:4000 docker_registry_insecure: yes Now when i am trying to add compute nodes then I get a registry error like the following. Not sure why its trying to use https://10.73.0.181:4000/v2 instead of http;// TASK [common : include_tasks] **************************************************************************************************************************************************** included: /root/venv-kolla/share/kolla-ansible/ansible/roles/common/tasks/pull.yml for kolla-comp-2 TASK [service-images-pull : common | Pull images] ******************************************************************************************************************************** FAILED - RETRYING: common | Pull images (3 retries left). FAILED - RETRYING: common | Pull images (2 retries left). FAILED - RETRYING: common | Pull images (1 retries left). failed: [kolla-comp-2] (item=fluentd) => {"ansible_loop_var": "item", "attempts": 3, "changed": true, "item": {"key": "fluentd", "value": {"container_name": "fluentd", "dimensions": {}, "enabled": true, "environment": {"KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS"}, "group": "fluentd", "image": " 10.73.0.181:4000/openstack.kolla/ubuntu-source-fluentd:yoga", "volumes": ["/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro", "/etc/localtime:/etc/localtime:ro", "/etc/timezone:/etc/timezone:ro", "kolla_logs:/var/log/kolla/", "fluentd_data:/var/lib/fluentd/data/"]}}, "msg": "'Traceback (most recent call last):\\n File \"/usr/lib/python3/dist-packages/docker/api/client.py\", line 261, in _raise_for_status\\n response.raise_for_status()\\n File \"/usr/lib/python3/dist-packages/requests/models.py\", line 940, in raise_for_status\\n raise HTTPError(http_error_msg, response=self)\\nrequests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/images/create?tag=yoga&fromImage=10.73.0.181%3A4000%2Fopenstack.kolla%2Fubuntu-source-fluentd\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File \"/tmp/ansible_kolla_docker_payload_6o9nw9xd/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 381, in main\\n File \"/tmp/ansible_kolla_docker_payload_6o9nw9xd/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\", line 450, in pull_image\\n json.loads(line.strip().decode(\\'utf-8\\')) for line in self.dc.pull(\\n File \"/usr/lib/python3/dist-packages/docker/api/image.py\", line 415, in pull\\n self._raise_for_status(response)\\n File \"/usr/lib/python3/dist-packages/docker/api/client.py\", line 263, in _raise_for_status\\n raise create_api_error_from_http_exception(e)\\n File \"/usr/lib/python3/dist-packages/docker/errors.py\", line 31, in create_api_error_from_http_exception\\n raise cls(e, response=response, explanation=explanation)\\ndocker.errors.APIError: 500 Server Error: Internal Server Error (\"Get \"https://10.73.0.181:4000/v2/\": http: server gave HTTP response to HTTPS client\")\\n'"} FAILED - RETRYING: common | Pull images (3 retries left).
You might need to configure daemon.json as described here: https://docs.openstack.org/kolla-ansible/rocky/user/multinode.html#configure... On Fri, Jan 27, 2023, 11:46 PM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I have set up a local registry using the following method as per official doc.
docker run -d \ --network host \ --name registry \ --restart=always \ -e REGISTRY_HTTP_ADDR=0.0.0.0:4000 \ -v registry:/var/lib/registry \ registry:2
After that I ran the following command to import all images to the local registry.
docker images | grep kolla | grep -v local | awk '{print $1,$2}' | while read -r image tag; do new_image_name=${image#"quay.io/"} docker tag ${image}:${tag} "localhost:4000"/${new_image_name}:${tag} docker push localhost:4000/${new_image_name}:${tag} done
Now i can see images in local registry using docker images command
(venv-kolla) root@kolla-infra-1:/etc/kolla# docker images | grep localhost localhost:4000/openstack.kolla/ubuntu-source-nova-novncproxy yoga 01dc100dc65a 2 days ago 1.2GB localhost:4000/openstack.kolla/ubuntu-source-horizon yoga a016e1f5b9af 2 days ago 1.1GB localhost:4000/openstack.kolla/ubuntu-source-nova-conductor yoga 381a64a368bd 2 days ago 1.11GB localhost:4000/openstack.kolla/ubuntu-source-nova-api yoga f90a15a09142 2 days ago 1.11GB .... ....
I have added it in global.yml
docker_registry: 10.73.0.181:4000 docker_registry_insecure: yes
Now when i am trying to add compute nodes then I get a registry error like the following. Not sure why its trying to use https://10.73.0.181:4000/v2 instead of http;//
TASK [common : include_tasks] **************************************************************************************************************************************************** included: /root/venv-kolla/share/kolla-ansible/ansible/roles/common/tasks/pull.yml for kolla-comp-2
TASK [service-images-pull : common | Pull images] ******************************************************************************************************************************** FAILED - RETRYING: common | Pull images (3 retries left). FAILED - RETRYING: common | Pull images (2 retries left). FAILED - RETRYING: common | Pull images (1 retries left). failed: [kolla-comp-2] (item=fluentd) => {"ansible_loop_var": "item", "attempts": 3, "changed": true, "item": {"key": "fluentd", "value": {"container_name": "fluentd", "dimensions": {}, "enabled": true, "environment": {"KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS"}, "group": "fluentd", "image": " 10.73.0.181:4000/openstack.kolla/ubuntu-source-fluentd:yoga", "volumes": ["/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro", "/etc/localtime:/etc/localtime:ro", "/etc/timezone:/etc/timezone:ro", "kolla_logs:/var/log/kolla/", "fluentd_data:/var/lib/fluentd/data/"]}}, "msg": "'Traceback (most recent call last):\\n File \"/usr/lib/python3/dist-packages/docker/api/client.py\", line 261, in _raise_for_status\\n response.raise_for_status()\\n File \"/usr/lib/python3/dist-packages/requests/models.py\", line 940, in raise_for_status\\n raise HTTPError(http_error_msg, response=self)\\nrequests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/images/create?tag=yoga&fromImage=10.73.0.181%3A4000%2Fopenstack.kolla%2Fubuntu-source-fluentd\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File \"/tmp/ansible_kolla_docker_payload_6o9nw9xd/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 381, in main\\n File \"/tmp/ansible_kolla_docker_payload_6o9nw9xd/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\", line 450, in pull_image\\n json.loads(line.strip().decode(\\'utf-8\\')) for line in self.dc.pull(\\n File \"/usr/lib/python3/dist-packages/docker/api/image.py\", line 415, in pull\\n self._raise_for_status(response)\\n File \"/usr/lib/python3/dist-packages/docker/api/client.py\", line 263, in _raise_for_status\\n raise create_api_error_from_http_exception(e)\\n File \"/usr/lib/python3/dist-packages/docker/errors.py\", line 31, in create_api_error_from_http_exception\\n raise cls(e, response=response, explanation=explanation)\\ndocker.errors.APIError: 500 Server Error: Internal Server Error (\"Get \"https://10.73.0.181:4000/v2/\": http: server gave HTTP response to HTTPS client\")\\n'"} FAILED - RETRYING: common | Pull images (3 retries left).
Great!! I don't know how I missed that part. It seems working but related to i got following error 10.73.0.181:4000/openstack.kolla/ubuntu-source-nova-ssh:yoga not found: manifest unknown I have checked and found I don't have that image in the registry. How do I download that image from quay.io and push it to the local registry? On Fri, Jan 27, 2023 at 11:59 PM Tobias McNulty <tobias@caktusgroup.com> wrote:
You might need to configure daemon.json as described here: https://docs.openstack.org/kolla-ansible/rocky/user/multinode.html#configure...
On Fri, Jan 27, 2023, 11:46 PM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I have set up a local registry using the following method as per official doc.
docker run -d \ --network host \ --name registry \ --restart=always \ -e REGISTRY_HTTP_ADDR=0.0.0.0:4000 \ -v registry:/var/lib/registry \ registry:2
After that I ran the following command to import all images to the local registry.
docker images | grep kolla | grep -v local | awk '{print $1,$2}' | while read -r image tag; do new_image_name=${image#"quay.io/"} docker tag ${image}:${tag} "localhost:4000"/${new_image_name}:${tag} docker push localhost:4000/${new_image_name}:${tag} done
Now i can see images in local registry using docker images command
(venv-kolla) root@kolla-infra-1:/etc/kolla# docker images | grep localhost localhost:4000/openstack.kolla/ubuntu-source-nova-novncproxy yoga 01dc100dc65a 2 days ago 1.2GB localhost:4000/openstack.kolla/ubuntu-source-horizon yoga a016e1f5b9af 2 days ago 1.1GB localhost:4000/openstack.kolla/ubuntu-source-nova-conductor yoga 381a64a368bd 2 days ago 1.11GB localhost:4000/openstack.kolla/ubuntu-source-nova-api yoga f90a15a09142 2 days ago 1.11GB .... ....
I have added it in global.yml
docker_registry: 10.73.0.181:4000 docker_registry_insecure: yes
Now when i am trying to add compute nodes then I get a registry error like the following. Not sure why its trying to use https://10.73.0.181:4000/v2 instead of http;//
TASK [common : include_tasks] **************************************************************************************************************************************************** included: /root/venv-kolla/share/kolla-ansible/ansible/roles/common/tasks/pull.yml for kolla-comp-2
TASK [service-images-pull : common | Pull images] ******************************************************************************************************************************** FAILED - RETRYING: common | Pull images (3 retries left). FAILED - RETRYING: common | Pull images (2 retries left). FAILED - RETRYING: common | Pull images (1 retries left). failed: [kolla-comp-2] (item=fluentd) => {"ansible_loop_var": "item", "attempts": 3, "changed": true, "item": {"key": "fluentd", "value": {"container_name": "fluentd", "dimensions": {}, "enabled": true, "environment": {"KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS"}, "group": "fluentd", "image": " 10.73.0.181:4000/openstack.kolla/ubuntu-source-fluentd:yoga", "volumes": ["/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro", "/etc/localtime:/etc/localtime:ro", "/etc/timezone:/etc/timezone:ro", "kolla_logs:/var/log/kolla/", "fluentd_data:/var/lib/fluentd/data/"]}}, "msg": "'Traceback (most recent call last):\\n File \"/usr/lib/python3/dist-packages/docker/api/client.py\", line 261, in _raise_for_status\\n response.raise_for_status()\\n File \"/usr/lib/python3/dist-packages/requests/models.py\", line 940, in raise_for_status\\n raise HTTPError(http_error_msg, response=self)\\nrequests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/images/create?tag=yoga&fromImage=10.73.0.181%3A4000%2Fopenstack.kolla%2Fubuntu-source-fluentd\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File \"/tmp/ansible_kolla_docker_payload_6o9nw9xd/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 381, in main\\n File \"/tmp/ansible_kolla_docker_payload_6o9nw9xd/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\", line 450, in pull_image\\n json.loads(line.strip().decode(\\'utf-8\\')) for line in self.dc.pull(\\n File \"/usr/lib/python3/dist-packages/docker/api/image.py\", line 415, in pull\\n self._raise_for_status(response)\\n File \"/usr/lib/python3/dist-packages/docker/api/client.py\", line 263, in _raise_for_status\\n raise create_api_error_from_http_exception(e)\\n File \"/usr/lib/python3/dist-packages/docker/errors.py\", line 31, in create_api_error_from_http_exception\\n raise cls(e, response=response, explanation=explanation)\\ndocker.errors.APIError: 500 Server Error: Internal Server Error (\"Get \"https://10.73.0.181:4000/v2/\": http: server gave HTTP response to HTTPS client\")\\n'"} FAILED - RETRYING: common | Pull images (3 retries left).
Nevermind, figured it out $ docker pull quay.io/openstack.kolla/ubuntu-source-nova-ssh:yoga $ docker tag quay.io/openstack.kolla/ubuntu-source-nova-ssh:yoga " 10.73.0.181:4000"/openstack.kolla/ubuntu-source-nova-ssh $ docker push "10.73.0.181:4000"/openstack.kolla/ubuntu-source-nova-ssh I have a question related to the registry so let's say if I don't have a local registry in that case, does kolla always pull new images when running $ kolla-ansible -i multinode deploy ? Are there any best practices for local registry in production? like download images and push them to local-registry with specific tags to keep track or patch etc? On Sat, Jan 28, 2023 at 3:07 PM Satish Patel <satish.txt@gmail.com> wrote:
Great!! I don't know how I missed that part. It seems working but related to i got following error
10.73.0.181:4000/openstack.kolla/ubuntu-source-nova-ssh:yoga not found: manifest unknown
I have checked and found I don't have that image in the registry. How do I download that image from quay.io and push it to the local registry?
On Fri, Jan 27, 2023 at 11:59 PM Tobias McNulty <tobias@caktusgroup.com> wrote:
You might need to configure daemon.json as described here: https://docs.openstack.org/kolla-ansible/rocky/user/multinode.html#configure...
On Fri, Jan 27, 2023, 11:46 PM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I have set up a local registry using the following method as per official doc.
docker run -d \ --network host \ --name registry \ --restart=always \ -e REGISTRY_HTTP_ADDR=0.0.0.0:4000 \ -v registry:/var/lib/registry \ registry:2
After that I ran the following command to import all images to the local registry.
docker images | grep kolla | grep -v local | awk '{print $1,$2}' | while read -r image tag; do new_image_name=${image#"quay.io/"} docker tag ${image}:${tag} "localhost:4000"/${new_image_name}:${tag} docker push localhost:4000/${new_image_name}:${tag} done
Now i can see images in local registry using docker images command
(venv-kolla) root@kolla-infra-1:/etc/kolla# docker images | grep localhost localhost:4000/openstack.kolla/ubuntu-source-nova-novncproxy yoga 01dc100dc65a 2 days ago 1.2GB localhost:4000/openstack.kolla/ubuntu-source-horizon yoga a016e1f5b9af 2 days ago 1.1GB localhost:4000/openstack.kolla/ubuntu-source-nova-conductor yoga 381a64a368bd 2 days ago 1.11GB localhost:4000/openstack.kolla/ubuntu-source-nova-api yoga f90a15a09142 2 days ago 1.11GB .... ....
I have added it in global.yml
docker_registry: 10.73.0.181:4000 docker_registry_insecure: yes
Now when i am trying to add compute nodes then I get a registry error like the following. Not sure why its trying to use https://10.73.0.181:4000/v2 instead of http;//
TASK [common : include_tasks] **************************************************************************************************************************************************** included: /root/venv-kolla/share/kolla-ansible/ansible/roles/common/tasks/pull.yml for kolla-comp-2
TASK [service-images-pull : common | Pull images] ******************************************************************************************************************************** FAILED - RETRYING: common | Pull images (3 retries left). FAILED - RETRYING: common | Pull images (2 retries left). FAILED - RETRYING: common | Pull images (1 retries left). failed: [kolla-comp-2] (item=fluentd) => {"ansible_loop_var": "item", "attempts": 3, "changed": true, "item": {"key": "fluentd", "value": {"container_name": "fluentd", "dimensions": {}, "enabled": true, "environment": {"KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS"}, "group": "fluentd", "image": " 10.73.0.181:4000/openstack.kolla/ubuntu-source-fluentd:yoga", "volumes": ["/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro", "/etc/localtime:/etc/localtime:ro", "/etc/timezone:/etc/timezone:ro", "kolla_logs:/var/log/kolla/", "fluentd_data:/var/lib/fluentd/data/"]}}, "msg": "'Traceback (most recent call last):\\n File \"/usr/lib/python3/dist-packages/docker/api/client.py\", line 261, in _raise_for_status\\n response.raise_for_status()\\n File \"/usr/lib/python3/dist-packages/requests/models.py\", line 940, in raise_for_status\\n raise HTTPError(http_error_msg, response=self)\\nrequests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/images/create?tag=yoga&fromImage=10.73.0.181%3A4000%2Fopenstack.kolla%2Fubuntu-source-fluentd\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File \"/tmp/ansible_kolla_docker_payload_6o9nw9xd/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 381, in main\\n File \"/tmp/ansible_kolla_docker_payload_6o9nw9xd/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\", line 450, in pull_image\\n json.loads(line.strip().decode(\\'utf-8\\')) for line in self.dc.pull(\\n File \"/usr/lib/python3/dist-packages/docker/api/image.py\", line 415, in pull\\n self._raise_for_status(response)\\n File \"/usr/lib/python3/dist-packages/docker/api/client.py\", line 263, in _raise_for_status\\n raise create_api_error_from_http_exception(e)\\n File \"/usr/lib/python3/dist-packages/docker/errors.py\", line 31, in create_api_error_from_http_exception\\n raise cls(e, response=response, explanation=explanation)\\ndocker.errors.APIError: 500 Server Error: Internal Server Error (\"Get \"https://10.73.0.181:4000/v2/\": http: server gave HTTP response to HTTPS client\")\\n'"} FAILED - RETRYING: common | Pull images (3 retries left).
If the whole point is to download images into a local registry to speed up the process, you could just set up a proxy registry instead. This is how I do on my dev environment: # docker run -d \ # -p 5001:5000 \ # --restart=always \ # --name registry-quay \ # -e REGISTRY_PROXY_REMOTEURL=https://quay.io \ # -v /srv/quay:/var/lib/registry \ # registry:2 Notice port 5001 because I have another one set ready for docker.io as well. You can change it for your liking. For production, we are building our own images with the kolla project and pushing it to a local registry. That's the way to go if you are planning to patch things later on. Cheers On Sat, 28 Jan 2023 at 23:23, Satish Patel <satish.txt@gmail.com> wrote:
Nevermind, figured it out
$ docker pull quay.io/openstack.kolla/ubuntu-source-nova-ssh:yoga $ docker tag quay.io/openstack.kolla/ubuntu-source-nova-ssh:yoga "10.73.0.181:4000"/openstack.kolla/ubuntu-source-nova-ssh $ docker push "10.73.0.181:4000"/openstack.kolla/ubuntu-source-nova-ssh
I have a question related to the registry so let's say if I don't have a local registry in that case, does kolla always pull new images when running $ kolla-ansible -i multinode deploy ?
Are there any best practices for local registry in production? like download images and push them to local-registry with specific tags to keep track or patch etc?
On Sat, Jan 28, 2023 at 3:07 PM Satish Patel <satish.txt@gmail.com> wrote:
Great!! I don't know how I missed that part. It seems working but related to i got following error
10.73.0.181:4000/openstack.kolla/ubuntu-source-nova-ssh:yoga not found: manifest unknown
I have checked and found I don't have that image in the registry. How do I download that image from quay.io and push it to the local registry?
On Fri, Jan 27, 2023 at 11:59 PM Tobias McNulty <tobias@caktusgroup.com> wrote:
You might need to configure daemon.json as described here: https://docs.openstack.org/kolla-ansible/rocky/user/multinode.html#configure...
On Fri, Jan 27, 2023, 11:46 PM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I have set up a local registry using the following method as per official doc.
docker run -d \ --network host \ --name registry \ --restart=always \ -e REGISTRY_HTTP_ADDR=0.0.0.0:4000 \ -v registry:/var/lib/registry \ registry:2
After that I ran the following command to import all images to the local registry.
docker images | grep kolla | grep -v local | awk '{print $1,$2}' | while read -r image tag; do new_image_name=${image#"quay.io/"} docker tag ${image}:${tag} "localhost:4000"/${new_image_name}:${tag} docker push localhost:4000/${new_image_name}:${tag} done
Now i can see images in local registry using docker images command
(venv-kolla) root@kolla-infra-1:/etc/kolla# docker images | grep localhost localhost:4000/openstack.kolla/ubuntu-source-nova-novncproxy yoga 01dc100dc65a 2 days ago 1.2GB localhost:4000/openstack.kolla/ubuntu-source-horizon yoga a016e1f5b9af 2 days ago 1.1GB localhost:4000/openstack.kolla/ubuntu-source-nova-conductor yoga 381a64a368bd 2 days ago 1.11GB localhost:4000/openstack.kolla/ubuntu-source-nova-api yoga f90a15a09142 2 days ago 1.11GB .... ....
I have added it in global.yml
docker_registry: 10.73.0.181:4000 docker_registry_insecure: yes
Now when i am trying to add compute nodes then I get a registry error like the following. Not sure why its trying to use https://10.73.0.181:4000/v2 instead of http;//
TASK [common : include_tasks] **************************************************************************************************************************************************** included: /root/venv-kolla/share/kolla-ansible/ansible/roles/common/tasks/pull.yml for kolla-comp-2
TASK [service-images-pull : common | Pull images] ******************************************************************************************************************************** FAILED - RETRYING: common | Pull images (3 retries left). FAILED - RETRYING: common | Pull images (2 retries left). FAILED - RETRYING: common | Pull images (1 retries left). failed: [kolla-comp-2] (item=fluentd) => {"ansible_loop_var": "item", "attempts": 3, "changed": true, "item": {"key": "fluentd", "value": {"container_name": "fluentd", "dimensions": {}, "enabled": true, "environment": {"KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS"}, "group": "fluentd", "image": "10.73.0.181:4000/openstack.kolla/ubuntu-source-fluentd:yoga", "volumes": ["/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro", "/etc/localtime:/etc/localtime:ro", "/etc/timezone:/etc/timezone:ro", "kolla_logs:/var/log/kolla/", "fluentd_data:/var/lib/fluentd/data/"]}}, "msg": "'Traceback (most recent call last):\\n File \"/usr/lib/python3/dist-packages/docker/api/client.py\", line 261, in _raise_for_status\\n response.raise_for_status()\\n File \"/usr/lib/python3/dist-packages/requests/models.py\", line 940, in raise_for_status\\n raise HTTPError(http_error_msg, response=self)\\nrequests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/images/create?tag=yoga&fromImage=10.73.0.181%3A4000%2Fopenstack.kolla%2Fubuntu-source-fluentd\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File \"/tmp/ansible_kolla_docker_payload_6o9nw9xd/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 381, in main\\n File \"/tmp/ansible_kolla_docker_payload_6o9nw9xd/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\", line 450, in pull_image\\n json.loads(line.strip().decode(\\'utf-8\\')) for line in self.dc.pull(\\n File \"/usr/lib/python3/dist-packages/docker/api/image.py\", line 415, in pull\\n self._raise_for_status(response)\\n File \"/usr/lib/python3/dist-packages/docker/api/client.py\", line 263, in _raise_for_status\\n raise create_api_error_from_http_exception(e)\\n File \"/usr/lib/python3/dist-packages/docker/errors.py\", line 31, in create_api_error_from_http_exception\\n raise cls(e, response=response, explanation=explanation)\\ndocker.errors.APIError: 500 Server Error: Internal Server Error (\"Get \"https://10.73.0.181:4000/v2/\": http: server gave HTTP response to HTTPS client\")\\n'"} FAILED - RETRYING: common | Pull images (3 retries left).
Also forgot to mention that you shouldn't have to configure insecure proxies manually. If I am not mistaken, bootstrap-servers should do the trick if globals.yml is set properly. On Thu, 2 Feb 2023 at 13:33, Can Özyurt <acozyurt@gmail.com> wrote:
If the whole point is to download images into a local registry to speed up the process, you could just set up a proxy registry instead. This is how I do on my dev environment:
# docker run -d \ # -p 5001:5000 \ # --restart=always \ # --name registry-quay \ # -e REGISTRY_PROXY_REMOTEURL=https://quay.io \ # -v /srv/quay:/var/lib/registry \ # registry:2
Notice port 5001 because I have another one set ready for docker.io as well. You can change it for your liking.
For production, we are building our own images with the kolla project and pushing it to a local registry. That's the way to go if you are planning to patch things later on.
Cheers
On Sat, 28 Jan 2023 at 23:23, Satish Patel <satish.txt@gmail.com> wrote:
Nevermind, figured it out
$ docker pull quay.io/openstack.kolla/ubuntu-source-nova-ssh:yoga $ docker tag quay.io/openstack.kolla/ubuntu-source-nova-ssh:yoga "10.73.0.181:4000"/openstack.kolla/ubuntu-source-nova-ssh $ docker push "10.73.0.181:4000"/openstack.kolla/ubuntu-source-nova-ssh
I have a question related to the registry so let's say if I don't have a local registry in that case, does kolla always pull new images when running $ kolla-ansible -i multinode deploy ?
Are there any best practices for local registry in production? like download images and push them to local-registry with specific tags to keep track or patch etc?
On Sat, Jan 28, 2023 at 3:07 PM Satish Patel <satish.txt@gmail.com> wrote:
Great!! I don't know how I missed that part. It seems working but related to i got following error
10.73.0.181:4000/openstack.kolla/ubuntu-source-nova-ssh:yoga not found: manifest unknown
I have checked and found I don't have that image in the registry. How do I download that image from quay.io and push it to the local registry?
On Fri, Jan 27, 2023 at 11:59 PM Tobias McNulty <tobias@caktusgroup.com> wrote:
You might need to configure daemon.json as described here: https://docs.openstack.org/kolla-ansible/rocky/user/multinode.html#configure...
On Fri, Jan 27, 2023, 11:46 PM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I have set up a local registry using the following method as per official doc.
docker run -d \ --network host \ --name registry \ --restart=always \ -e REGISTRY_HTTP_ADDR=0.0.0.0:4000 \ -v registry:/var/lib/registry \ registry:2
After that I ran the following command to import all images to the local registry.
docker images | grep kolla | grep -v local | awk '{print $1,$2}' | while read -r image tag; do new_image_name=${image#"quay.io/"} docker tag ${image}:${tag} "localhost:4000"/${new_image_name}:${tag} docker push localhost:4000/${new_image_name}:${tag} done
Now i can see images in local registry using docker images command
(venv-kolla) root@kolla-infra-1:/etc/kolla# docker images | grep localhost localhost:4000/openstack.kolla/ubuntu-source-nova-novncproxy yoga 01dc100dc65a 2 days ago 1.2GB localhost:4000/openstack.kolla/ubuntu-source-horizon yoga a016e1f5b9af 2 days ago 1.1GB localhost:4000/openstack.kolla/ubuntu-source-nova-conductor yoga 381a64a368bd 2 days ago 1.11GB localhost:4000/openstack.kolla/ubuntu-source-nova-api yoga f90a15a09142 2 days ago 1.11GB .... ....
I have added it in global.yml
docker_registry: 10.73.0.181:4000 docker_registry_insecure: yes
Now when i am trying to add compute nodes then I get a registry error like the following. Not sure why its trying to use https://10.73.0.181:4000/v2 instead of http;//
TASK [common : include_tasks] **************************************************************************************************************************************************** included: /root/venv-kolla/share/kolla-ansible/ansible/roles/common/tasks/pull.yml for kolla-comp-2
TASK [service-images-pull : common | Pull images] ******************************************************************************************************************************** FAILED - RETRYING: common | Pull images (3 retries left). FAILED - RETRYING: common | Pull images (2 retries left). FAILED - RETRYING: common | Pull images (1 retries left). failed: [kolla-comp-2] (item=fluentd) => {"ansible_loop_var": "item", "attempts": 3, "changed": true, "item": {"key": "fluentd", "value": {"container_name": "fluentd", "dimensions": {}, "enabled": true, "environment": {"KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS"}, "group": "fluentd", "image": "10.73.0.181:4000/openstack.kolla/ubuntu-source-fluentd:yoga", "volumes": ["/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro", "/etc/localtime:/etc/localtime:ro", "/etc/timezone:/etc/timezone:ro", "kolla_logs:/var/log/kolla/", "fluentd_data:/var/lib/fluentd/data/"]}}, "msg": "'Traceback (most recent call last):\\n File \"/usr/lib/python3/dist-packages/docker/api/client.py\", line 261, in _raise_for_status\\n response.raise_for_status()\\n File \"/usr/lib/python3/dist-packages/requests/models.py\", line 940, in raise_for_status\\n raise HTTPError(http_error_msg, response=self)\\nrequests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/images/create?tag=yoga&fromImage=10.73.0.181%3A4000%2Fopenstack.kolla%2Fubuntu-source-fluentd\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File \"/tmp/ansible_kolla_docker_payload_6o9nw9xd/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 381, in main\\n File \"/tmp/ansible_kolla_docker_payload_6o9nw9xd/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\", line 450, in pull_image\\n json.loads(line.strip().decode(\\'utf-8\\')) for line in self.dc.pull(\\n File \"/usr/lib/python3/dist-packages/docker/api/image.py\", line 415, in pull\\n self._raise_for_status(response)\\n File \"/usr/lib/python3/dist-packages/docker/api/client.py\", line 263, in _raise_for_status\\n raise create_api_error_from_http_exception(e)\\n File \"/usr/lib/python3/dist-packages/docker/errors.py\", line 31, in create_api_error_from_http_exception\\n raise cls(e, response=response, explanation=explanation)\\ndocker.errors.APIError: 500 Server Error: Internal Server Error (\"Get \"https://10.73.0.181:4000/v2/\": http: server gave HTTP response to HTTPS client\")\\n'"} FAILED - RETRYING: common | Pull images (3 retries left).
participants (3)
-
Can Özyurt
-
Satish Patel
-
Tobias McNulty