We ran into a problem while deploying magnum when another project exhausted the dockerhub limit on anonymous pulls from a few of the kube-system pods that are deployed. Namely: daemonset.apps/k8s-keystone-auth daemonset.apps/openstack-cloud-controller-manager deployment.apps/kubernetes-dashboard deployment.apps/dashboard-metrics-scraper
This would fail with an error noting that dockerhub was blocking the request as too many pulls had happened. We could get around this by adding in a secret with a docker login, and editing those deployments and daemonsets to use that credential.
It would appear the container_infra_prefix label can be modified to point to a different registry. Though this would mean we would have to clone all of the images, including images that are from registries other than dockerhub. Leading me to wonder if there isn't an existing registry that one can use using magnum, on quay.io or some host that isn't limiting pulls?
Alternatively, is it possible that the dockerhub images that do not pull (some do, coredns for instance does, I suspect it is due to it having "Sponsored OSS" status on dockerhub) without limits could be hosted elsewhere? Or perhaps already are and the default that magnum sets to pull could be updated to those?
Alternatively, alternatively, I haven't found an option for giving a dockerhub user/pass to magnum in the documentation, and looking at the code it doesn't appear that there is a variable for one, so I suspect it is not there. Could such an option be added?
Thank you
Hi Vivian,
We’ve found that running deployment local container registry speeds up the deployment and makes it more reliable too!
Thanks Mohammed
On Thu, Feb 23, 2023 at 7:35 PM Vivian Rook vrook@wikimedia.org wrote:
We ran into a problem while deploying magnum when another project exhausted the dockerhub limit on anonymous pulls from a few of the kube-system pods that are deployed. Namely: daemonset.apps/k8s-keystone-auth daemonset.apps/openstack-cloud-controller-manager deployment.apps/kubernetes-dashboard deployment.apps/dashboard-metrics-scraper
This would fail with an error noting that dockerhub was blocking the request as too many pulls had happened. We could get around this by adding in a secret with a docker login, and editing those deployments and daemonsets to use that credential.
It would appear the container_infra_prefix label can be modified to point to a different registry. Though this would mean we would have to clone all of the images, including images that are from registries other than dockerhub. Leading me to wonder if there isn't an existing registry that one can use using magnum, on quay.io or some host that isn't limiting pulls?
Alternatively, is it possible that the dockerhub images that do not pull (some do, coredns for instance does, I suspect it is due to it having "Sponsored OSS" status on dockerhub) without limits could be hosted elsewhere? Or perhaps already are and the default that magnum sets to pull could be updated to those?
Alternatively, alternatively, I haven't found an option for giving a dockerhub user/pass to magnum in the documentation, and looking at the code it doesn't appear that there is a variable for one, so I suspect it is not there. Could such an option be added?
Thank you
--
*Vivian Rook (They/Them)* Site Reliability Engineer Wikimedia Foundation https://wikimediafoundation.org/
Echo'ing Mohammed's comment, we will always deploy a local container reg to help speed things up and to keep things consistent during a deployment or upgrade.
Cheers Michael
On Thu, Feb 23, 2023 at 7:46 PM Mohammed Naser mnaser@vexxhost.com wrote:
Hi Vivian,
We’ve found that running deployment local container registry speeds up the deployment and makes it more reliable too!
Thanks Mohammed
On Thu, Feb 23, 2023 at 7:35 PM Vivian Rook vrook@wikimedia.org wrote:
We ran into a problem while deploying magnum when another project exhausted the dockerhub limit on anonymous pulls from a few of the kube-system pods that are deployed. Namely: daemonset.apps/k8s-keystone-auth daemonset.apps/openstack-cloud-controller-manager deployment.apps/kubernetes-dashboard deployment.apps/dashboard-metrics-scraper
This would fail with an error noting that dockerhub was blocking the request as too many pulls had happened. We could get around this by adding in a secret with a docker login, and editing those deployments and daemonsets to use that credential.
It would appear the container_infra_prefix label can be modified to point to a different registry. Though this would mean we would have to clone all of the images, including images that are from registries other than dockerhub. Leading me to wonder if there isn't an existing registry that one can use using magnum, on quay.io or some host that isn't limiting pulls?
Alternatively, is it possible that the dockerhub images that do not pull (some do, coredns for instance does, I suspect it is due to it having "Sponsored OSS" status on dockerhub) without limits could be hosted elsewhere? Or perhaps already are and the default that magnum sets to pull could be updated to those?
Alternatively, alternatively, I haven't found an option for giving a dockerhub user/pass to magnum in the documentation, and looking at the code it doesn't appear that there is a variable for one, so I suspect it is not there. Could such an option be added?
Thank you
--
*Vivian Rook (They/Them)* Site Reliability Engineer Wikimedia Foundation https://wikimediafoundation.org/
-- Mohammed Naser VEXXHOST, Inc.
participants (4)
-
Michael Knox
-
Mohammed Naser
-
Oliver Weinmann
-
Vivian Rook