Hi Herve,
Thanks for noticing this. The pull rate limit refreshes every 6 hours, is specific to the IP used by the job, and in the case of the build/publish jobs we only require pulling a single image - the base OS image - per job. I suggest we reenqueue the failing jobs.
For build/publish jobs we have discussed using the infra Docker registry mirrors, which should avoid hitting Dockerhub too often.
Cheers, Mark
On Fri, 8 Jan 2021 at 08:28, Herve Beraud hberaud@redhat.com wrote:
Hello,
FYI it seems that kolla met docker's new limitation during its release jobs, especially with your publish jobs, I saw that you (the kolla team) already discussed this limitation [1] on the ML.
2021-01-07 17:21:03.396355 | primary | ERROR:kolla.common.utils.base:Error'd with the following message 2021-01-07 17:21:03.396465 | primary | ERROR:kolla.common.utils.base:toomanyrequests: You have reached your pull rate limit. You may increaseThree jobs here failed for the same reason.
I don't think that reenqueue the failing jobs without a specific action to manage this limitation will help us here.
Let us know if we can help us in some manner.
[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019148....
Le jeu. 7 janv. 2021 à 20:41, zuul@openstack.org a écrit :
Build failed.
- openstack-upload-github-mirror
https://zuul.opendev.org/t/openstack/build/a64b4c4bc398481f9afffa2ad465a012 : SUCCESS in 1m 03s
- release-openstack-python
https://zuul.opendev.org/t/openstack/build/2ed64dfcbcaf483abf003cfdf4f25837 : SUCCESS in 3m 11s
- announce-release
https://zuul.opendev.org/t/openstack/build/84d803d496a2485c983f6233dafcfd71 : SUCCESS in 4m 07s
- propose-update-constraints
https://zuul.opendev.org/t/openstack/build/d0cac6a077054bc8ba9eb92e56c21799 : SUCCESS in 4m 15s
- kolla-publish-centos-source
https://zuul.opendev.org/t/openstack/build/5e8f6aa2f56940a7be77a8dfe1c8ecc6 : SUCCESS in 2h 20m 47s
- kolla-publish-centos-binary
https://zuul.opendev.org/t/openstack/build/8a8a021d9f9c4ca79755b06309710cc7 : SUCCESS in 1h 56m 32s (non-voting)
- kolla-publish-centos8-source
https://zuul.opendev.org/t/openstack/build/fb6891d3f5e4493b880fce263a92e086 : SUCCESS in 1h 50m 57s
- kolla-publish-centos8-binary
https://zuul.opendev.org/t/openstack/build/c312c05e5d084fdbb3f372755221f186 : SUCCESS in 1h 13m 12s (non-voting)
- kolla-publish-debian-source
https://zuul.opendev.org/t/openstack/build/e24c12751b8c4aba881adb6c9ae8dc07 : SUCCESS in 1h 27m 27s (non-voting)
- kolla-publish-debian-source-aarch64
https://zuul.opendev.org/t/openstack/build/1ff3b02df53847d0aa54bf12ea7fa666 : FAILURE in 1h 51m 49s (non-voting)
- kolla-publish-debian-binary
https://zuul.opendev.org/t/openstack/build/012c2de475fe45ea83f1cd8a7420aa6d : SUCCESS in 1h 15m 15s (non-voting)
- kolla-publish-ubuntu-source
https://zuul.opendev.org/t/openstack/build/88ca21d972514cce954ecb586324fa29 : FAILURE in 4m 14s
- kolla-publish-ubuntu-binary
https://zuul.opendev.org/t/openstack/build/f3d44e4d1b6c4e9799161b156290b238 : FAILURE in 4m 19s (non-voting)
Release-job-failures mailing list Release-job-failures@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/release-job-failures
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----