Hi Herve,

Thanks for noticing this. The pull rate limit refreshes every 6 hours, is specific to the IP used by the job, and in the case of the build/publish jobs we only require pulling a single image - the base OS image - per job. I suggest we reenqueue the failing jobs.

For build/publish jobs we have discussed using the infra Docker registry mirrors, which should avoid hitting Dockerhub too often.

Cheers,
Mark

On Fri, 8 Jan 2021 at 08:28, Herve Beraud <hberaud@redhat.com> wrote:
Hello,

FYI it seems that kolla met docker's new limitation during its release jobs, especially with your publish jobs, I saw that you (the kolla team) already discussed this limitation [1] on the ML.

```
2021-01-07 17:21:03.396355 | primary | ERROR:kolla.common.utils.base:Error'd with the following message

2021-01-07 17:21:03.396465 | primary | ERROR:kolla.common.utils.base:toomanyrequests: You have reached your pull rate limit. You may increase 
```

Three jobs here failed for the same reason.
I don't think that reenqueue the failing jobs without a specific action to manage this limitation will help us here.
Let us know if we can help us in some manner.

[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019148.html


Le jeu. 7 janv. 2021 à 20:41, <zuul@openstack.org> a écrit :
Build failed.

- openstack-upload-github-mirror https://zuul.opendev.org/t/openstack/build/a64b4c4bc398481f9afffa2ad465a012 : SUCCESS in 1m 03s
- release-openstack-python https://zuul.opendev.org/t/openstack/build/2ed64dfcbcaf483abf003cfdf4f25837 : SUCCESS in 3m 11s
- announce-release https://zuul.opendev.org/t/openstack/build/84d803d496a2485c983f6233dafcfd71 : SUCCESS in 4m 07s
- propose-update-constraints https://zuul.opendev.org/t/openstack/build/d0cac6a077054bc8ba9eb92e56c21799 : SUCCESS in 4m 15s
- kolla-publish-centos-source https://zuul.opendev.org/t/openstack/build/5e8f6aa2f56940a7be77a8dfe1c8ecc6 : SUCCESS in 2h 20m 47s
- kolla-publish-centos-binary https://zuul.opendev.org/t/openstack/build/8a8a021d9f9c4ca79755b06309710cc7 : SUCCESS in 1h 56m 32s (non-voting)
- kolla-publish-centos8-source https://zuul.opendev.org/t/openstack/build/fb6891d3f5e4493b880fce263a92e086 : SUCCESS in 1h 50m 57s
- kolla-publish-centos8-binary https://zuul.opendev.org/t/openstack/build/c312c05e5d084fdbb3f372755221f186 : SUCCESS in 1h 13m 12s (non-voting)
- kolla-publish-debian-source https://zuul.opendev.org/t/openstack/build/e24c12751b8c4aba881adb6c9ae8dc07 : SUCCESS in 1h 27m 27s (non-voting)
- kolla-publish-debian-source-aarch64 https://zuul.opendev.org/t/openstack/build/1ff3b02df53847d0aa54bf12ea7fa666 : FAILURE in 1h 51m 49s (non-voting)
- kolla-publish-debian-binary https://zuul.opendev.org/t/openstack/build/012c2de475fe45ea83f1cd8a7420aa6d : SUCCESS in 1h 15m 15s (non-voting)
- kolla-publish-ubuntu-source https://zuul.opendev.org/t/openstack/build/88ca21d972514cce954ecb586324fa29 : FAILURE in 4m 14s
- kolla-publish-ubuntu-binary https://zuul.opendev.org/t/openstack/build/f3d44e4d1b6c4e9799161b156290b238 : FAILURE in 4m 19s (non-voting)

_______________________________________________
Release-job-failures mailing list
Release-job-failures@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/release-job-failures


--
Hervé Beraud
Senior Software Engineer at Red Hat
irc: hberaud
-----BEGIN PGP SIGNATURE-----

wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+
Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+
RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP
F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G
5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g
glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw
m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ
hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0
qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y
F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3
B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O
v6rDpkeNksZ9fFSyoY2o
=ECSj
-----END PGP SIGNATURE-----