Also notice that yesterday we met an AFS issue [1] during the merge of the following patches:

- https://review.opendev.org/c/openstack/releases/+/769325 - https://review.opendev.org/c/openstack/releases/+/769322 - https://review.opendev.org/c/openstack/releases/+/769324

The problem was that afs an server got stuck in a pathological way from 05:50 utc today until 16:10 utc when we hard rebooted the server instance.

The consequence of this is that the related tarballs haven't been published:

- https://tarballs.opendev.org/openstack/kolla/?C=M;O=D - https://tarballs.opendev.org/openstack/kolla-ansible/?C=M;O=D - https://tarballs.opendev.org/openstack/kayobe/?C=M;O=D

And so the RDO CI fail to build them for train, ussuri and victoria:

- https://review.rdoproject.org/r/#/c/31499/ - https://review.rdoproject.org/r/#/c/31498/ - https://review.rdoproject.org/r/#/c/31497/

So I think we need to reenqueue these jobs too.

Thanks for reading

[1] http://eavesdrop.openstack.org/irclogs/%23openstack-release/%23openstack-rel...

Le ven. 8 janv. 2021 à 09:45, Mark Goddard mark@stackhpc.com a écrit :

Hi Herve,

Thanks for noticing this. The pull rate limit refreshes every 6 hours, is specific to the IP used by the job, and in the case of the build/publish jobs we only require pulling a single image - the base OS image - per job. I suggest we reenqueue the failing jobs.

For build/publish jobs we have discussed using the infra Docker registry mirrors, which should avoid hitting Dockerhub too often.

Cheers, Mark

On Fri, 8 Jan 2021 at 08:28, Herve Beraud hberaud@redhat.com wrote:

...

Hello,

FYI it seems that kolla met docker's new limitation during its release jobs, especially with your publish jobs, I saw that you (the kolla team) already discussed this limitation [1] on the ML.

2021-01-07 17:21:03.396355 | primary | ERROR:kolla.common.utils.base:Error'd with the following message

2021-01-07 17:21:03.396465 | primary | ERROR:kolla.common.utils.base:toomanyrequests: You have reached your pull rate limit. You may increase

Three jobs here failed for the same reason.

I don't think that reenqueue the failing jobs without a specific action to manage this limitation will help us here.

Let us know if we can help us in some manner.

[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019148....

Le jeu. 7 janv. 2021 à 20:41, zuul@openstack.org a écrit :

...

Build failed.

• openstack-upload-github-mirror

https://zuul.opendev.org/t/openstack/build/a64b4c4bc398481f9afffa2ad465a012 : SUCCESS in 1m 03s

• release-openstack-python

https://zuul.opendev.org/t/openstack/build/2ed64dfcbcaf483abf003cfdf4f25837 : SUCCESS in 3m 11s

• announce-release

https://zuul.opendev.org/t/openstack/build/84d803d496a2485c983f6233dafcfd71 : SUCCESS in 4m 07s

• propose-update-constraints

https://zuul.opendev.org/t/openstack/build/d0cac6a077054bc8ba9eb92e56c21799 : SUCCESS in 4m 15s

• kolla-publish-centos-source

https://zuul.opendev.org/t/openstack/build/5e8f6aa2f56940a7be77a8dfe1c8ecc6 : SUCCESS in 2h 20m 47s

• kolla-publish-centos-binary

https://zuul.opendev.org/t/openstack/build/8a8a021d9f9c4ca79755b06309710cc7 : SUCCESS in 1h 56m 32s (non-voting)

• kolla-publish-centos8-source

https://zuul.opendev.org/t/openstack/build/fb6891d3f5e4493b880fce263a92e086 : SUCCESS in 1h 50m 57s

• kolla-publish-centos8-binary

https://zuul.opendev.org/t/openstack/build/c312c05e5d084fdbb3f372755221f186 : SUCCESS in 1h 13m 12s (non-voting)

• kolla-publish-debian-source

https://zuul.opendev.org/t/openstack/build/e24c12751b8c4aba881adb6c9ae8dc07 : SUCCESS in 1h 27m 27s (non-voting)

• kolla-publish-debian-source-aarch64

https://zuul.opendev.org/t/openstack/build/1ff3b02df53847d0aa54bf12ea7fa666 : FAILURE in 1h 51m 49s (non-voting)

• kolla-publish-debian-binary

https://zuul.opendev.org/t/openstack/build/012c2de475fe45ea83f1cd8a7420aa6d : SUCCESS in 1h 15m 15s (non-voting)

• kolla-publish-ubuntu-source

https://zuul.opendev.org/t/openstack/build/88ca21d972514cce954ecb586324fa29 : FAILURE in 4m 14s

• kolla-publish-ubuntu-binary

https://zuul.opendev.org/t/openstack/build/f3d44e4d1b6c4e9799161b156290b238 : FAILURE in 4m 19s (non-voting)

---

Release-job-failures mailing list Release-job-failures@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/release-job-failures

-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE-----

wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----