Re: [Release-job-failures] Release of openstack/kolla for ref refs/tags/9.3.0 failed
Hello, FYI it seems that kolla met docker's new limitation during its release jobs, especially with your publish jobs, I saw that you (the kolla team) already discussed this limitation [1] on the ML. ``` 2021-01-07 17:21:03.396355 | primary | ERROR:kolla.common.utils.base:Error'd with the following message 2021-01-07 17:21:03.396465 | primary | ERROR:kolla.common.utils.base:toomanyrequests: You have reached your pull rate limit. You may increase ``` Three jobs here failed for the same reason. I don't think that reenqueue the failing jobs without a specific action to manage this limitation will help us here. Let us know if we can help us in some manner. [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019148.... Le jeu. 7 janv. 2021 à 20:41, <zuul@openstack.org> a écrit :
Build failed.
- openstack-upload-github-mirror https://zuul.opendev.org/t/openstack/build/a64b4c4bc398481f9afffa2ad465a012 : SUCCESS in 1m 03s - release-openstack-python https://zuul.opendev.org/t/openstack/build/2ed64dfcbcaf483abf003cfdf4f25837 : SUCCESS in 3m 11s - announce-release https://zuul.opendev.org/t/openstack/build/84d803d496a2485c983f6233dafcfd71 : SUCCESS in 4m 07s - propose-update-constraints https://zuul.opendev.org/t/openstack/build/d0cac6a077054bc8ba9eb92e56c21799 : SUCCESS in 4m 15s - kolla-publish-centos-source https://zuul.opendev.org/t/openstack/build/5e8f6aa2f56940a7be77a8dfe1c8ecc6 : SUCCESS in 2h 20m 47s - kolla-publish-centos-binary https://zuul.opendev.org/t/openstack/build/8a8a021d9f9c4ca79755b06309710cc7 : SUCCESS in 1h 56m 32s (non-voting) - kolla-publish-centos8-source https://zuul.opendev.org/t/openstack/build/fb6891d3f5e4493b880fce263a92e086 : SUCCESS in 1h 50m 57s - kolla-publish-centos8-binary https://zuul.opendev.org/t/openstack/build/c312c05e5d084fdbb3f372755221f186 : SUCCESS in 1h 13m 12s (non-voting) - kolla-publish-debian-source https://zuul.opendev.org/t/openstack/build/e24c12751b8c4aba881adb6c9ae8dc07 : SUCCESS in 1h 27m 27s (non-voting) - kolla-publish-debian-source-aarch64 https://zuul.opendev.org/t/openstack/build/1ff3b02df53847d0aa54bf12ea7fa666 : FAILURE in 1h 51m 49s (non-voting) - kolla-publish-debian-binary https://zuul.opendev.org/t/openstack/build/012c2de475fe45ea83f1cd8a7420aa6d : SUCCESS in 1h 15m 15s (non-voting) - kolla-publish-ubuntu-source https://zuul.opendev.org/t/openstack/build/88ca21d972514cce954ecb586324fa29 : FAILURE in 4m 14s - kolla-publish-ubuntu-binary https://zuul.opendev.org/t/openstack/build/f3d44e4d1b6c4e9799161b156290b238 : FAILURE in 4m 19s (non-voting)
_______________________________________________ Release-job-failures mailing list Release-job-failures@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/release-job-failures
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
Hi Herve, Thanks for noticing this. The pull rate limit refreshes every 6 hours, is specific to the IP used by the job, and in the case of the build/publish jobs we only require pulling a single image - the base OS image - per job. I suggest we reenqueue the failing jobs. For build/publish jobs we have discussed using the infra Docker registry mirrors, which should avoid hitting Dockerhub too often. Cheers, Mark On Fri, 8 Jan 2021 at 08:28, Herve Beraud <hberaud@redhat.com> wrote:
Hello,
FYI it seems that kolla met docker's new limitation during its release jobs, especially with your publish jobs, I saw that you (the kolla team) already discussed this limitation [1] on the ML.
```
2021-01-07 17:21:03.396355 | primary | ERROR:kolla.common.utils.base:Error'd with the following message
2021-01-07 17:21:03.396465 | primary | ERROR:kolla.common.utils.base:toomanyrequests: You have reached your pull rate limit. You may increase ```
Three jobs here failed for the same reason.
I don't think that reenqueue the failing jobs without a specific action to manage this limitation will help us here.
Let us know if we can help us in some manner.
[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019148....
Le jeu. 7 janv. 2021 à 20:41, <zuul@openstack.org> a écrit :
Build failed.
- openstack-upload-github-mirror https://zuul.opendev.org/t/openstack/build/a64b4c4bc398481f9afffa2ad465a012 : SUCCESS in 1m 03s - release-openstack-python https://zuul.opendev.org/t/openstack/build/2ed64dfcbcaf483abf003cfdf4f25837 : SUCCESS in 3m 11s - announce-release https://zuul.opendev.org/t/openstack/build/84d803d496a2485c983f6233dafcfd71 : SUCCESS in 4m 07s - propose-update-constraints https://zuul.opendev.org/t/openstack/build/d0cac6a077054bc8ba9eb92e56c21799 : SUCCESS in 4m 15s - kolla-publish-centos-source https://zuul.opendev.org/t/openstack/build/5e8f6aa2f56940a7be77a8dfe1c8ecc6 : SUCCESS in 2h 20m 47s - kolla-publish-centos-binary https://zuul.opendev.org/t/openstack/build/8a8a021d9f9c4ca79755b06309710cc7 : SUCCESS in 1h 56m 32s (non-voting) - kolla-publish-centos8-source https://zuul.opendev.org/t/openstack/build/fb6891d3f5e4493b880fce263a92e086 : SUCCESS in 1h 50m 57s - kolla-publish-centos8-binary https://zuul.opendev.org/t/openstack/build/c312c05e5d084fdbb3f372755221f186 : SUCCESS in 1h 13m 12s (non-voting) - kolla-publish-debian-source https://zuul.opendev.org/t/openstack/build/e24c12751b8c4aba881adb6c9ae8dc07 : SUCCESS in 1h 27m 27s (non-voting) - kolla-publish-debian-source-aarch64 https://zuul.opendev.org/t/openstack/build/1ff3b02df53847d0aa54bf12ea7fa666 : FAILURE in 1h 51m 49s (non-voting) - kolla-publish-debian-binary https://zuul.opendev.org/t/openstack/build/012c2de475fe45ea83f1cd8a7420aa6d : SUCCESS in 1h 15m 15s (non-voting) - kolla-publish-ubuntu-source https://zuul.opendev.org/t/openstack/build/88ca21d972514cce954ecb586324fa29 : FAILURE in 4m 14s - kolla-publish-ubuntu-binary https://zuul.opendev.org/t/openstack/build/f3d44e4d1b6c4e9799161b156290b238 : FAILURE in 4m 19s (non-voting)
_______________________________________________ Release-job-failures mailing list Release-job-failures@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/release-job-failures
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
Also notice that yesterday we met an AFS issue [1] during the merge of the following patches: - https://review.opendev.org/c/openstack/releases/+/769325 - https://review.opendev.org/c/openstack/releases/+/769322 - https://review.opendev.org/c/openstack/releases/+/769324 The problem was that afs an server got stuck in a pathological way from 05:50 utc today until 16:10 utc when we hard rebooted the server instance. The consequence of this is that the related tarballs haven't been published: - https://tarballs.opendev.org/openstack/kolla/?C=M;O=D - https://tarballs.opendev.org/openstack/kolla-ansible/?C=M;O=D - https://tarballs.opendev.org/openstack/kayobe/?C=M;O=D And so the RDO CI fail to build them for train, ussuri and victoria: - https://review.rdoproject.org/r/#/c/31499/ - https://review.rdoproject.org/r/#/c/31498/ - https://review.rdoproject.org/r/#/c/31497/ So I think we need to reenqueue these jobs too. Thanks for reading [1] http://eavesdrop.openstack.org/irclogs/%23openstack-release/%23openstack-rel... Le ven. 8 janv. 2021 à 09:45, Mark Goddard <mark@stackhpc.com> a écrit :
Hi Herve,
Thanks for noticing this. The pull rate limit refreshes every 6 hours, is specific to the IP used by the job, and in the case of the build/publish jobs we only require pulling a single image - the base OS image - per job. I suggest we reenqueue the failing jobs.
For build/publish jobs we have discussed using the infra Docker registry mirrors, which should avoid hitting Dockerhub too often.
Cheers, Mark
On Fri, 8 Jan 2021 at 08:28, Herve Beraud <hberaud@redhat.com> wrote:
Hello,
FYI it seems that kolla met docker's new limitation during its release jobs, especially with your publish jobs, I saw that you (the kolla team) already discussed this limitation [1] on the ML.
```
2021-01-07 17:21:03.396355 | primary | ERROR:kolla.common.utils.base:Error'd with the following message
2021-01-07 17:21:03.396465 | primary | ERROR:kolla.common.utils.base:toomanyrequests: You have reached your pull rate limit. You may increase ```
Three jobs here failed for the same reason.
I don't think that reenqueue the failing jobs without a specific action to manage this limitation will help us here.
Let us know if we can help us in some manner.
[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019148....
Le jeu. 7 janv. 2021 à 20:41, <zuul@openstack.org> a écrit :
Build failed.
- openstack-upload-github-mirror https://zuul.opendev.org/t/openstack/build/a64b4c4bc398481f9afffa2ad465a012 : SUCCESS in 1m 03s - release-openstack-python https://zuul.opendev.org/t/openstack/build/2ed64dfcbcaf483abf003cfdf4f25837 : SUCCESS in 3m 11s - announce-release https://zuul.opendev.org/t/openstack/build/84d803d496a2485c983f6233dafcfd71 : SUCCESS in 4m 07s - propose-update-constraints https://zuul.opendev.org/t/openstack/build/d0cac6a077054bc8ba9eb92e56c21799 : SUCCESS in 4m 15s - kolla-publish-centos-source https://zuul.opendev.org/t/openstack/build/5e8f6aa2f56940a7be77a8dfe1c8ecc6 : SUCCESS in 2h 20m 47s - kolla-publish-centos-binary https://zuul.opendev.org/t/openstack/build/8a8a021d9f9c4ca79755b06309710cc7 : SUCCESS in 1h 56m 32s (non-voting) - kolla-publish-centos8-source https://zuul.opendev.org/t/openstack/build/fb6891d3f5e4493b880fce263a92e086 : SUCCESS in 1h 50m 57s - kolla-publish-centos8-binary https://zuul.opendev.org/t/openstack/build/c312c05e5d084fdbb3f372755221f186 : SUCCESS in 1h 13m 12s (non-voting) - kolla-publish-debian-source https://zuul.opendev.org/t/openstack/build/e24c12751b8c4aba881adb6c9ae8dc07 : SUCCESS in 1h 27m 27s (non-voting) - kolla-publish-debian-source-aarch64 https://zuul.opendev.org/t/openstack/build/1ff3b02df53847d0aa54bf12ea7fa666 : FAILURE in 1h 51m 49s (non-voting) - kolla-publish-debian-binary https://zuul.opendev.org/t/openstack/build/012c2de475fe45ea83f1cd8a7420aa6d : SUCCESS in 1h 15m 15s (non-voting) - kolla-publish-ubuntu-source https://zuul.opendev.org/t/openstack/build/88ca21d972514cce954ecb586324fa29 : FAILURE in 4m 14s - kolla-publish-ubuntu-binary https://zuul.opendev.org/t/openstack/build/f3d44e4d1b6c4e9799161b156290b238 : FAILURE in 4m 19s (non-voting)
_______________________________________________ Release-job-failures mailing list Release-job-failures@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/release-job-failures
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
On 2021-01-08 14:42:14 +0100 (+0100), Herve Beraud wrote:
Also notice that yesterday we met an AFS issue [1] during the merge of the following patches:
- https://review.opendev.org/c/openstack/releases/+/769325 - https://review.opendev.org/c/openstack/releases/+/769322 - https://review.opendev.org/c/openstack/releases/+/769324
The problem was that afs an server got stuck in a pathological way from 05:50 utc today until 16:10 utc when we hard rebooted the server instance.
The patches you reference merged later, so wasn't directly impacted by write failures (the jobs actually succeeded).
The consequence of this is that the related tarballs haven't been published:
- https://tarballs.opendev.org/openstack/kolla/?C=M;O=D - https://tarballs.opendev.org/openstack/kolla-ansible/?C=M;O=D - https://tarballs.opendev.org/openstack/kayobe/?C=M;O=D
And so the RDO CI fail to build them for train, ussuri and victoria:
- https://review.rdoproject.org/r/#/c/31499/ - https://review.rdoproject.org/r/#/c/31498/ - https://review.rdoproject.org/r/#/c/31497/
So I think we need to reenqueue these jobs too. [...]
The files were written into the tarballs volume just fine, but the read-only replicas which back the tarballs.opendev.org site hadn't been synchronized. I found a stuck process (waiting since yesterday for a response from the server which had previously died) and killed it to get the periodic synchronization of the read-only volumes working again so, the site is no longer stale and has those releases on it now as of 15:15:40 UTC today. -- Jeremy Stanley
participants (3)
-
Herve Beraud
-
Jeremy Stanley
-
Mark Goddard