Open Stack

Also notice that yesterday we met an AFS issue [1] during the merge of the
following patches:

- https://review.opendev.org/c/openstack/releases/+/769325
- https://review.opendev.org/c/openstack/releases/+/769322
- https://review.opendev.org/c/openstack/releases/+/769324

The problem was that afs an server got stuck in a pathological way from
05:50 utc today until 16:10 utc when we hard rebooted the server instance.

The consequence of this is that the related tarballs haven't been published:

- https://tarballs.opendev.org/openstack/kolla/?C=M;O=D
- https://tarballs.opendev.org/openstack/kolla-ansible/?C=M;O=D
- https://tarballs.opendev.org/openstack/kayobe/?C=M;O=D

And so the RDO CI fail to build them for train, ussuri and victoria:

- https://review.rdoproject.org/r/#/c/31499/
- https://review.rdoproject.org/r/#/c/31498/
- https://review.rdoproject.org/r/#/c/31497/

So I think we need to reenqueue these jobs too.

Thanks for reading

[1]
http://eavesdrop.openstack.org/irclogs/%23openstack-release/%23openstack-release.2021-01-07.log.html#t2021-01-07T16:48:57

Le ven. 8 janv. 2021 à 09:45, Mark Goddard <mark at stackhpc.com> a écrit :

> Hi Herve,
>
> Thanks for noticing this. The pull rate limit refreshes every 6 hours, is
> specific to the IP used by the job, and in the case of the build/publish
> jobs we only require pulling a single image - the base OS image - per job.
> I suggest we reenqueue the failing jobs.
>
> For build/publish jobs we have discussed using the infra Docker registry
> mirrors, which should avoid hitting Dockerhub too often.
>
> Cheers,
> Mark
>
> On Fri, 8 Jan 2021 at 08:28, Herve Beraud <hberaud at redhat.com> wrote:
>
>> Hello,
>>
>> FYI it seems that kolla met docker's new limitation during its release
>> jobs, especially with your publish jobs, I saw that you (the kolla team)
>> already discussed this limitation [1] on the ML.
>>
>> ```
>>
>> 2021-01-07 17:21:03.396355 | primary | ERROR:kolla.common.utils.base:Error'd with the following message
>>
>> 2021-01-07 17:21:03.396465 | primary | ERROR:kolla.common.utils.base:toomanyrequests: You have reached your pull rate limit. You may increase
>> ```
>>
>> Three jobs here failed for the same reason.
>>
>> I don't think that reenqueue the failing jobs without a specific action to manage this limitation will help us here.
>>
>> Let us know if we can help us in some manner.
>>
>>
>> [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019148.html
>>
>>
>> Le jeu. 7 janv. 2021 à 20:41, <zuul at openstack.org> a écrit :
>>
>>> Build failed.
>>>
>>> - openstack-upload-github-mirror
>>> https://zuul.opendev.org/t/openstack/build/a64b4c4bc398481f9afffa2ad465a012
>>> : SUCCESS in 1m 03s
>>> - release-openstack-python
>>> https://zuul.opendev.org/t/openstack/build/2ed64dfcbcaf483abf003cfdf4f25837
>>> : SUCCESS in 3m 11s
>>> - announce-release
>>> https://zuul.opendev.org/t/openstack/build/84d803d496a2485c983f6233dafcfd71
>>> : SUCCESS in 4m 07s
>>> - propose-update-constraints
>>> https://zuul.opendev.org/t/openstack/build/d0cac6a077054bc8ba9eb92e56c21799
>>> : SUCCESS in 4m 15s
>>> - kolla-publish-centos-source
>>> https://zuul.opendev.org/t/openstack/build/5e8f6aa2f56940a7be77a8dfe1c8ecc6
>>> : SUCCESS in 2h 20m 47s
>>> - kolla-publish-centos-binary
>>> https://zuul.opendev.org/t/openstack/build/8a8a021d9f9c4ca79755b06309710cc7
>>> : SUCCESS in 1h 56m 32s (non-voting)
>>> - kolla-publish-centos8-source
>>> https://zuul.opendev.org/t/openstack/build/fb6891d3f5e4493b880fce263a92e086
>>> : SUCCESS in 1h 50m 57s
>>> - kolla-publish-centos8-binary
>>> https://zuul.opendev.org/t/openstack/build/c312c05e5d084fdbb3f372755221f186
>>> : SUCCESS in 1h 13m 12s (non-voting)
>>> - kolla-publish-debian-source
>>> https://zuul.opendev.org/t/openstack/build/e24c12751b8c4aba881adb6c9ae8dc07
>>> : SUCCESS in 1h 27m 27s (non-voting)
>>> - kolla-publish-debian-source-aarch64
>>> https://zuul.opendev.org/t/openstack/build/1ff3b02df53847d0aa54bf12ea7fa666
>>> : FAILURE in 1h 51m 49s (non-voting)
>>> - kolla-publish-debian-binary
>>> https://zuul.opendev.org/t/openstack/build/012c2de475fe45ea83f1cd8a7420aa6d
>>> : SUCCESS in 1h 15m 15s (non-voting)
>>> - kolla-publish-ubuntu-source
>>> https://zuul.opendev.org/t/openstack/build/88ca21d972514cce954ecb586324fa29
>>> : FAILURE in 4m 14s
>>> - kolla-publish-ubuntu-binary
>>> https://zuul.opendev.org/t/openstack/build/f3d44e4d1b6c4e9799161b156290b238
>>> : FAILURE in 4m 19s (non-voting)
>>>
>>> _______________________________________________
>>> Release-job-failures mailing list
>>> Release-job-failures at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/release-job-failures
>>>
>>
>>
>> --
>> Hervé Beraud
>> Senior Software Engineer at Red Hat
>> irc: hberaud
>> https://github.com/4383/
>> https://twitter.com/4383hberaud
>> -----BEGIN PGP SIGNATURE-----
>>
>> wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+
>> Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+
>> RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP
>> F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G
>> 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g
>> glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw
>> m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ
>> hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0
>> qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y
>> F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3
>> B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O
>> v6rDpkeNksZ9fFSyoY2o
>> =ECSj
>> -----END PGP SIGNATURE-----
>>
>>

-- 
Hervé Beraud
Senior Software Engineer at Red Hat
irc: hberaud
https://github.com/4383/
https://twitter.com/4383hberaud
-----BEGIN PGP SIGNATURE-----

wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+
Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+
RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP
F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G
5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g
glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw
m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ
hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0
qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y
F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3
B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O
v6rDpkeNksZ9fFSyoY2o
=ECSj
-----END PGP SIGNATURE-----
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210108/e4d05d72/attachment-0001.html>