On 2021-01-08 14:42:14 +0100 (+0100), Herve Beraud wrote:
Also notice that yesterday we met an AFS issue [1] during the merge of the following patches:
- https://review.opendev.org/c/openstack/releases/+/769325 - https://review.opendev.org/c/openstack/releases/+/769322 - https://review.opendev.org/c/openstack/releases/+/769324
The problem was that afs an server got stuck in a pathological way from 05:50 utc today until 16:10 utc when we hard rebooted the server instance.
The patches you reference merged later, so wasn't directly impacted by write failures (the jobs actually succeeded).
The consequence of this is that the related tarballs haven't been published:
- https://tarballs.opendev.org/openstack/kolla/?C=M;O=D - https://tarballs.opendev.org/openstack/kolla-ansible/?C=M;O=D - https://tarballs.opendev.org/openstack/kayobe/?C=M;O=D
And so the RDO CI fail to build them for train, ussuri and victoria:
- https://review.rdoproject.org/r/#/c/31499/ - https://review.rdoproject.org/r/#/c/31498/ - https://review.rdoproject.org/r/#/c/31497/
So I think we need to reenqueue these jobs too. [...]
The files were written into the tarballs volume just fine, but the read-only replicas which back the tarballs.opendev.org site hadn't been synchronized. I found a stuck process (waiting since yesterday for a response from the server which had previously died) and killed it to get the periodic synchronization of the read-only volumes working again so, the site is no longer stale and has those releases on it now as of 15:15:40 UTC today. -- Jeremy Stanley