Open Stack

On 2020-07-23 14:08:59 +0200 (+0200), Thierry Carrez wrote:
[...]
> oslo.messaging 12.2.2 - tag OK, build OK, pyPI OK, tarball not published
> https://zuul.opendev.org/t/openstack/build/dfce7b106189491b9d9026a079c06bdd
> 
> designate 8.0.1 - tag OK, build OK, pyPI OK, tarball not published
> https://zuul.opendev.org/t/openstack/build/0f6c659223df46278c26460b2f3281fe
> 
> Error:
> There was an issue creating /afs/.openstack.org as requested: [Errno 13]
> Permission denied: b'/afs/.openstack.org'
> 
> Impact:
> - Tarballs are missing from tarballs.o.o
> - Missing release announces
> - Missing constraint updates
[...]

I've retrieved the copies of the artifacts for these failed writes
from PyPI, verified their integrity using the release key signatures
included in the build logs, and uploaded the artifacts and
signatures to the tarballs site.

Both of these builds ran less than 30 minutes apart and,
coincidentally, from the same executor (we presently have 12
executors). I tested writing to that same tree from ze11, where the
original failures occurred, and encountered no trouble, but that was
many hours later. System logs weren't particularly helpful at
narrowing down various theories to any one obvious cause (the
executor had spontaneously rebooted less than a day earlier, and saw
a fairly large time skip at boot due to a >10-minute discrepancy
between the system clock and NTP, but I have no evidence to suggest
that would have caused this). As a number of other release builds
ran successfully in the same timeframe, the most I can surmise is
that one of our executors was temporarily unable to write to that
AFS volume over the course of half an hour.

I'll keep an eye out for any similar issues.
-- 
Jeremy Stanley
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20200723/77fa8b26/attachment.sig>