On 2020-07-23 14:08:59 +0200 (+0200), Thierry Carrez wrote: [...]

oslo.messaging 12.2.2 - tag OK, build OK, pyPI OK, tarball not published https://zuul.opendev.org/t/openstack/build/dfce7b106189491b9d9026a079c06bdd

designate 8.0.1 - tag OK, build OK, pyPI OK, tarball not published https://zuul.opendev.org/t/openstack/build/0f6c659223df46278c26460b2f3281fe

Error: There was an issue creating /afs/.openstack.org as requested: [Errno 13] Permission denied: b'/afs/.openstack.org'

Impact: - Tarballs are missing from tarballs.o.o - Missing release announces - Missing constraint updates [...]

I've retrieved the copies of the artifacts for these failed writes from PyPI, verified their integrity using the release key signatures included in the build logs, and uploaded the artifacts and signatures to the tarballs site. Both of these builds ran less than 30 minutes apart and, coincidentally, from the same executor (we presently have 12 executors). I tested writing to that same tree from ze11, where the original failures occurred, and encountered no trouble, but that was many hours later. System logs weren't particularly helpful at narrowing down various theories to any one obvious cause (the executor had spontaneously rebooted less than a day earlier, and saw a fairly large time skip at boot due to a >10-minute discrepancy between the system clock and NTP, but I have no evidence to suggest that would have caused this). As a number of other release builds ran successfully in the same timeframe, the most I can surmise is that one of our executors was temporarily unable to write to that AFS volume over the course of half an hour. I'll keep an eye out for any similar issues. -- Jeremy Stanley