On 2022-01-20 14:09:09 +0000 (+0000), Jeremy Stanley wrote: [...]
The content of /centos/8-stream/AppStream/x86_64/os/repodata/ on mirror.facebook.net is identical to what we're serving already. I checked some other mirrors, e.g. linuxsoft.cern.ch, and see the same. The repomd.xml indices on them all match too.
Further investigation of our mirror update logs indicates there was some (likely global) upheaval for CentOS Stream 8 package indices, which we then mirrored on what was probably a several hour delay as we're multiple mirror "hops" from their primary. The mirror at LeaseWeb, which we pull from, had an index update around 06:00 UTC which seems to roughly coincide with when the problems began, and then we saw those indices switch back around 12:00 UTC to what they had been previously. The timeframe where the suspected problem indices were being served from our mirrors was approximately 06:55-12:57 UTC. We also saw a failure to upload updated centos-8-stream images to a significant proportion of our providers shortly prior to this, so out of an abundance of caution I've issued a delete for that image (falling back to the one built yesterday), and our builders are presently refreshing it from what is hopefully now a sane mirror of the packages and indices. -- Jeremy Stanley