Open Stack

On Thu, Jan 20, 2022 at 8:25 PM Jeremy Stanley <fungi at yuggoth.org> wrote:
>
> On 2022-01-20 14:09:09 +0000 (+0000), Jeremy Stanley wrote:
> [...]
> > The content of /centos/8-stream/AppStream/x86_64/os/repodata/ on
> > mirror.facebook.net is identical to what we're serving already. I
> > checked some other mirrors, e.g. linuxsoft.cern.ch, and see the
> > same. The repomd.xml indices on them all match too.
>
> Further investigation of our mirror update logs indicates there was
> some (likely global) upheaval for CentOS Stream 8 package indices,
> which we then mirrored on what was probably a several hour delay as
> we're multiple mirror "hops" from their primary. The mirror at
> LeaseWeb, which we pull from, had an index update around 06:00 UTC
> which seems to roughly coincide with when the problems began, and
> then we saw those indices switch back around 12:00 UTC to what they
> had been previously.
>
> The timeframe where the suspected problem indices were being served
> from our mirrors was approximately 06:55-12:57 UTC. We also saw a
> failure to upload updated centos-8-stream images to a significant
> proportion of our providers shortly prior to this, so out of an
> abundance of caution I've issued a delete for that image (falling
> back to the one built yesterday), and our builders are presently
> refreshing it from what is hopefully now a sane mirror of the
> packages and indices.

Thank you Jeremy for looking into it.
It seems the issue was from the CentOS side itself.
Maybe the RDO team can help here to avoid these kinds of issues in future.

Thanks,

Chandan Kumar