[tripleo][ci] mirror issues failing jobs
https://bugs.launchpad.net/tripleo/+bug/1929461 something is going on w/ either centos or the infra mirrors... I suspect centos atm. Mirror monitoring is here: http://cacti.openstack.org/cacti/graph_view.php
On Mon, May 24, 2021, at 1:26 PM, Wesley Hayutin wrote:
https://bugs.launchpad.net/tripleo/+bug/1929461
something is going on w/ either centos or the infra mirrors... I suspect centos atm. Mirror monitoring is here: http://cacti.openstack.org/cacti/graph_view.php
Looks like https://mirror.ord.rax.opendev.org/centos/8-stream/AppStream/x86_64/os/repod... updated to point at repodata/d0435a46fff272cacc6c2d5433e8e2b0b2d70d57141116c8b9fa7624aaf01aaf-filelists.xml.gz but that filelist wasn't present in the mirror we sync from. Looking at the location we sync from they appear to be suffering the same issue: http://mirror.dal10.us.leaseweb.net/centos/8-stream/AppStream/x86_64/os/repo... To fix this we can wait for our upstreams to get in sync or switch to an upstream that is in sync (if one exists). Clark
On Mon, May 24, 2021 at 02:26:10PM -0600, Wesley Hayutin wrote: 65;6401;1c> something is going on w/ either centos or the infra mirrors... I suspect
centos atm.
Just FYI, logs of all mirroring processes are available at https://static.opendev.org/mirror/logs/ (centos is under "rsync-mirrors"). This is all driven by https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/... If anything ever seems out of sync, these are the places to start. -i
On Mon, May 24, 2021 at 9:22 PM Ian Wienand <iwienand@redhat.com> wrote:
On Mon, May 24, 2021 at 02:26:10PM -0600, Wesley Hayutin wrote: 65;6401;1c> something is going on w/ either centos or the infra mirrors... I suspect
centos atm.
Just FYI, logs of all mirroring processes are available at
https://static.opendev.org/mirror/logs/
(centos is under "rsync-mirrors").
This is all driven by
https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/...
If anything ever seems out of sync, these are the places to start.
-i
Thanks Clark and Ian! I took some notes on those links. Things are recovering but it's still a little choppy atm, but improving. TripleO folks.. probably want to keep their +2 / wf to a minimum today until we're in the clear. Thanks all!
On Tue, May 25, 2021 at 6:57 AM Wesley Hayutin <whayutin@redhat.com> wrote:
On Mon, May 24, 2021 at 9:22 PM Ian Wienand <iwienand@redhat.com> wrote:
On Mon, May 24, 2021 at 02:26:10PM -0600, Wesley Hayutin wrote: 65;6401;1c> something is going on w/ either centos or the infra mirrors... I suspect
centos atm.
Just FYI, logs of all mirroring processes are available at
https://static.opendev.org/mirror/logs/
(centos is under "rsync-mirrors").
This is all driven by
https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/...
If anything ever seems out of sync, these are the places to start.
-i
Thanks Clark and Ian! I took some notes on those links. Things are recovering but it's still a little choppy atm, but improving.
TripleO folks.. probably want to keep their +2 / wf to a minimum today until we're in the clear.
Thanks all!
The mirror misses came back today, looks like it's a limestone issue http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22All%20mirrors%20were%20tried%5C%22%20AND%20(tags:%5C%22console%5C%22)%20AND%20voting:1&from=864000.0s
participants (3)
-
Clark Boylan
-
Ian Wienand
-
Wesley Hayutin