On Tue, Sep 3, 2019 at 10:43 AM Gabriele Cerami <gcerami@redhat.com> wrote:

Hi,

this weekend, and outage caused trunk.rdoprojects.org servers to become unreachable. As main effect, all tripleo ci jobs were unable to download and install dlrn repositories for the needed hashes and failed.

The outage has been resolved yesterday but there's a problem with DNS propagation and we're still seeing DNS queries returning incorrect IPs, and as a result, jobs are not consistently passing. You may see problems dowloading repos, building changes, installing packages

We would advise to limit the rechecks until we are sure the DNS results are stable.

Thanks.

Just adding a little more clarity. You can see the pass rate of TripleO jobs start to drop on 8/31 and recover on 9/3 in the screenshot. We have not yet fully recovered quite yet, we will update this thread when that is the case. Thanks!