On Mon, Aug 6, 2018 at 12:56 PM Wesley Hayutin <whayutin at redhat.com> wrote: > Greetings, > > There is currently an unplanned outtage atm for the tripleo 3rd party OVB > based jobs. > We will contact the list when there are more details. > > Thank you! > OK, I'm going to call an end to the current outtage. We are closely monitoring the ovb 3rd party jobs. I'll called for the outtage when we hit [1]. Once I deleted the stack that moved teh HA routers to back_up state, the networking came back online. Additionally Kieran and I had to work through a number of instances that required admin access to remove. Once those resources were cleaned up our CI tooling removed the rest of the stacks in delete_failed status. The stacks in delete_failed status were holding ip address that were causing new stacks to fail [2] There are still active issues that could cause OVB jobs to fail. This connection issues [3] was originaly thought to be DNS, however that turned out to not be the case. You may also see your job have a "node_failure" status, Paul has sent updates about this issue and is working on a patch and integration into rdo software factory. The CI team is close to including all the console logs into the regular job logs, however if needed atm they can be viewed at [5]. We are also adding the bmc to the list of instances that we collect logs from. *To summarize* the most recent outtage was infra related and the errors were swallowed up in the bmc console log that at the time was not available to users. We continue to monitor that ovb jobs at http://cistatus.tripleo.org/ The legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master job is at a 53% pass rate, it needs to move to a > 85% pass rate to match other check jobs. Thanks all! [1] https://bugzilla.redhat.com/show_bug.cgi?id=1570136 [2] http://paste.openstack.org/show/727444/ [3] https://bugs.launchpad.net/tripleo/+bug/1785342 [4] https://review.openstack.org/#/c/584488/ [5] http://38.145.34.41/console-logs/?C=M;O=D > > -- > > Wes Hayutin > > Associate MANAGER > > Red Hat > > <https://www.redhat.com/> > > w <cclayton at redhat.com>hayutin at redhat.com T: +1919 <+19197544114> > 4232509 IRC: weshay > <https://red.ht/sig> > > View my calendar and check my availability for meetings HERE > <https://calendar.google.com/calendar/b/1/embed?src=whayutin@redhat.com&ctz=America/New_York> > -- Wes Hayutin Associate MANAGER Red Hat <https://www.redhat.com/> w <cclayton at redhat.com>hayutin at redhat.com T: +1919 <+19197544114>4232509 IRC: weshay <https://red.ht/sig> View my calendar and check my availability for meetings HERE <https://calendar.google.com/calendar/b/1/embed?src=whayutin@redhat.com&ctz=America/New_York> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180806/b21da480/attachment.html>