<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Mon, Aug 6, 2018 at 5:55 PM Wesley Hayutin <<a href="mailto:whayutin@redhat.com">whayutin@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Mon, Aug 6, 2018 at 12:56 PM Wesley Hayutin <<a href="mailto:whayutin@redhat.com" target="_blank">whayutin@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Greetings,<div><br></div><div>There is currently an unplanned outtage atm for the tripleo 3rd party OVB based jobs.</div><div>We will contact the list when there are more details.  </div><div><br></div><div>Thank you!</div></div></blockquote><div><br></div></div></div><div dir="ltr"><div class="gmail_quote"><div>OK, </div><div>I'm going to call an end to the current outtage. We are closely monitoring the ovb 3rd party jobs.</div><div>I'll called for the outtage when we hit [1].  Once I deleted the stack that moved teh HA routers to back_up state, the networking came back online.</div><div><br></div><div>Additionally Kieran and I had to work through a number of instances that required admin access to remove.</div><div>Once those resources  were cleaned up our CI tooling removed the rest of the stacks in delete_failed status.    The stacks in delete_failed status</div><div>were holding ip address that were causing new stacks to fail [2]</div><div><br></div><div>There are still active issues that could cause OVB jobs to fail.</div><div>This connection issues [3] was originaly thought to be DNS, however that turned out to not be the case.</div><div>You may also see your job have a "node_failure" status, Paul has sent updates about this issue and is working on a patch and integration into rdo software factory.</div><div><br></div><div>The CI team is close to including all the console logs into the regular job logs, however if needed atm they can be viewed at [5].</div><div>We are also adding the bmc to the list of instances that we collect logs from.</div><div><br></div><div><b>To summarize</b> the most recent outtage was infra related and the errors were swallowed up in the bmc console log that at the time was not available to users.</div><div><br></div><div>We continue to monitor that ovb jobs at <a href="http://cistatus.tripleo.org/" target="_blank">http://cistatus.tripleo.org/</a></div><div>The  legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master job is at a 53% pass rate, it needs to move to a > 85% pass rate to match other check jobs.</div><div><br></div><div>Thanks all!</div></div></div></blockquote><div><br>Following up,</div><div> legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master job is at a 78.6% pass rate today.   Certainly an improvement.<br></div><div><br></div><div>We had a quick sync meeting this morning w/ RDO-Cloud admins, tripleo and infra folks.  There are two remaining issues.</div><div>There is an active issue w/ network connections, and an issue w/ instances booting into node_failure status.   New issues</div><div>creep up all the time and we're actively monitoring those as well.  Still shooting for 85% pass rate.</div><div><br></div><div>Thanks all </div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br></div><div>[1] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1570136" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1570136</a></div><div>[2] <a href="http://paste.openstack.org/show/727444/" target="_blank">http://paste.openstack.org/show/727444/</a></div><div>[3] <a href="https://bugs.launchpad.net/tripleo/+bug/1785342" target="_blank">https://bugs.launchpad.net/tripleo/+bug/1785342</a></div><div>[4] <a href="https://review.openstack.org/#/c/584488/" target="_blank">https://review.openstack.org/#/c/584488/</a></div><div>[5] <a href="http://38.145.34.41/console-logs/?C=M;O=D" target="_blank">http://38.145.34.41/console-logs/?C=M;O=D</a></div></div></div><div dir="ltr"><div class="gmail_quote"><div><br></div><div><br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div></div>-- <br><div dir="ltr" class="m_-7706401370584748751m_-4652934776678972526gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div style="font-family:sans-serif"><p style="font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;text-transform:uppercase"><span style="font-size:14px">Wes Hayutin</span></p><p style="font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;text-transform:uppercase"><span style="color:rgb(0,0,0);font-size:10px">Associate MANAGER</span><br></p><p style="color:rgb(153,153,153);font-family:overpass,sans-serif;margin:0px;font-size:10px"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat <br><br></a></p><span style="color:rgb(153,153,153);font-family:overpass,sans-serif;font-size:10px;margin:0px"><p style="margin:0px"><span style="color:rgb(153,153,153);margin:0px;padding:0px"><a href="mailto:cclayton@redhat.com" style="color:rgb(0,136,206);margin:0px" target="_blank">w</a><a href="mailto:hayutin@redhat.com" target="_blank">hayutin@redhat.com</a>   </span><span style="color:rgb(153,153,153)"> T: </span><a href="tel:+19197544114" style="color:rgb(0,136,206);margin:0px" target="_blank">+1919</a>4232509<span style="color:rgb(153,153,153)">     IRC:  </span>weshay<br></p></span><table border="0" style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium"><tbody><tr><td width="100px"><a href="https://red.ht/sig" target="_blank"><img width="90" height="auto" src="https://www.redhat.com/files/brand/email/sig-redhat.png"></a></td></tr></tbody></table></div><br style="font-family:sans-serif"><span style="font-family:sans-serif">View</span><span style="font-family:sans-serif"> </span><span style="font-family:sans-serif">my</span><span style="font-family:sans-serif"> </span><span style="font-family:sans-serif">calendar</span><span style="font-family:sans-serif"> and check </span><span style="font-family:sans-serif">my</span><span style="font-family:sans-serif"> availability for meetings </span><span id="m_-7706401370584748751m_-4652934776678972526inbox-inbox-docs-internal-guid-37a711e5-1f4c-b3c1-221b-f589048200c7"><a href="https://calendar.google.com/calendar/b/1/embed?src=whayutin@redhat.com&ctz=America/New_York" target="_blank"><span style="font-size:11pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">HERE</span></a></span><br></div></div>
</blockquote></div></div>-- <br><div dir="ltr" class="m_-7706401370584748751gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div style="font-family:sans-serif"><p style="font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;text-transform:uppercase"><span style="font-size:14px">Wes Hayutin</span></p><p style="font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;text-transform:uppercase"><span style="color:rgb(0,0,0);font-size:10px">Associate MANAGER</span><br></p><p style="color:rgb(153,153,153);font-family:overpass,sans-serif;margin:0px;font-size:10px"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat <br><br></a></p><span style="color:rgb(153,153,153);font-family:overpass,sans-serif;font-size:10px;margin:0px"><p style="margin:0px"><span style="color:rgb(153,153,153);margin:0px;padding:0px"><a href="mailto:cclayton@redhat.com" style="color:rgb(0,136,206);margin:0px" target="_blank">w</a><a href="mailto:hayutin@redhat.com" target="_blank">hayutin@redhat.com</a>   </span><span style="color:rgb(153,153,153)"> T: </span><a href="tel:+19197544114" style="color:rgb(0,136,206);margin:0px" target="_blank">+1919</a>4232509<span style="color:rgb(153,153,153)">     IRC:  </span>weshay<br></p></span><table border="0" style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium"><tbody><tr><td width="100px"><a href="https://red.ht/sig" target="_blank"><img width="90" height="auto" src="https://www.redhat.com/files/brand/email/sig-redhat.png"></a></td></tr></tbody></table></div><br style="font-family:sans-serif"><span style="font-family:sans-serif">View</span><span style="font-family:sans-serif"> </span><span style="font-family:sans-serif">my</span><span style="font-family:sans-serif"> </span><span style="font-family:sans-serif">calendar</span><span style="font-family:sans-serif"> and check </span><span style="font-family:sans-serif">my</span><span style="font-family:sans-serif"> availability for meetings </span><span id="m_-7706401370584748751inbox-inbox-docs-internal-guid-37a711e5-1f4c-b3c1-221b-f589048200c7"><a href="https://calendar.google.com/calendar/b/1/embed?src=whayutin@redhat.com&ctz=America/New_York" target="_blank"><span style="font-size:11pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">HERE</span></a></span><br></div></div>
</blockquote></div></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div style="font-family:sans-serif"><p style="font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;text-transform:uppercase"><span style="font-size:14px">Wes Hayutin</span></p><p style="font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;text-transform:uppercase"><span style="color:rgb(0,0,0);font-size:10px">Associate MANAGER</span><br></p><p style="color:rgb(153,153,153);font-family:overpass,sans-serif;margin:0px;font-size:10px"><a href="https://www.redhat.com/" target="_blank" style="color:rgb(0,136,206);margin:0px">Red Hat <br><br></a></p><span style="color:rgb(153,153,153);font-family:overpass,sans-serif;font-size:10px;margin:0px"><p style="margin:0px"><span style="color:rgb(153,153,153);margin:0px;padding:0px"><a href="mailto:cclayton@redhat.com" target="_blank" style="color:rgb(0,136,206);margin:0px">w</a><a href="mailto:hayutin@redhat.com">hayutin@redhat.com</a>   </span><span style="color:rgb(153,153,153)"> T: </span><a href="tel:+19197544114" target="_blank" style="color:rgb(0,136,206);margin:0px">+1919</a>4232509<span style="color:rgb(153,153,153)">     IRC:  </span>weshay<br></p></span><table border="0" style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium"><tbody><tr><td width="100px"><a href="https://red.ht/sig" target="_blank"><img width="90" height="auto" src="https://www.redhat.com/files/brand/email/sig-redhat.png"></a></td></tr></tbody></table></div><br style="font-family:sans-serif"><span style="font-family:sans-serif">View</span><span style="font-family:sans-serif"> </span><span style="font-family:sans-serif">my</span><span style="font-family:sans-serif"> </span><span style="font-family:sans-serif">calendar</span><span style="font-family:sans-serif"> and check </span><span style="font-family:sans-serif">my</span><span style="font-family:sans-serif"> availability for meetings </span><span id="inbox-inbox-docs-internal-guid-37a711e5-1f4c-b3c1-221b-f589048200c7"><a href="https://calendar.google.com/calendar/b/1/embed?src=whayutin@redhat.com&ctz=America/New_York"><span style="font-size:11pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">HERE</span></a></span><br></div></div>