[openstack-dev] [tripleo] 3rd party ovb jobs are down

Wesley Hayutin whayutin at redhat.com
Wed Aug 8 16:42:07 UTC 2018


[image: rdo-ci-3rd-party.png]

RDO 3rd party jobs at 85% :)
If you round up.

Nice work all!

On Tue, Aug 7, 2018 at 10:14 AM Wesley Hayutin <whayutin at redhat.com> wrote:

> On Mon, Aug 6, 2018 at 5:55 PM Wesley Hayutin <whayutin at redhat.com> wrote:
>
>> On Mon, Aug 6, 2018 at 12:56 PM Wesley Hayutin <whayutin at redhat.com>
>> wrote:
>>
>>> Greetings,
>>>
>>> There is currently an unplanned outtage atm for the tripleo 3rd party
>>> OVB based jobs.
>>> We will contact the list when there are more details.
>>>
>>> Thank you!
>>>
>>
>> OK,
>> I'm going to call an end to the current outtage. We are closely
>> monitoring the ovb 3rd party jobs.
>> I'll called for the outtage when we hit [1].  Once I deleted the stack
>> that moved teh HA routers to back_up state, the networking came back online.
>>
>> Additionally Kieran and I had to work through a number of instances that
>> required admin access to remove.
>> Once those resources  were cleaned up our CI tooling removed the rest of
>> the stacks in delete_failed status.    The stacks in delete_failed status
>> were holding ip address that were causing new stacks to fail [2]
>>
>> There are still active issues that could cause OVB jobs to fail.
>> This connection issues [3] was originaly thought to be DNS, however that
>> turned out to not be the case.
>> You may also see your job have a "node_failure" status, Paul has sent
>> updates about this issue and is working on a patch and integration into rdo
>> software factory.
>>
>> The CI team is close to including all the console logs into the regular
>> job logs, however if needed atm they can be viewed at [5].
>> We are also adding the bmc to the list of instances that we collect logs
>> from.
>>
>> *To summarize* the most recent outtage was infra related and the errors
>> were swallowed up in the bmc console log that at the time was not available
>> to users.
>>
>> We continue to monitor that ovb jobs at http://cistatus.tripleo.org/
>> The  legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master job
>> is at a 53% pass rate, it needs to move to a > 85% pass rate to match other
>> check jobs.
>>
>> Thanks all!
>>
>
> Following up,
>  legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master job is at
> a 78.6% pass rate today.   Certainly an improvement.
>
> We had a quick sync meeting this morning w/ RDO-Cloud admins, tripleo and
> infra folks.  There are two remaining issues.
> There is an active issue w/ network connections, and an issue w/ instances
> booting into node_failure status.   New issues
> creep up all the time and we're actively monitoring those as well.  Still
> shooting for 85% pass rate.
>
> Thanks all
>
>
>
>>
>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1570136
>> [2] http://paste.openstack.org/show/727444/
>> [3] https://bugs.launchpad.net/tripleo/+bug/1785342
>> [4] https://review.openstack.org/#/c/584488/
>> [5] http://38.145.34.41/console-logs/?C=M;O=D
>>
>>
>>
>>
>>
>>
>>>
>>> --
>>>
>>> Wes Hayutin
>>>
>>> Associate MANAGER
>>>
>>> Red Hat
>>>
>>> <https://www.redhat.com/>
>>>
>>> w <cclayton at redhat.com>hayutin at redhat.com    T: +1919 <+19197544114>
>>> 4232509     IRC:  weshay
>>> <https://red.ht/sig>
>>>
>>> View my calendar and check my availability for meetings HERE
>>> <https://calendar.google.com/calendar/b/1/embed?src=whayutin@redhat.com&ctz=America/New_York>
>>>
>> --
>>
>> Wes Hayutin
>>
>> Associate MANAGER
>>
>> Red Hat
>>
>> <https://www.redhat.com/>
>>
>> w <cclayton at redhat.com>hayutin at redhat.com    T: +1919 <+19197544114>
>> 4232509     IRC:  weshay
>> <https://red.ht/sig>
>>
>> View my calendar and check my availability for meetings HERE
>> <https://calendar.google.com/calendar/b/1/embed?src=whayutin@redhat.com&ctz=America/New_York>
>>
> --
>
> Wes Hayutin
>
> Associate MANAGER
>
> Red Hat
>
> <https://www.redhat.com/>
>
> w <cclayton at redhat.com>hayutin at redhat.com    T: +1919 <+19197544114>
> 4232509     IRC:  weshay
> <https://red.ht/sig>
>
> View my calendar and check my availability for meetings HERE
> <https://calendar.google.com/calendar/b/1/embed?src=whayutin@redhat.com&ctz=America/New_York>
>
-- 

Wes Hayutin

Associate MANAGER

Red Hat

<https://www.redhat.com/>

w <cclayton at redhat.com>hayutin at redhat.com    T: +1919 <+19197544114>4232509
   IRC:  weshay
<https://red.ht/sig>

View my calendar and check my availability for meetings HERE
<https://calendar.google.com/calendar/b/1/embed?src=whayutin@redhat.com&ctz=America/New_York>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180808/e348f476/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rdo-ci-3rd-party.png
Type: image/png
Size: 161915 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180808/e348f476/attachment-0001.png>


More information about the OpenStack-dev mailing list