On Wed, May 8, 2019 at 8:00 PM Wesley Hayutin <whayutin@redhat.com> wrote:
Seem the jobs go red at midnight May 9th UTC.

What I'm mainly seeing is the overcloud hang after the ssh keys are created, it seems the overcloud nodes do not have network connectivity. 
http://logs.openstack.org/47/657547/5/check/tripleo-ci-centos-7-containers-multinode/af82c9f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

http://logs.openstack.org/47/657547/5/check/tripleo-ci-centos-7-containers-multinode/af82c9f/logs/subnode-2/var/log/extra/failed_services.txt.gz

This looks normal eth0, eth1 come up
http://logs.openstack.org/47/657547/5/check/tripleo-ci-centos-7-containers-multinode/af82c9f/logs/subnode-2/var/log/journal.txt.gz#_May_08_21_42_19

I'm not 100% if this related to some of the latest patches, or if this impacts all jobs atm.
Looking into it.

AFAICT, the issue is either with a few of the patches submitted or a blip in infra.
A clean test patch is working w/ multinode-containers and ovb fs001
CI should be green'ish quoting #tripleo 

Sorry for the spam