Open Stack

Ok so here's the latest. We've switched scenario001 to non-voting[0]
for now until Bug 1731063[1] can be resolved. We should be OK to start
merging thing in master as the other current issues don't appear to be
affecting the gate significantly as it stands.  We still need to
understand why we're hitting Bug 1731063 and address the problem so we
can revert the non-voting change ASAP.  Scenario001 provides lots of
coverage for TripleO so I do not want to see it non-voting for long.
If scenario001 is failing on your change, please make sure it is not
Bug 1731063 before rechecking or approving.  If you are approving
changes or rechecking and it fails, do not blindly recheck. Please
file a new bug and ping #tripleo so we can make sure we don't have
other things that may affect the gate.

Thanks,
-Alex

[0] https://review.openstack.org/#/c/520155/
[1] https://bugs.launchpad.net/tripleo/+bug/1731063

On Sat, Nov 11, 2017 at 8:47 PM, Alex Schultz <aschultz at redhat.com> wrote:
> Ok so here's the current status of things.  I've gone through some of
> the pending patches and sent them to the gate over the weekend since
> the gate was empty (yay!).  We've managed to land a bunch of patches.
> That being said for any patch for master with scenario jobs, please do
> not recheck/approve. Currently the non-containerized scenario001/004
> jobs are broken due to Bug 1731688[0] (these run on
> tripleo-quickstart-extras/tripleo-ci).  There is a patch[1] out for a
> revert of the breaking change. The scenario001-container job is super
> flaky due to Bug 1731063[2] and we could use some help figuring out
> what's going on.  We're also seeing some issues around heat
> interactions[3][4] but those seems to be less of a problem than the
> previously mentioned bugs.
>
> So at the moment any changes that don't have scenario jobs associated
> with them may be approved/rechecked freely.  We can discuss on Monday
> what to do about the scenario jobs if we still are running into issues
> without a solution in sight.  Also please keep an eye on the gate
> queue[5] and don't approve things if it starts getting excessively
> long.
>
> Thanks,
> -Alex
>
>
> [0] https://bugs.launchpad.net/tripleo/+bug/1731688
> [1] https://review.openstack.org/#/c/519041/
> [2] https://bugs.launchpad.net/tripleo/+bug/1731063
> [3] https://bugs.launchpad.net/tripleo/+bug/1731032
> [4] https://bugs.launchpad.net/tripleo/+bug/1731540
> [5] http://zuulv3.openstack.org/
>
> On Wed, Nov 8, 2017 at 3:39 PM, Alex Schultz <aschultz at redhat.com> wrote:
>> So we have some good news and some bad news.  The good news is that
>> we've managed to get the gate queue[0] under control since we've held
>> off on pushing new things to the gate.  The bad news is that we've
>> still got some random failures occurring during the deployment of
>> master.  Since we're not seeing infra related issues, we should be OK
>> to merge things to stable/* branches.  Unfortunately until we resolve
>> the issues in master[1] we could potentially backup the queue.  Please
>> do not merge things that are not critical bugs.  I would ask that
>> folks please take a look at the open bugs and help figure out what is
>> going wrong. I've created two issues today that I've seen in the gate
>> that we don't appear to have open patches for. One appears to be an
>> issue in the heat deployment process[3] and the other is related to
>> the tempest verification of being able to launch a VM & ssh to it[4].
>>
>> Thanks,
>> -Alex
>>
>> [3] https://bugs.launchpad.net/tripleo/+bug/1731032
>> [4] https://bugs.launchpad.net/tripleo/+bug/1731063
>>
>> On Tue, Nov 7, 2017 at 8:33 AM, Alex Schultz <aschultz at redhat.com> wrote:
>>> Hey Folks
>>>
>>> So we're at 24+ hours again in the gate[0] and the queue only
>>> continues to grow. We currently have 6 ci/alert bugs[1]. Please do not
>>> approve of recheck anything that isn't related to these bugs.  I will
>>> most likely need to go through the queue and abandon everything to
>>> clear it up as we are consistently hitting timeouts on various jobs
>>> which is preventing anything from merging.
>>>
>>> Thanks,
>>> -Alex
>>>
>> [0] http://zuulv3.openstack.org/
>> [1] https://bugs.launchpad.net/tripleo/+bugs?field.searchtext=&orderby=-importance&field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.importance%3Alist=CRITICAL&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=ci+alert&field.tags_combinator=ALL&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search