Open Stack

On 1/2/19 12:18 PM, Dmitry Tantsur wrote:
> Hi all and happy new year :)
> 
> As you know, tempest plugins are branchless, so the CI of ironic-tempest-plugin 
> has to run tests on all supported branches. Currently it amounts to 16 (!) 
> voting devstack jobs. With each of them have some small probability of a random 
> failure, it is impossible to land anything without at least one recheck, usually 
> more.
> 
> The bad news is, we only run master API tests job, and these tests are changed 
> more often that the other. We already had a minor stable branch breakage because 
> of it [1]. We need to run 3 more jobs: for Pike, Queens and Rocky. And I've just 
> spotted a missing master multinode job, which is defined but does not run for 
> some reason :(

Better news: the API tests did not have a separate job before Rocky, so we only 
need to add Rocky. However, we'll get to 4 jobs in the future.

The multinode job is missing because it was renamed on master, and apparently 
Zuul does not report it Oo

> 
> Here is my proposal to deal with gate bloat on ironic-tempest-plugin:
> 
> 1. Do not run CI jobs at all for unsupported branches and branches in extended 
> maintenance. For Ocata this has already been done in [2].
> 
> 2. Make jobs running with N-3 (currently Pike) and older non-voting (and thus 
> remove them from the gate queue). I have a gut feeling that a change that breaks 
> N-3 is very likely to break N-2 (currently Queens) as well, so it's enough to 
> have N-2 voting.
> 
> 3. Make the discovery and the multinode jobs from all stable branches 
> non-voting. These jobs cover the tests that get changed very infrequently (if 
> ever). These are also the jobs with the highest random failure rate.
> 
> 4. Add the API tests, voting for Queens to master, non-voting for Pike (as 
> proposed above).

Only Rocky here for now.

> 
> This should leave us with 20 jobs, but with only 11 of them voting. Which is 
> still a lot, but probably manageable.
> 
> The corresponding change is [3], please comment here or there.
> 
> Dmitry
> 
> [1] https://review.openstack.org/622177
> [2] https://review.openstack.org/621537
> [3] https://review.openstack.org/627955