Open Stack

Hi all and happy new year :)

As you know, tempest plugins are branchless, so the CI of ironic-tempest-plugin 
has to run tests on all supported branches. Currently it amounts to 16 (!) 
voting devstack jobs. With each of them have some small probability of a random 
failure, it is impossible to land anything without at least one recheck, usually 
more.

The bad news is, we only run master API tests job, and these tests are changed 
more often that the other. We already had a minor stable branch breakage because 
of it [1]. We need to run 3 more jobs: for Pike, Queens and Rocky. And I've just 
spotted a missing master multinode job, which is defined but does not run for 
some reason :(

Here is my proposal to deal with gate bloat on ironic-tempest-plugin:

1. Do not run CI jobs at all for unsupported branches and branches in extended 
maintenance. For Ocata this has already been done in [2].

2. Make jobs running with N-3 (currently Pike) and older non-voting (and thus 
remove them from the gate queue). I have a gut feeling that a change that breaks 
N-3 is very likely to break N-2 (currently Queens) as well, so it's enough to 
have N-2 voting.

3. Make the discovery and the multinode jobs from all stable branches 
non-voting. These jobs cover the tests that get changed very infrequently (if 
ever). These are also the jobs with the highest random failure rate.

4. Add the API tests, voting for Queens to master, non-voting for Pike (as 
proposed above).

This should leave us with 20 jobs, but with only 11 of them voting. Which is 
still a lot, but probably manageable.

The corresponding change is [3], please comment here or there.

Dmitry

[1] https://review.openstack.org/622177
[2] https://review.openstack.org/621537
[3] https://review.openstack.org/627955