[ironic] [qa] ironic-tempest-plugin CI bloat
Dmitry Tantsur
dtantsur at redhat.com
Wed Jan 2 13:08:00 UTC 2019
On 1/2/19 12:18 PM, Dmitry Tantsur wrote:
> Hi all and happy new year :)
>
> As you know, tempest plugins are branchless, so the CI of ironic-tempest-plugin
> has to run tests on all supported branches. Currently it amounts to 16 (!)
> voting devstack jobs. With each of them have some small probability of a random
> failure, it is impossible to land anything without at least one recheck, usually
> more.
>
> The bad news is, we only run master API tests job, and these tests are changed
> more often that the other. We already had a minor stable branch breakage because
> of it [1]. We need to run 3 more jobs: for Pike, Queens and Rocky. And I've just
> spotted a missing master multinode job, which is defined but does not run for
> some reason :(
Better news: the API tests did not have a separate job before Rocky, so we only
need to add Rocky. However, we'll get to 4 jobs in the future.
The multinode job is missing because it was renamed on master, and apparently
Zuul does not report it Oo
>
> Here is my proposal to deal with gate bloat on ironic-tempest-plugin:
>
> 1. Do not run CI jobs at all for unsupported branches and branches in extended
> maintenance. For Ocata this has already been done in [2].
>
> 2. Make jobs running with N-3 (currently Pike) and older non-voting (and thus
> remove them from the gate queue). I have a gut feeling that a change that breaks
> N-3 is very likely to break N-2 (currently Queens) as well, so it's enough to
> have N-2 voting.
>
> 3. Make the discovery and the multinode jobs from all stable branches
> non-voting. These jobs cover the tests that get changed very infrequently (if
> ever). These are also the jobs with the highest random failure rate.
>
> 4. Add the API tests, voting for Queens to master, non-voting for Pike (as
> proposed above).
Only Rocky here for now.
>
> This should leave us with 20 jobs, but with only 11 of them voting. Which is
> still a lot, but probably manageable.
>
> The corresponding change is [3], please comment here or there.
>
> Dmitry
>
> [1] https://review.openstack.org/622177
> [2] https://review.openstack.org/621537
> [3] https://review.openstack.org/627955
More information about the openstack-discuss
mailing list