Open Stack

 ---- On Wed, 02 Jan 2019 20:18:40 +0900 Dmitry Tantsur <dtantsur at redhat.com> wrote ---- 
 > Hi all and happy new year :) 
 >  
 > As you know, tempest plugins are branchless, so the CI of ironic-tempest-plugin  
 > has to run tests on all supported branches. Currently it amounts to 16 (!)  
 > voting devstack jobs. With each of them have some small probability of a random  
 > failure, it is impossible to land anything without at least one recheck, usually  
 > more. 
 >  
 > The bad news is, we only run master API tests job, and these tests are changed  
 > more often that the other. We already had a minor stable branch breakage because  
 > of it [1]. We need to run 3 more jobs: for Pike, Queens and Rocky. And I've just  
 > spotted a missing master multinode job, which is defined but does not run for  
 > some reason :( 
 >  
 > Here is my proposal to deal with gate bloat on ironic-tempest-plugin: 
 >  
 > 1. Do not run CI jobs at all for unsupported branches and branches in extended  
 > maintenance. For Ocata this has already been done in [2]. 

+1. We have the same policy in Tempest also[1].
You mean not to run CI for unsupported/EM branches on the master testing right? 
CI on Unsupported/EM branch can be run until they all are passing or EM maintainers want to
run them. 

 >  
 > 2. Make jobs running with N-3 (currently Pike) and older non-voting (and thus  
 > remove them from the gate queue). I have a gut feeling that a change that breaks  
 > N-3 is very likely to break N-2 (currently Queens) as well, so it's enough to  
 > have N-2 voting. 

IMO, running all supported stable branches as voting make sense than running oldest
one(N-3 as you mentioned) as n-v. That way, tempest-plugins will be successfully maintained
to run on N-3  otherwise it is likely to be broken for that branch especially in case of
feature discovery based tests. 

 >  
 > 3. Make the discovery and the multinode jobs from all stable branches  
 > non-voting. These jobs cover the tests that get changed very infrequently (if  
 > ever). These are also the jobs with the highest random failure rate. 
 >  
 > 4. Add the API tests, voting for Queens to master, non-voting for Pike (as  
 > proposed above). 
 >  
 > This should leave us with 20 jobs, but with only 11 of them voting. Which is  
 > still a lot, but probably manageable. 
 >  
 > The corresponding change is [3], please comment here or there. 
 >  
 > Dmitry 
 >  
 > [1] https://review.openstack.org/622177 
 > [2] https://review.openstack.org/621537 
 > [3] https://review.openstack.org/627955 
 >  
 > 

[1]  https://docs.openstack.org/tempest/latest/stable_branch_support_policy.html 

-gmann