Open Stack

The tempest-full job used to run API and scenario tests concurrently, 
and if you go back far enough I think it also ran slow tests.

Sometime in the last year or so, the full job was changed to run the 
scenario tests in serial and exclude the slow tests altogether. So the 
API tests run concurrently first, and then the scenario tests run in 
serial. During that change, some other tests were identified as 'slow' 
and marked as such, meaning they don't get run in the normal 
tempest-full job.

There are some valuable scenario tests marked as slow, however, like the 
only encrypted volume testing we have in tempest is marked slow so it 
doesn't get run on every change for at least nova.

There is only one job that can be run against nova changes which runs 
the slow tests but it's in the experimental queue so people forget to 
run it.

As a test, I've proposed a nova-slow job [1] which only runs the slow 
tests and only the compute API and scenario tests. Since there currently 
no compute API tests marked as slow, it's really just running slow 
scenario tests. Results show it runs 37 tests in about 37 minutes [2]. 
The overall job runtime was 1 hour and 9 minutes, which is on average 
less than the tempest-full job. The nova-slow job is also running 
scenarios that nova patches don't actually care about, like the neutron 
IPv6 scenario tests.

My question is, should we make this a generic tempest-slow job which can 
be run either in the integrated-gate or at least in nova/neutron/cinder 
consistently (I'm not sure if there are slow tests for just keystone or 
glance)? I don't know if the other projects already have something like 
this that they gate on. If so, a nova-specific job for nova changes is 
fine for me.

[1] https://review.openstack.org/#/c/567697/
[2] 
http://logs.openstack.org/97/567697/1/check/nova-slow/bedfafb/job-output.txt.gz#_2018-05-10_23_46_47_588138

-- 

Thanks,

Matt