Open Stack

On Fri, May 11, 2018 at 10:45 PM, Matt Riedemann <mriedemos at gmail.com> wrote:
> The tempest-full job used to run API and scenario tests concurrently, and if
> you go back far enough I think it also ran slow tests.
>
> Sometime in the last year or so, the full job was changed to run the
> scenario tests in serial and exclude the slow tests altogether. So the API
> tests run concurrently first, and then the scenario tests run in serial.
> During that change, some other tests were identified as 'slow' and marked as
> such, meaning they don't get run in the normal tempest-full job.
>
> There are some valuable scenario tests marked as slow, however, like the
> only encrypted volume testing we have in tempest is marked slow so it
> doesn't get run on every change for at least nova.

Yes, basically slow tests were selected based on
https://ethercalc.openstack.org/nu56u2wrfb2b and there were frequent
gate failure for heavy tests mainly from ssh checks so we tried to
mark more tests as slow.
I agree that some of them are not really slow at least in today situation.

>
> There is only one job that can be run against nova changes which runs the
> slow tests but it's in the experimental queue so people forget to run it.

Tempest job "legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend"
run those slow tests including migration and LVM  multibackend tests.
This job runs on tempest check pipeline and experimental (as you
mentioned) on nova and cinder [3]. We marked this as n-v to check its
stability and now it is good to go as voting on tempest.

>
> As a test, I've proposed a nova-slow job [1] which only runs the slow tests
> and only the compute API and scenario tests. Since there currently no
> compute API tests marked as slow, it's really just running slow scenario
> tests. Results show it runs 37 tests in about 37 minutes [2]. The overall
> job runtime was 1 hour and 9 minutes, which is on average less than the
> tempest-full job. The nova-slow job is also running scenarios that nova
> patches don't actually care about, like the neutron IPv6 scenario tests.
>
> My question is, should we make this a generic tempest-slow job which can be
> run either in the integrated-gate or at least in nova/neutron/cinder
> consistently (I'm not sure if there are slow tests for just keystone or
> glance)? I don't know if the other projects already have something like this
> that they gate on. If so, a nova-specific job for nova changes is fine for
> me.

+1 on idea. As of now slow marked tests are from nova, cinder and
neutron scenario tests and 2 API swift tests only [4]. I agree that
making a generic job in tempest is better for maintainability. We can
use existing job for that with below modification-
-  We can migrate
"legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend" job
zuulv3 in tempest repo
-  We can see if we can move migration tests out of it and use
"nova-live-migration" job (in tempest check pipeline ) which is much
better in live migration env setup and controlled by nova.
-  then it can be name something like
"tempest-scenario-multinode-lvm-multibackend".
-  run this job in nova, cinder, neutron check pipeline instead of experimental.

Another update on slow tests is that we are trying the possibility of
taking back the slow tests in tempest-full with new job
"tempest-full-parallel" [5]. Currently this job is n-v and if
everything works fine in this new job then, we can make tempest-full
job to run the slow tests are it used to do previously.

>
> [1] https://review.openstack.org/#/c/567697/
> [2]
> http://logs.openstack.org/97/567697/1/check/nova-slow/bedfafb/job-output.txt.gz#_2018-05-10_23_46_47_588138

..3 http://codesearch.openstack.org/?q=legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend&i=nope&files=&repos=
..4 https://github.com/openstack/tempest/search?utf8=%E2%9C%93&q=%22type%3D%27slow%27%22&type=
..5 https://github.com/openstack/tempest/blob/9c628189e798f46de8c4b9484237f4d6dc6ade7e/.zuul.yaml#L48


-gmann

>
> --
>
> Thanks,
>
> Matt
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev