Open Stack

On Fri, May 11, 2018 at 08:45:39AM -0500, Matt Riedemann wrote:
> The tempest-full job used to run API and scenario tests concurrently, and if
> you go back far enough I think it also ran slow tests.

Well it's a bit more subtle than that. Skipping slow tests was added right
before we introduced parallel execution to tempest ~5 years ago:

https://github.com/openstack/tempest/commit/68a8060b24abd6b6bf99c4f9296bf418a8349a2d

Note those are in separate testr jobs which we migrated to the full job a bit
later in that cycle. The full job back then ran using nose and ran things
serially. But back then we didn't actually have any tests tagged as slow. It was
more of a future proofing thing because we were planning to add a bunch of
really slow heat tests we didn't want to run on every commit to each project.
The slow tags were first added for heat tests which came later in the havana
cycle.

> 
> Sometime in the last year or so, the full job was changed to run the
> scenario tests in serial and exclude the slow tests altogether. So the API
> tests run concurrently first, and then the scenario tests run in serial.
> During that change, some other tests were identified as 'slow' and marked as
> such, meaning they don't get run in the normal tempest-full job.

It was changed in:

https://github.com/openstack/tempest/commit/49505df20f3dc578506e479c2afa4a4f02e464bf

> 
> There are some valuable scenario tests marked as slow, however, like the
> only encrypted volume testing we have in tempest is marked slow so it
> doesn't get run on every change for at least nova.
> 
> There is only one job that can be run against nova changes which runs the
> slow tests but it's in the experimental queue so people forget to run it.
> 
> As a test, I've proposed a nova-slow job [1] which only runs the slow tests
> and only the compute API and scenario tests. Since there currently no
> compute API tests marked as slow, it's really just running slow scenario
> tests. Results show it runs 37 tests in about 37 minutes [2]. The overall
> job runtime was 1 hour and 9 minutes, which is on average less than the
> tempest-full job. The nova-slow job is also running scenarios that nova
> patches don't actually care about, like the neutron IPv6 scenario tests.
> 
> My question is, should we make this a generic tempest-slow job which can be
> run either in the integrated-gate or at least in nova/neutron/cinder
> consistently (I'm not sure if there are slow tests for just keystone or
> glance)? I don't know if the other projects already have something like this
> that they gate on. If so, a nova-specific job for nova changes is fine for
> me.

So there used to be an experimental queue tempest-all job which ran everything
in tempest, including the slow tests. I can't find it in the .zuul.yaml in the
tempest repo, so my assumption is that got dropped during the v3 migration.

I'm fine with adding a general purpose job for just running the slow tests to
the integrated gate if we think there is enough value from that. It's mostly
just a question of weighing the potential value from the increased coverage vs
the increased resource consumption for adding yet another job to the integrated
gate. Personally, I'm fine with that tradeoff.

-Matt Treinish

> 
> [1] https://review.openstack.org/#/c/567697/
> [2] http://logs.openstack.org/97/567697/1/check/nova-slow/bedfafb/job-output.txt.gz#_2018-05-10_23_46_47_588138
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180511/e970021d/attachment.sig>