Open Stack

On Fri, May 11, 2018 at 11:32 PM, Matthew Treinish <mtreinish at kortar.org> wrote:
> On Fri, May 11, 2018 at 08:45:39AM -0500, Matt Riedemann wrote:
>> The tempest-full job used to run API and scenario tests concurrently, and if
>> you go back far enough I think it also ran slow tests.
>
> Well it's a bit more subtle than that. Skipping slow tests was added right
> before we introduced parallel execution to tempest ~5 years ago:
>
> https://github.com/openstack/tempest/commit/68a8060b24abd6b6bf99c4f9296bf418a8349a2d
>
> Note those are in separate testr jobs which we migrated to the full job a bit
> later in that cycle. The full job back then ran using nose and ran things
> serially. But back then we didn't actually have any tests tagged as slow. It was
> more of a future proofing thing because we were planning to add a bunch of
> really slow heat tests we didn't want to run on every commit to each project.
> The slow tags were first added for heat tests which came later in the havana
> cycle.
>
>>
>> Sometime in the last year or so, the full job was changed to run the
>> scenario tests in serial and exclude the slow tests altogether. So the API
>> tests run concurrently first, and then the scenario tests run in serial.
>> During that change, some other tests were identified as 'slow' and marked as
>> such, meaning they don't get run in the normal tempest-full job.
>
> It was changed in:
>
> https://github.com/openstack/tempest/commit/49505df20f3dc578506e479c2afa4a4f02e464bf
>
>>
>> There are some valuable scenario tests marked as slow, however, like the
>> only encrypted volume testing we have in tempest is marked slow so it
>> doesn't get run on every change for at least nova.
>>
>> There is only one job that can be run against nova changes which runs the
>> slow tests but it's in the experimental queue so people forget to run it.
>>
>> As a test, I've proposed a nova-slow job [1] which only runs the slow tests
>> and only the compute API and scenario tests. Since there currently no
>> compute API tests marked as slow, it's really just running slow scenario
>> tests. Results show it runs 37 tests in about 37 minutes [2]. The overall
>> job runtime was 1 hour and 9 minutes, which is on average less than the
>> tempest-full job. The nova-slow job is also running scenarios that nova
>> patches don't actually care about, like the neutron IPv6 scenario tests.
>>
>> My question is, should we make this a generic tempest-slow job which can be
>> run either in the integrated-gate or at least in nova/neutron/cinder
>> consistently (I'm not sure if there are slow tests for just keystone or
>> glance)? I don't know if the other projects already have something like this
>> that they gate on. If so, a nova-specific job for nova changes is fine for
>> me.
>
> So there used to be an experimental queue tempest-all job which ran everything
> in tempest, including the slow tests. I can't find it in the .zuul.yaml in the
> tempest repo, so my assumption is that got dropped during the v3 migration.

It is there with name "legacy-periodic-tempest-dsvm-all-master" [3].
This runs as experimental and periodic for Tempest. It is not yet
migrated, i will plan to migrate that in tempest repo.

>
> I'm fine with adding a general purpose job for just running the slow tests to
> the integrated gate if we think there is enough value from that. It's mostly
> just a question of weighing the potential value from the increased coverage vs
> the increased resource consumption for adding yet another job to the integrated
> gate. Personally, I'm fine with that tradeoff.
>
> -Matt Treinish
>
>>
>> [1] https://review.openstack.org/#/c/567697/
>> [2] http://logs.openstack.org/97/567697/1/check/nova-slow/bedfafb/job-output.txt.gz#_2018-05-10_23_46_47_588138
>>

..3 http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-jobs.yaml#n1579

-gmann
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>