Open Stack

On Sun, May 13, 2018 at 1:20 PM, Ghanshyam Mann <gmann at ghanshyammann.com> wrote:
> On Fri, May 11, 2018 at 10:45 PM, Matt Riedemann <mriedemos at gmail.com> wrote:
>> The tempest-full job used to run API and scenario tests concurrently, and if
>> you go back far enough I think it also ran slow tests.
>>
>> Sometime in the last year or so, the full job was changed to run the
>> scenario tests in serial and exclude the slow tests altogether. So the API
>> tests run concurrently first, and then the scenario tests run in serial.
>> During that change, some other tests were identified as 'slow' and marked as
>> such, meaning they don't get run in the normal tempest-full job.
>>
>> There are some valuable scenario tests marked as slow, however, like the
>> only encrypted volume testing we have in tempest is marked slow so it
>> doesn't get run on every change for at least nova.
>
> Yes, basically slow tests were selected based on
> https://ethercalc.openstack.org/nu56u2wrfb2b and there were frequent
> gate failure for heavy tests mainly from ssh checks so we tried to
> mark more tests as slow.
> I agree that some of them are not really slow at least in today situation.
>
>>
>> There is only one job that can be run against nova changes which runs the
>> slow tests but it's in the experimental queue so people forget to run it.
>
> Tempest job "legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend"
> run those slow tests including migration and LVM  multibackend tests.
> This job runs on tempest check pipeline and experimental (as you
> mentioned) on nova and cinder [3]. We marked this as n-v to check its
> stability and now it is good to go as voting on tempest.
>
>>
>> As a test, I've proposed a nova-slow job [1] which only runs the slow tests
>> and only the compute API and scenario tests. Since there currently no
>> compute API tests marked as slow, it's really just running slow scenario
>> tests. Results show it runs 37 tests in about 37 minutes [2]. The overall
>> job runtime was 1 hour and 9 minutes, which is on average less than the
>> tempest-full job. The nova-slow job is also running scenarios that nova
>> patches don't actually care about, like the neutron IPv6 scenario tests.
>>
>> My question is, should we make this a generic tempest-slow job which can be
>> run either in the integrated-gate or at least in nova/neutron/cinder
>> consistently (I'm not sure if there are slow tests for just keystone or
>> glance)? I don't know if the other projects already have something like this
>> that they gate on. If so, a nova-specific job for nova changes is fine for
>> me.
>
> +1 on idea. As of now slow marked tests are from nova, cinder and
> neutron scenario tests and 2 API swift tests only [4]. I agree that
> making a generic job in tempest is better for maintainability. We can
> use existing job for that with below modification-
> -  We can migrate
> "legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend" job
> zuulv3 in tempest repo
> -  We can see if we can move migration tests out of it and use
> "nova-live-migration" job (in tempest check pipeline ) which is much
> better in live migration env setup and controlled by nova.
> -  then it can be name something like
> "tempest-scenario-multinode-lvm-multibackend".
> -  run this job in nova, cinder, neutron check pipeline instead of experimental.

Like this - https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:scenario-tests-job

That makes scenario job as generic with running all scenario tests
including slow tests with concurrency 2. I made few cleanup and moved
live migration tests out of it which is being run by
'nova-live-migration' job. Last patch making this job as voting on
tempest side.

If looks good, we can use this to run on project side pipeline as voting.

-gmann

>
> Another update on slow tests is that we are trying the possibility of
> taking back the slow tests in tempest-full with new job
> "tempest-full-parallel" [5]. Currently this job is n-v and if
> everything works fine in this new job then, we can make tempest-full
> job to run the slow tests are it used to do previously.
>
>>
>> [1] https://review.openstack.org/#/c/567697/
>> [2]
>> http://logs.openstack.org/97/567697/1/check/nova-slow/bedfafb/job-output.txt.gz#_2018-05-10_23_46_47_588138
>
> ..3 http://codesearch.openstack.org/?q=legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend&i=nope&files=&repos=
> ..4 https://github.com/openstack/tempest/search?utf8=%E2%9C%93&q=%22type%3D%27slow%27%22&type=
> ..5 https://github.com/openstack/tempest/blob/9c628189e798f46de8c4b9484237f4d6dc6ade7e/.zuul.yaml#L48
>
>
> -gmann
>
>>
>> --
>>
>> Thanks,
>>
>> Matt
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev