Open Stack

Hi,

Thanks. I just send patch [1] to add this new job to Neutron failure rate Grafana dashboard.

[1] https://review.openstack.org/#/c/583870/

> Wiadomość napisana przez Ghanshyam Mann <gmann at ghanshyammann.com> w dniu 19.07.2018, o godz. 06:55:
> 
>> On Sun, May 13, 2018 at 1:20 PM, Ghanshyam Mann <gmann at ghanshyammann.com> wrote: 
>>> On Fri, May 11, 2018 at 10:45 PM, Matt Riedemann <mriedemos at gmail.com> wrote: 
>>>> The tempest-full job used to run API and scenario tests concurrently, and if 
>>>> you go back far enough I think it also ran slow tests. 
>>>> 
>>>> Sometime in the last year or so, the full job was changed to run the 
>>>> scenario tests in serial and exclude the slow tests altogether. So the API 
>>>> tests run concurrently first, and then the scenario tests run in serial. 
>>>> During that change, some other tests were identified as 'slow' and marked as 
>>>> such, meaning they don't get run in the normal tempest-full job. 
>>>> 
>>>> There are some valuable scenario tests marked as slow, however, like the 
>>>> only encrypted volume testing we have in tempest is marked slow so it 
>>>> doesn't get run on every change for at least nova. 
>>> 
>>> Yes, basically slow tests were selected based on 
>>> https://ethercalc.openstack.org/nu56u2wrfb2b and there were frequent 
>>> gate failure for heavy tests mainly from ssh checks so we tried to 
>>> mark more tests as slow. 
>>> I agree that some of them are not really slow at least in today situation. 
>>> 
>>>> 
>>>> There is only one job that can be run against nova changes which runs the 
>>>> slow tests but it's in the experimental queue so people forget to run it. 
>>> 
>>> Tempest job "legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend" 
>>> run those slow tests including migration and LVM  multibackend tests. 
>>> This job runs on tempest check pipeline and experimental (as you 
>>> mentioned) on nova and cinder [3]. We marked this as n-v to check its 
>>> stability and now it is good to go as voting on tempest. 
>>> 
>>>> 
>>>> As a test, I've proposed a nova-slow job [1] which only runs the slow tests 
>>>> and only the compute API and scenario tests. Since there currently no 
>>>> compute API tests marked as slow, it's really just running slow scenario 
>>>> tests. Results show it runs 37 tests in about 37 minutes [2]. The overall 
>>>> job runtime was 1 hour and 9 minutes, which is on average less than the 
>>>> tempest-full job. The nova-slow job is also running scenarios that nova 
>>>> patches don't actually care about, like the neutron IPv6 scenario tests. 
>>>> 
>>>> My question is, should we make this a generic tempest-slow job which can be 
>>>> run either in the integrated-gate or at least in nova/neutron/cinder 
>>>> consistently (I'm not sure if there are slow tests for just keystone or 
>>>> glance)? I don't know if the other projects already have something like this 
>>>> that they gate on. If so, a nova-specific job for nova changes is fine for 
>>>> me. 
>>> 
>>> +1 on idea. As of now slow marked tests are from nova, cinder and 
>>> neutron scenario tests and 2 API swift tests only [4]. I agree that 
>>> making a generic job in tempest is better for maintainability. We can 
>>> use existing job for that with below modification- 
>>> -  We can migrate 
>>> "legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend" job 
>>> zuulv3 in tempest repo 
>>> -  We can see if we can move migration tests out of it and use 
>>> "nova-live-migration" job (in tempest check pipeline ) which is much 
>>> better in live migration env setup and controlled by nova. 
>>> -  then it can be name something like 
>>> "tempest-scenario-multinode-lvm-multibackend". 
>>> -  run this job in nova, cinder, neutron check pipeline instead of experimental. 
>> 
>> Like this - https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:scenario-tests-job 
>> 
>> That makes scenario job as generic with running all scenario tests 
>> including slow tests with concurrency 2. I made few cleanup and moved 
>> live migration tests out of it which is being run by 
>> 'nova-live-migration' job. Last patch making this job as voting on 
>> tempest side. 
>> 
>> If looks good, we can use this to run on project side pipeline as voting. 
> 
> Update on this thread:
> Old Scenario  job "legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend" has been migrated to Tempest as new job named "tempest-scenario-all" job[1] 
> 
> Changes from old job to new job:
> - This new job will run all the scenario tests including slow with lvm multibackend. Same as old job
> -  Executed the live migration API tests out of it. Live migration API tests runs on separate  nova job "nova-live-migration".
> - This new job runs as voting on Tempest check and gate pipeline.
> 
> This is ready to use for cross project also. i have pushed the patch to nova, neutron, cinder to use this new job[3] and remove "legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend" from project-config[4]. 
> 
> Let me know your feedback on proposed patches. 
> 
> [1] http://git.openstack.org/cgit/openstack/tempest/tree/.zuul.yaml#n147
> [2] https://review.openstack.org/#/q/topic:run-tempest-scenario-all-job+(status:open+OR+status:merged)
> [3] https://review.openstack.org/#/q/topic:run-tempest-scenario-all-job+(status:open+OR+status:merged)
> [4] https://review.openstack.org/#/q/topic:drop-legacy-scenario-job+(status:open+OR+status:merged)
> 
>> 
>> -gmann 
>> 
>>> 
>>> Another update on slow tests is that we are trying the possibility of 
>>> taking back the slow tests in tempest-full with new job 
>>> "tempest-full-parallel" [5]. Currently this job is n-v and if 
>>> everything works fine in this new job then, we can make tempest-full 
>>> job to run the slow tests are it used to do previously. 
>>> 
>>>> 
>>>> [1] https://review.openstack.org/#/c/567697/ 
>>>> [2] 
>>>> http://logs.openstack.org/97/567697/1/check/nova-slow/bedfafb/job-output.txt.gz#_2018-05-10_23_46_47_588138 
>>> 
>>> ..3 http://codesearch.openstack.org/?q=legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend&i=nope&files=&repos= 
>>> ..4 https://github.com/openstack/tempest/search?utf8=%E2%9C%93&q=%22type%3D%27slow%27%22&type= 
>>> ..5 https://github.com/openstack/tempest/blob/9c628189e798f46de8c4b9484237f4d6dc6ade7e/.zuul.yaml#L48 
>>> 
>>> 
>>> -gmann 
>>> 
>>>> 
>>>> -- 
>>>> 
>>>> Thanks, 
>>>> 
>>>> Matt 
>>>> 
>>>> __________________________________________________________________________ 
>>>> OpenStack Development Mailing List (not for usage questions) 
>>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe 
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
>> 
> 
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

— 
Slawek Kaplonski
Senior software engineer
Red Hat