Open Stack

 ---- On Tue, 07 May 2019 08:25:11 +0900 Tim Burke <tim at swiftstack.com> wrote ----
 >            
 >      
 >      On 5/5/19 12:18 AM, Ghanshyam Mann       wrote:
 >                  Current integrated-gate jobs (tempest-full) is not so stable for various bugs specially timeout. We triedto improve it via filtering the slow tests in the separate tempest-slow job but the situation has not been improved much.We talked about the Ideas to make it more stable and fast for projects especially when failure is notrelated to each project. We are planning to split the integrated-gate template (only tempest-full job asfirst step) per related services. Idea:- Run only dependent service tests on project gate.          I love this plan already.
 >             - Tempest gate will keep running all the services tests as the integrated gate at a centeralized  place without any change in the current job.- Each project can run the below mentioned template. - All below template will be defined and maintained by QA team.           My biggest regret is that I couldn't figure out how to do this     myself. Much thanks to the QA team!
 >             I would like to know each 6 services which run integrated-gate jobs1."Integrated-gate-networking" (job to run on neutron gate) Tests to run in this template: neutron APIs , nova APIs,  keystone APIs ? All scenario currently running in tempest-full in the same way ( means non-slow and in serial)Improvement for neutron gate: exlcude the cinder API tests,  glance API tests, swift API tests,2."Integrated-gate-storage" (job to run on cinder gate, glance gate)Tests to run in this template: Cinder APIs , Glance APIs, Swift APIs, Nova APIs and All scenario currently running in tempest-full in the same way ( means non-slow and in serial)Improvement for cinder, glance gate: excluded the neutron APIs tests, Keystone APIs tests3. "Integrated-gate-object-storage" (job to run on swift gate)Tests to run in this template: Cinder APIs , Glance APIs, Swift APIs and All scenario currently running in tempest-full in the same way ( means non-slow and in serial)Improvement for swift gate: excluded the neutron APIs tests, - Keystone APIs tests, - Nova APIs tests.          This sounds great. My only question is why Cinder tests are still     included, but I trust that it's there for a reason and I'm just     revealing my own ignorance of Swift's consumers, however removed.

As Cinder use Swift as one of the backend, I think it is worth running cinder tests on the swift gate. But honestly saying I am covering the most possible broader coverage so that we do not lose any coverage among dependent services. Later we can always optimize this template more once we are very clear about the isolation of services.

 >             Note: swift does not run integrated-gate as of now.      

Yeah, Kota too brought this in QA team. We suggested to wait for the stability of integrated-gate (this mailing thread) and after that, swift can add the integrated-gate-* template.


>  Correct, and for all the reasons that you're seeking to address.       Some eight months ago I'd gotten tired of seeing spurious failures       that had nothing to do with Swift, and I was hard pressed to find       an instance where the tempest tests caught a regression or       behavior change that wasn't already caught by Swift's own       functional tests. In short, the signal-to-noise ratio for those       particular tests was low enough that a failure only told me "you       should leave a recheck comment," so I proposed       https://review.opendev.org/#/c/601813/ . There was also a side       benefit of having our longest-running job change from       legacy-tempest-dsvm-neutron-full (at 90-100 minutes) to       swift-probetests-centos-7 (at ~30 minutes), tightening developer       feedback loops.
 >      It sounds like this proposal addresses both concerns: by reducing       the scope of tests to what might actually exercise the Swift API       (if indirectly), the signal-to-noise ratio should be much better       and the wall-clock time will be reduced.

True, many other project gate faces a similar problem. Let's see how much this idea can improve integrated gate testing. Thanks for confirmation from the Swift side.

-gmann

 >      
 >             4. "Integrated-gate-compute" (job to run on Nova gate)tests to run is : Nova APIs, Cinder APIs , Glance APIs ?, neutron APIs and All scenario currently running in tempest-full in same way ( means non-slow and in serial)Improvement for Nova gate: excluded the swift APIs tests(not running in current job but in future, it might), Keystone API tests. 5. "Integrated-gate-identity" (job to run on keystone gate)Tests to run is : all as all project use keystone, we might need to run all tests as it is running in integrated-gate.But does keystone is being unsed differently by all services? if no then, is it enough to run only single service tests say Nova or neutron ?6. "Integrated-gate-placement" (job to run on placement gate)Tests to run in this template: Nova APIs tests, Neutron APIs tests + scenario tests + any new service depends on placement APIs Improvement for placement gate: excluded the  glance APIs tests, cinder APIs tests, swift APIs tests, keystone APIs testsThoughts on this approach?The important point is we must not lose the coverage of integrated testing per project. So I would like toget each project view if we are missing any dependency (proposed tests removal) in above proposed templates.          As far as Swift is aware, these dependencies seem accurate; at any     rate, *we* don't use anything other than Keystone, even by way of     another API. Further, Swift does not use particularly esoteric     Keysonte APIs; I would be OK with integrated-gate-identity not     exercising Swift's API with the assumption that some other (or     indeed, almost *any* other) service would likely exercise the parts     that we care about.
 >             - https:/etherpad.openstack.org/p/qa-train-ptg -gmann