Open Stack

 ---- On Tue, 28 May 2019 22:21:44 +0900 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
 >  ---- On Mon, 27 May 2019 18:43:35 +0900 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ---- 
 >  >  ---- On Tue, 07 May 2019 07:06:23 +0900 Morgan Fainberg <morgan.fainberg at gmail.com> wrote ----  
 >  >  >   
 >  >  >   
 >  >  > On Sun, May 5, 2019 at 12:19 AM Ghanshyam Mann <gmann at ghanshyammann.com> wrote:  
 >  >  >   
 >  >  > For the "Integrated-gate-identity", I have a slight worry that we might lose some coverage with this change. I am unsure of how varied the use of Keystone is outside of KeystoneMiddleware (i.e. token validation) consumption that all services perform, Heat (not part of the integrated gate) and it's usage of Trusts, and some newer emerging uses such as "look up limit data" (potentially in Train, would be covered by Nova). Worst case, we could run all the integrated tests for Keystone changes (at least initially) until we have higher confidence and minimize the tests once we have a clearer audit of how the services use Keystone. The changes would speed up/minimize the usage for the other services directly and Keystone can follow down the line.  
 >  >  > I want to be as close to 100% sure we're not going to suddenly break everyone because of some change we land. Keystone fortunately and unfortunately sits below most other services in an OpenStack deployment and is heavily relied throughout almost every single request.  
 >  >  > --Morgan   
 >  >   
 >  >   
 >  > Thanks Morgan. That was what we were worried during PTG discussion. I agree with your point about not to lose coverage and first get to know how Keystone is being used by each service. Let's keep running the all service tests for keystone gate as of now and later we can shorten the tests run based on the clarity of usage.   
 >  
 > We can disable the ssh validation for  "Integrated-gate-identity" which keystone does not need to care about. This can save the keystone gate for ssh timeout failure.  
 >  
 > -gmann 
 >  
 >  >   
 >  > -gmann   
 >  >   
 >  >   
 >  >  > Current integrated-gate jobs (tempest-full) is not so stable for various bugs specially timeout. We tried  
 >  >  >  to improve it via filtering the slow tests in the separate tempest-slow job but the situation has not been improved much.  
 >  >  >    
 >  >  >  We talked about the Ideas to make it more stable and fast for projects especially when failure is not  
 >  >  >  related to each project. We are planning to split the integrated-gate template (only tempest-full job as  
 >  >  >  first step) per related services.   
 >  >  >    
 >  >  >  Idea:  
 >  >  >  - Run only dependent service tests on project gate.  
 >  >  >  - Tempest gate will keep running all the services tests as the integrated gate at a centeralized  place without any change in the current job.  
 >  >  >  - Each project can run the below mentioned template.   
 >  >  >  - All below template will be defined and maintained by QA team.   
 >  >  >    
 >  >  >  I would like to know each 6 services which run integrated-gate jobs  
 >  >  >    
 >  >  >  1."Integrated-gate-networking" (job to run on neutron gate)  
 >  >  >   Tests to run in this template: neutron APIs , nova APIs,  keystone APIs ? All scenario currently running in tempest-full in the same way ( means non-slow and in serial)  
 >  >  >  Improvement for neutron gate: exlcude the cinder API tests,  glance API tests, swift API tests,  
 >  >  >    
 >  >  >  2."Integrated-gate-storage" (job to run on cinder gate, glance gate)  
 >  >  >  Tests to run in this template: Cinder APIs , Glance APIs, Swift APIs, Nova APIs and All scenario currently running in tempest-full in the same way ( means non-slow and in serial)  
 >  >  >  Improvement for cinder, glance gate: excluded the neutron APIs tests, Keystone APIs tests  
 >  >  >    
 >  >  >  3. "Integrated-gate-object-storage" (job to run on swift gate)  
 >  >  >  Tests to run in this template: Cinder APIs , Glance APIs, Swift APIs and All scenario currently running in tempest-full in the same way ( means non-slow and in serial)  
 >  >  >  Improvement for swift gate: excluded the neutron APIs tests, - Keystone APIs tests, - Nova APIs tests.  
 >  >  >  Note: swift does not run integrated-gate as of now.  
 >  >  >    
 >  >  >  4. "Integrated-gate-compute" (job to run on Nova gate)  
 >  >  >  tests to run is : Nova APIs, Cinder APIs , Glance APIs ?, neutron APIs and All scenario currently running in tempest-full in same way ( means non-slow and in serial)  
 >  >  >  Improvement for Nova gate: excluded the swift APIs tests(not running in current job but in future, it might), Keystone API tests.   
 >  >  >    
 >  >  >  5. "Integrated-gate-identity" (job to run on keystone gate)  
 >  >  >  Tests to run is : all as all project use keystone, we might need to run all tests as it is running in integrated-gate.  
 >  >  >  But does keystone is being unsed differently by all services? if no then, is it enough to run only single service tests say Nova or neutron ?  
 >  >  >    
 >  >  >  6. "Integrated-gate-placement" (job to run on placement gate)  
 >  >  >  Tests to run in this template: Nova APIs tests, Neutron APIs tests + scenario tests + any new service depends on placement APIs  
 >  >  >   Improvement for placement gate: excluded the  glance APIs tests, cinder APIs tests, swift APIs tests, keystone APIs tests  
 >  >  >   

I have prepared the new template for integrated gate testing[1] and tested in DNM patch [2].
You can observe ~20 min less time on new jobs(except compute one). But the main thing is will improve
the stability of gate.

Once they are merged, I will propose the patch to replace those template on the projects gate.

NOTE: Along with APIs tests, I have back listed  the non-dependent scenario tests also.

[1] https://review.opendev.org/#/q/topic:refactor-integrated-gate-testing+(status:open+OR+status:merged)
[2] https://review.opendev.org/#/c/669313/

-gmann

 
 >  >  >  Thoughts on this approach?  
 >  >  >    
 >  >  >  The important point is we must not lose the coverage of integrated testing per project. So I would like to  
 >  >  >  get each project view if we are missing any dependency (proposed tests removal) in above proposed templates.  
 >  >  >    
 >  >  >  - https://etherpad.openstack.org/p/qa-train-ptg   
 >  >  >    
 >  >  >  -gmann  
 >  >  >    
 >  >  >    
 >  >  >   
 >  >  
 >