Open Stack

Patrole is emerging as a good tool for RBAC testing. AT&T already running it on their production cloud and
we have got a good amount of interest/feedback from other operators. 

We had few discussions regarding the Patrole testing improvement during PTG among QA, Nova, Keystone team.
I am writing the summary of those discussions below and would like to get the opinion from Felipe & Sergey also.

1. How to improve the Patrole testing time:
Currently Patrole test perform the complete API operaion which takes time and make Patrole testing
very long. Patrole is responsible to test the policies only so does not need to wait for API complete operation
to be completed. 
John has a good idea to handle that via flag. If that flag is enabled (per service and disabled by default) then
oslo.policy can return some different error code on success (other than 403). The API can return the response
with that error code which can be treated as pass case in Patrole.
Morgan raises a good point on making it per API call than global. We can do that as next step and let's
start with the global flag per service as of now?
- https://etherpad.openstack.org/p/ptg-train-xproj-nova-keystone 

Another thing we should improve in current Patrole jobs is to separate the jobs per service. Currently, all 5 services
are installed and run in a single job. Running all on Patrole gate is good but the project side gate does not need to run
any other service tests. For example, patrole-keystone which can install the only keystone and run only
keystone tests. This way project can reuse the patrole jobs only and does not need to prepare a separate job.

2. How to run patrole tests with all negative, positive combination for all scope + defaults roles combinations:
  - Current jobs patrole-admin/member/reader are able to test the negative pattern. For example:
     patrole-member job tests the admin APIs in a negative way and make sure test is passed only if member
     role gets 403. 
  - As we have scope_type support also we need to extend the jobs to run for all 9 combinations of 3 scopes
    (system, project, domain) and 3 roles(admin, member, reader).
  - option1:  running 9 different jobs with each combination as we do have currently
     for admin, member, reader role. The issue with this approach is gate will take a lot of time to
     run these 9 jobs separately. 
  - option2: Run all the 9 combinations in a single job with running the tests in the loop with different
     combination of scope_roles. This might require the current config option [role] to convert to list type
     and per service so that the user can configure what all default roles are available for corresponding service.
     This option can save a lot of time to avoid devstack installation time as compared to 9 different jobs option.

-gmann