Hey all, We met today to walk through testing approaches for secure RBAC. The meeting was hosted on bluejeans and recorded [0]. The notes are available on etherpad [1]. Below is a summary with what we talked about and introduces a plan for testing moving forward. Secure RBAC provides a framework and tooling to group and generalize authorization around common clouds users. Three scopes (system, domain, project) and three default roles (admin, member, reader) resulting in nine different personas (system-admin, system-member, system-reader, domain-admin, domain-member, domain-reader, project-admin, project-member, and project-reader) that cover the most common requests for customized policies for deployers. To implement these new personas into default policies, we need to test them. We currently have three testing approaches. The first approach is for each service to implement functional/unit protection tests in-tree. This approach doesn't have a direct dependency on keystone, making it self-contained. Instead of relying on tokens from keystone and keystonemiddleware putting authorization information into request headers, the functional tests would need to simulate that for the various personas. Each service will need to approach these tests in whatever was is best for their project, and implementations will likely be different. For example, nova's functional tests use mocked context objects to represent the various personas and expose them to underlying tests to use in functional API requests [2]. Placement uses gabbi to tests APIs, bypasses auth_token middleware, and sets the appropriate request headers to simulate what keystonemiddleware does before handing the request to oslo.context [3]. These tests are unlikely to be reused by deployers to verify functionality in an actual deployment. The second approach is to implement protection testing as tempest tests. Keystone recently experimented with this approach [4]. This approach provides robust testing and requires building out new tests using tempest plugins to verify the new policies. The tests are bypassed unless a feature flag is enabled in tempest [5]. This approach contains both positive and negative testing. It's also easier for deployers to re-use if they want to verify their deployments policies adhere to secure RBAC. The current implementation separates protection testing in tempest from tempest API testing, so there is some duplication across tests. Additionally, test run times are a concern. Keystone's tempest protection tests take about 45 minutes to run. We expect those times to be higher for services with larger APIs. The third approach is to overhaul the existing tempest tests to support the nine different personas, or at least the ones that are applicable to the tests. Ghanshyam is working on this approach with nova and tempest [6]. Like the second approach, this approach requires tempest configuration changes to tell tempest how to wire up the appropriate clients. While this approach uses tempest, it doesn't duplicate protection tests and existing API tests because we're incorporating the protection testing into the existing API tests. The downside with this approach is that most tempest tests are positive tests, so we would potentially have a large gap to fill with negative protection testing (e.g., ensuring project-readers can't delete volumes or instances). Today's goal was to find out how we can realistically test secure RBAC in Wallaby. The first and second approaches are likely the most robust, but they require a lot of code. The third approach focuses on re-usability but leaves testing gaps. The proposed solution was a combination of approach #1 and #3. Services implementing secure RBAC should work with Ghanshyam or myself to introduce the necessary tempest plumbing to wire up clients using the correct persona based on tempest configuration options. Most of this code is enacapsulated in tempest libs or clients already and just needs to be consumed in the API test base classes. Additional negative or positive testing should be done as in-tree protection tests, where the tests are responsible for mocking context objects that represent the nine personas. Once we're past the migration phase or supporting deprecated policies, we should seriously consider adding support for approach #2 by porting functional tests to formal tempest protection tests. We realize the proposed solution isn't a silver bullet. There are downsides. One of them is that protection testing will potentially be in two separate places. Conversely, it gives us a relatively low bar for converting tempest to effectively test secure RBAC without it being painfully exhaustive, and leaving developers with an option to supply additional testing as they see fit. The majority of the folks on the call were from cinder, but we also want to sit down with other groups to discuss this approach with them. Ghanshyam is going to set up a time with the neutron developers in the next couple of weeks. I'm going to follow up with cinder in a week or two to see how things are going. We'd like to adjust the current proposal as early as possible, if needed. We realize how hard it is to satisfy all testing requirements and be realistic, but how do people feel about the current proposal? Does it lower the bar such that it's realistic for Wallaby without sacrificing testing? [0] https://bluejeans.com/s/o2sMXK51m7E/ [1] https://etherpad.opendev.org/p/wallaby-secure-rbac-testing [2] https://opendev.org/openstack/nova/src/branch/master/nova/tests/unit/policie... [3] https://review.opendev.org/#/c/760235/2/placement/tests/functional/gabbits/a... [4] https://review.opendev.org/#/c/686305/\ [5] https://review.opendev.org/#/c/686305/47/keystone_tempest_plugin/config.py@2... [6] https://review.opendev.org/#/c/740122/5 On Tue, Nov 10, 2020 at 12:50 PM Lance Bragstad <lbragstad@gmail.com> wrote:

Hey folks,

We held the meeting today and we started capturing some notes in an etherpad [0]. Most of today's discussion was focused on introducing context. We didn't come to a definitive answer as to which testing strategy is best.

We're going to have another meeting on Monday, November 16th (next Monday) at 16:00 UTC. We're also going to use the same conferencing room Ghanshyam sent out in the initial email [1].

Feel free to add thoughts to the etherpad [0] before hand.

[0] https://etherpad.opendev.org/p/wallaby-secure-rbac-testing [1] https://bluejeans.com/749897818

On Tue, Nov 3, 2020 at 1:25 PM Ghanshyam Mann <gmann@ghanshyammann.com> wrote:

...

Hello Everyone,

To continue the discussion on how projects should test the new RBAC (scope as well as the new defaults combination), we will host a call on bluejeans on Tuesday 10th Nov 16.00 UTC.

Lance has set up the room for that: https://bluejeans.com/749897818

Feel free to join if you are interested in that or would like to help with this effort.

-gmann