Open Stack

Hey all,

We met today to walk through testing approaches for secure RBAC. The
meeting was hosted on bluejeans and recorded [0]. The notes are available
on etherpad [1]. Below is a summary with what we talked about and
introduces a plan for testing moving forward.

Secure RBAC provides a framework and tooling to group and generalize
authorization around common clouds users. Three scopes (system, domain,
project) and three default roles (admin, member, reader) resulting in nine
different personas (system-admin, system-member, system-reader,
domain-admin, domain-member, domain-reader, project-admin, project-member,
and project-reader) that cover the most common requests for customized
policies for deployers.

To implement these new personas into default policies, we need to test
them. We currently have three testing approaches.

The first approach is for each service to implement functional/unit
protection tests in-tree. This approach doesn't have a direct dependency on
keystone, making it self-contained. Instead of relying on tokens from
keystone and keystonemiddleware putting authorization information into
request headers, the functional tests would need to simulate that for the
various personas. Each service will need to approach these tests in
whatever was is best for their project, and implementations will likely be
different. For example, nova's functional tests use mocked context objects
to represent the various personas and expose them to underlying tests to
use in functional API requests [2]. Placement uses gabbi to tests APIs,
bypasses auth_token middleware, and sets the appropriate request headers to
simulate what keystonemiddleware does before handing the request to
oslo.context [3]. These tests are unlikely to be reused by deployers to
verify functionality in an actual deployment.

The second approach is to implement protection testing as tempest tests.
Keystone recently experimented with this approach [4]. This approach
provides robust testing and requires building out new tests using tempest
plugins to verify the new policies. The tests are bypassed unless a feature
flag is enabled in tempest [5]. This approach contains both positive and
negative testing. It's also easier for deployers to re-use if they want to
verify their deployments policies adhere to secure RBAC. The current
implementation separates protection testing in tempest from tempest API
testing, so there is some duplication across tests. Additionally, test run
times are a concern. Keystone's tempest protection tests take about 45
minutes to run. We expect those times to be higher for services with larger
APIs.

The third approach is to overhaul the existing tempest tests to support the
nine different personas, or at least the ones that are applicable to the
tests. Ghanshyam is working on this approach with nova and tempest [6].
Like the second approach, this approach requires tempest configuration
changes to tell tempest how to wire up the appropriate clients. While this
approach uses tempest, it doesn't duplicate protection tests and existing
API tests because we're incorporating the protection testing into the
existing API tests. The downside with this approach is that most tempest
tests are positive tests, so we would potentially have a large gap to fill
with negative protection testing (e.g., ensuring project-readers can't
delete volumes or instances).

Today's goal was to find out how we can realistically test secure RBAC in
Wallaby. The first and second approaches are likely the most robust, but
they require a lot of code. The third approach focuses on re-usability but
leaves testing gaps.

The proposed solution was a combination of approach #1 and #3. Services
implementing secure RBAC should work with Ghanshyam or myself to introduce
the necessary tempest plumbing to wire up clients using the correct persona
based on tempest configuration options. Most of this code is enacapsulated
in tempest libs or clients already and just needs to be consumed in the API
test base classes. Additional negative or positive testing should be done
as in-tree protection tests, where the tests are responsible for mocking
context objects that represent the nine personas. Once we're past the
migration phase or supporting deprecated policies, we should seriously
consider adding support for approach #2 by porting functional tests to
formal tempest protection tests.

We realize the proposed solution isn't a silver bullet. There are
downsides. One of them is that protection testing will potentially be in
two separate places. Conversely, it gives us a relatively low bar for
converting tempest to effectively test secure RBAC without it being
painfully exhaustive, and leaving developers with an option to supply
additional testing as they see fit.

The majority of the folks on the call were from cinder, but we also want to
sit down with other groups to discuss this approach with them. Ghanshyam is
going to set up a time with the neutron developers in the next couple of
weeks. I'm going to follow up with cinder in a week or two to see how
things are going. We'd like to adjust the current proposal as early as
possible, if needed.

We realize how hard it is to satisfy all testing requirements and be
realistic, but how do people feel about the current proposal? Does it lower
the bar such that it's realistic for Wallaby without sacrificing testing?

[0] https://bluejeans.com/s/o2sMXK51m7E/
[1] https://etherpad.opendev.org/p/wallaby-secure-rbac-testing
[2]
https://opendev.org/openstack/nova/src/branch/master/nova/tests/unit/policies/base.py#L41-L121
[3]
https://review.opendev.org/#/c/760235/2/placement/tests/functional/gabbits/aggregate-system-admin-policy.yaml@9
[4] https://review.opendev.org/#/c/686305/\
[5]
https://review.opendev.org/#/c/686305/47/keystone_tempest_plugin/config.py@28
[6] https://review.opendev.org/#/c/740122/5

On Tue, Nov 10, 2020 at 12:50 PM Lance Bragstad <lbragstad at gmail.com> wrote:

> Hey folks,
>
> We held the meeting today and we started capturing some notes in an
> etherpad [0]. Most of today's discussion was focused on introducing
> context. We didn't come to a definitive answer as to which testing strategy
> is best.
>
> We're going to have another meeting on Monday, November 16th (next Monday)
> at 16:00 UTC. We're also going to use the same conferencing room Ghanshyam
> sent out in the initial email [1].
>
> Feel free to add thoughts to the etherpad [0] before hand.
>
> [0] https://etherpad.opendev.org/p/wallaby-secure-rbac-testing
> [1] https://bluejeans.com/749897818
>
> On Tue, Nov 3, 2020 at 1:25 PM Ghanshyam Mann <gmann at ghanshyammann.com>
> wrote:
>
>> Hello Everyone,
>>
>> To continue the discussion on how projects should test the new RBAC
>> (scope as well as the new defaults combination),
>> we will host a call on bluejeans on Tuesday 10th Nov 16.00 UTC.
>>
>> Lance has set up the room for that: https://bluejeans.com/749897818
>>
>> Feel free to join if you are interested in that or would like to help
>> with this effort.
>>
>> -gmann
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20201116/7639a251/attachment-0001.html>