On Mon, Jan 24, 2022 at 4:50 PM Takashi Kajinami <tkajinam@redhat.com> wrote:




On Sat, Jan 22, 2022 at 8:57 AM Ghanshyam Mann <gmann@ghanshyammann.com> wrote:
 ---- On Thu, 20 Jan 2022 14:41:00 -0600 Mark Goddard <mark@stackhpc.com> wrote ----
 > On Thu, 20 Jan 2022 at 19:55, Ghanshyam Mann <gmann@ghanshyammann.com> wrote:
 > >
 > >  ---- On Thu, 20 Jan 2022 13:36:53 -0600 Mark Goddard <mark@stackhpc.com> wrote ----
 > >  > On Thu, 20 Jan 2022 at 18:40, Ghanshyam Mann <gmann@ghanshyammann.com> wrote:
 > >  > >
 > >  > >
 > >  > >  ---- On Thu, 20 Jan 2022 03:35:33 -0600 Mark Goddard <mark@stackhpc.com> wrote ----
 > >  > >  > On Wed, 19 Jan 2022 at 16:12, Ghanshyam Mann <gmann@ghanshyammann.com> wrote:
 > >  > >  > >
 > >  > >  > >  ---- On Wed, 19 Jan 2022 04:35:53 -0600 Mark Goddard <mark@stackhpc.com> wrote ----
 > >  > >  > >  > Hi,
 > >  > >  > >  >
 > >  > >  > >  > If you haven't been paying close attention, it would be easy to miss
 > >  > >  > >  > some of the upcoming RBAC changes which will have an impact on
 > >  > >  > >  > deployment projects. I thought I'd start a thread so that we can share
 > >  > >  > >  > how we are approaching this, get answers to open questions, and
 > >  > >  > >  > ideally all end up with a fairly consistent approach.
 > >  > >  > >  >
 > >  > >  > >  > The secure RBAC work has a long history, and continues to evolve.
 > >  > >  > >  > According to [1], we should start to see some fairly substantial
 > >  > >  > >  > changes over the next few releases. That spec is fairly long, but
 > >  > >  > >  > worth a read.
 > >  > >  > >  >
 > >  > >  > >  > In the yoga timeline [2], there is one change in particular that has
 > >  > >  > >  > an impact on deployment projects, "3. Keystone enforces scope by
 > >  > >  > >  > default". After this change, all of the deprecated policies that many
 > >  > >  > >  > still rely on in Keystone will be removed.
 > >  > >  > >  >
 > >  > >  > >  > In kolla-ansible, we have an etherpad [5] with some notes, questions
 > >  > >  > >  > and half-baked plans. We made some changes in Xena [3] to use system
 > >  > >  > >  > scope in some places when interacting with system APIs in Ansible
 > >  > >  > >  > tasks.
 > >  > >  > >  >
 > >  > >  > >  > The next change we have staged is to add the service role to all
 > >  > >  > >  > service users [4], in preparation for [2].
 > >  > >  > >  >
 > >  > >  > >  > Question: should the role be added with system scope or in the
 > >  > >  > >  > existing service project? The obvious main use for this is token
 > >  > >  > >  > validation, which seems to allow system or project scope.
 > >  > >  > >  >
 > >  > >  > >  > We anticipate that some service users may still require some
 > >  > >  > >  > project-scoped roles, e.g. when creating resources for octavia. We'll
 > >  > >  > >  > deal with those on a case by case basis.
 > >  > >  > >
 > >  > >  > > Service roles are planned for phase2 which is Z release[1]. The Idea here is
 > >  > >  > > service to service communication will happen with 'service' role (which keystone
 > >  > >  > > need to implement yet) and end users will keep using the what ever role
 > >  > >  > > is default (or overridden in policy file) which can be project or system scoped
 > >  > >  > > depends on the APIs.
 > >  > >  > >
 > >  > >  > > So at the end service-service APIs policy default will looks like
 > >  > >  > >
 > >  > >  > >  '(role:admin and system:network and project_id:%(project_id)s) or (role:service and project_name:service)'
 > >  > >  > >
 > >  > >  > > Say nova will use that service role to communicate to cinder and cinder policy will pass
 > >  > >  > > as service role is in OR in default policy.
 > >  > >  > >
 > >  > >  > > But let's see how they are going to be and if any challenges when we will implement
 > >  > >  > > it in Z cycle.
 > >  > >  >
 > >  > >  > I'm not 100% on our reasoning for using the service role in yoga (I
 > >  > >  > wasn't in the discussion when we made the switch, although John
 > >  > >  > Garbutt was), although I can provide at least one reason.
 > >  > >  >
 > >  > >  > Currently, we have a bunch of service users doing things like keystone
 > >  > >  > token validation using the admin role in the service project. If we
 > >  > >  > enforce scopes & new defaults in keystone, this will no longer work,
 > >  > >  > due to the default policy:
 > >  > >  >
 > >  > >  > identity:validate_token: (role:reader and system_scope:all) or
 > >  > >  > rule:service_role or rule:token_subject
 > >  > >  >
 > >  > >  > Now we could go and assign system-reader to all these users, but if
 > >  > >  > the end goal is to give them all the service role, and that allows
 > >  > >  > token validation, then to me that seems like a better path.
 > >  > >  >
 > >  > >  > Currently, we're creating the service role during deploy & upgrade,
 > >  > >  > then assigning it to users. Keystone is supposed to create the service
 > >  > >  > role in yoga, so we can eventually drop that part.
 > >  > >  >
 > >  > >  > Does this seem reasonable? Is keystone still on track to create the
 > >  > >  > service role in yoga?
 > >  > >
 > >  > > I think this is a reasonable plan and once we have service roles implemented
 > >  > > in keystone as well as in all the services to request other service APIs then
 > >  > > deployment project (Kolla here) can update them from system_reader to
 > >  > > actual service role.
 > >  >
 > >  > To be clear, I am proposing to skip system-reader, and go straight to
 > >  > the service role in yoga.
 > >
 > > But that would not be doable until services implement service roles which is
 > > Yoga cycle target for keystone and Z cyle target for other projects. Or you mean
 > > to re-consider to target the service role for all projects also in Yoga so that
 > > deployment projects can go with service role directly?
 >
 > Our current plan is to add the service role to all service users in
 > yoga. This will allow keystone token validation to work when keystone
 > drops the deprecated policies.
 >
 > We will not remove the admin role from service users in the service
 > project during yoga. This will allow projects other than keystone to
 > continue to work as before.
 >
 > At some later point, we will remove the admin role from service users
 > in the service project, hopefully relying on the service role for most
 > service-service communication. There may be other roles we need to
 > assign in order to drop admin, but we'll assess that as we go.
 >
 > Hopefully that's a bit more of a clear picture, and it seems sensible?

+1, sounds good to me. Hopefully we will get in better shape by Z release
when all (or maximum) services will be migrated to new RBAC. Till than
your plan sounds reasonable.

-gmann

I'll follow the same approach in Puppet OpenStack and will add the project-scoped 'service' role
to each service user by default. IIUC This is consistent with the current devstack which assigns
the project-scoped service role to each service user, so I expect this approach will be tested
in dsvm jobs [1].

The same was already implemented in TripleO by [2]
 

I've spent some time going through all the keystone credentials managed by puppet modules
and I recorded my observations in my working note.

https://etherpad.opendev.org/p/puppet-secure-rbac#L122

If my observation is correct, credentials in the following sections/services are used to access APIs
which are not allowed for the service role and require an additional privilege like system-reader
when Keystone is running with only new policies and scope enforcement.

glance [oslo_limit]
 This calls get limits API to obtain the limit for the  project where a resource is being created.
 This requires a system-reader.

nova [keystone]
 This calls get project API to verify the project id passed in flavor access or quota sets.
 This operation requires a system-reader.

swift [s3api]
 This calls get EC2 credential API to cache credentials for the request user.
 This requires a system-reader.

swift [ceilometer]
 This calls list project API when ignore_projects is set, to look up these projects.
 This requires a system-reader.


 >
 > >
 > > -gmann
 > >
 > >  >
 > >  > >
 > >  > > And yes that can be done for token validation as well as
 > >  > > the service-to-service API calls for example nova to cinder or neutron to nova
 > >  > > APIs call. I do not think we can migrate everything (service tokens) together for all
 > >  > > the services in deployment projects until all these services are ready with the 'service'
 > >  > > role implementation (implementation means changing their default roles
 > >  > > to add 'service' role for service-to-service APIs).
 > >  > >
 > >  > > Regarding the keystone track on service role work in Yoga or not, I do not
 > >  > > have clear answer may be Lance or keystone team can answer it. But Lance
 > >  > > has spec up[1] but not yet merged.
 > >  > >
 > >  > > [1] https://review.opendev.org/c/openstack/keystone-specs/+/818616
 > >  > >
 > >  > > -gmann
 > >  > >
 > >  > >  >
 > >  > >  > >
 > >  > >  > >  >
 > >  > >  > >  > In anticipation of keystone setting enforce_scope=True and removing
 > >  > >  > >  > old default policies (which I assume effectively removes
 > >  > >  > >  > enforce_new_defaults?), we will set this in kolla-ansible, and try to
 > >  > >  > >  > deal with any fallout. Hopefully the previous work will make this
 > >  > >  > >  > minimal.
 > >  > >  > >  >
 > >  > >  > >  > How does that line up with other projects' approaches? What have we missed?
 > >  > >  > >
 > >  > >  > > Yeah, we want users/deployment projects/horizon etc to use the new policy from
 > >  > >  > > keystone as first and we will see feedback how they are (good, bad, really bad) from
 > >  > >  > > usage perspective. Why we choose keystone is, because new policy are there since
 > >  > >  > > many cycle and ready to use. Other projects needs to work their policy as per new
 > >  > >  > > SRBAC design/direction (for example nova needs to modify their policy before we ask
 > >  > >  > > users to use new policy and work is under progress[2]).
 > >  > >  > >
 > >  > >  > > I think trying in kolla will be good way to know if we can move to keystone's new policy
 > >  > >  > > completely in yoga.
 > >  > >  >
 > >  > >  > We have a scope-enforcing preview patch [1], and it's passing our base
 > >  > >  > set of tests. I have another that triggers all of the jobs.
 > >  > >  >
 > >  > >  > [1] https://review.opendev.org/c/openstack/kolla-ansible/+/825406
 > >  > >  > >
 > >  > >  > > [1] https://opendev.org/openstack/governance/src/branch/master/goals/selected/consistent-and-secure-rbac.rst#z-release-timeline
 > >  > >  > > [2] https://blueprints.launchpad.net/nova/+spec/policy-defaults-refresh-2
 > >  > >  > >
 > >  > >  > > -gmann
 > >  > >  > >
 > >  > >  > >  >
 > >  > >  > >  > Mark
 > >  > >  > >  >
 > >  > >  > >  > [1] https://opendev.org/openstack/governance/src/branch/master/goals/selected/consistent-and-secure-rbac.rst
 > >  > >  > >  > [2] https://opendev.org/openstack/governance/src/branch/master/goals/selected/consistent-and-secure-rbac.rst#yoga-timeline-7th-mar-2022
 > >  > >  > >  > [3] https://opendev.org/openstack/kolla-ansible/commit/2e933dceb591c3505f35c2c1de924f3978fb81a7
 > >  > >  > >  > [4] https://review.opendev.org/c/openstack/kolla-ansible/+/815577
 > >  > >  > >  > [5] https://etherpad.opendev.org/p/enabling-system-scope-in-kolla-ansible
 > >  > >  > >  >
 > >  > >  > >  >
 > >  > >  >
 > >  > >  >
 > >  >
 > >  >
 >
 >