<div dir="ltr"><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jan 22, 2022 at 8:57 AM Ghanshyam Mann <<a href="mailto:gmann@ghanshyammann.com">gmann@ghanshyammann.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> ---- On Thu, 20 Jan 2022 14:41:00 -0600 Mark Goddard <<a href="mailto:mark@stackhpc.com" target="_blank">mark@stackhpc.com</a>> wrote ----<br>
> On Thu, 20 Jan 2022 at 19:55, Ghanshyam Mann <<a href="mailto:gmann@ghanshyammann.com" target="_blank">gmann@ghanshyammann.com</a>> wrote:<br>
> ><br>
> > ---- On Thu, 20 Jan 2022 13:36:53 -0600 Mark Goddard <<a href="mailto:mark@stackhpc.com" target="_blank">mark@stackhpc.com</a>> wrote ----<br>
> > > On Thu, 20 Jan 2022 at 18:40, Ghanshyam Mann <<a href="mailto:gmann@ghanshyammann.com" target="_blank">gmann@ghanshyammann.com</a>> wrote:<br>
> > > ><br>
> > > ><br>
> > > > ---- On Thu, 20 Jan 2022 03:35:33 -0600 Mark Goddard <<a href="mailto:mark@stackhpc.com" target="_blank">mark@stackhpc.com</a>> wrote ----<br>
> > > > > On Wed, 19 Jan 2022 at 16:12, Ghanshyam Mann <<a href="mailto:gmann@ghanshyammann.com" target="_blank">gmann@ghanshyammann.com</a>> wrote:<br>
> > > > > ><br>
> > > > > > ---- On Wed, 19 Jan 2022 04:35:53 -0600 Mark Goddard <<a href="mailto:mark@stackhpc.com" target="_blank">mark@stackhpc.com</a>> wrote ----<br>
> > > > > > > Hi,<br>
> > > > > > ><br>
> > > > > > > If you haven't been paying close attention, it would be easy to miss<br>
> > > > > > > some of the upcoming RBAC changes which will have an impact on<br>
> > > > > > > deployment projects. I thought I'd start a thread so that we can share<br>
> > > > > > > how we are approaching this, get answers to open questions, and<br>
> > > > > > > ideally all end up with a fairly consistent approach.<br>
> > > > > > ><br>
> > > > > > > The secure RBAC work has a long history, and continues to evolve.<br>
> > > > > > > According to [1], we should start to see some fairly substantial<br>
> > > > > > > changes over the next few releases. That spec is fairly long, but<br>
> > > > > > > worth a read.<br>
> > > > > > ><br>
> > > > > > > In the yoga timeline [2], there is one change in particular that has<br>
> > > > > > > an impact on deployment projects, "3. Keystone enforces scope by<br>
> > > > > > > default". After this change, all of the deprecated policies that many<br>
> > > > > > > still rely on in Keystone will be removed.<br>
> > > > > > ><br>
> > > > > > > In kolla-ansible, we have an etherpad [5] with some notes, questions<br>
> > > > > > > and half-baked plans. We made some changes in Xena [3] to use system<br>
> > > > > > > scope in some places when interacting with system APIs in Ansible<br>
> > > > > > > tasks.<br>
> > > > > > ><br>
> > > > > > > The next change we have staged is to add the service role to all<br>
> > > > > > > service users [4], in preparation for [2].<br>
> > > > > > ><br>
> > > > > > > Question: should the role be added with system scope or in the<br>
> > > > > > > existing service project? The obvious main use for this is token<br>
> > > > > > > validation, which seems to allow system or project scope.<br>
> > > > > > ><br>
> > > > > > > We anticipate that some service users may still require some<br>
> > > > > > > project-scoped roles, e.g. when creating resources for octavia. We'll<br>
> > > > > > > deal with those on a case by case basis.<br>
> > > > > ><br>
> > > > > > Service roles are planned for phase2 which is Z release[1]. The Idea here is<br>
> > > > > > service to service communication will happen with 'service' role (which keystone<br>
> > > > > > need to implement yet) and end users will keep using the what ever role<br>
> > > > > > is default (or overridden in policy file) which can be project or system scoped<br>
> > > > > > depends on the APIs.<br>
> > > > > ><br>
> > > > > > So at the end service-service APIs policy default will looks like<br>
> > > > > ><br>
> > > > > > '(role:admin and system:network and project_id:%(project_id)s) or (role:service and project_name:service)'<br>
> > > > > ><br>
> > > > > > Say nova will use that service role to communicate to cinder and cinder policy will pass<br>
> > > > > > as service role is in OR in default policy.<br>
> > > > > ><br>
> > > > > > But let's see how they are going to be and if any challenges when we will implement<br>
> > > > > > it in Z cycle.<br>
> > > > ><br>
> > > > > I'm not 100% on our reasoning for using the service role in yoga (I<br>
> > > > > wasn't in the discussion when we made the switch, although John<br>
> > > > > Garbutt was), although I can provide at least one reason.<br>
> > > > ><br>
> > > > > Currently, we have a bunch of service users doing things like keystone<br>
> > > > > token validation using the admin role in the service project. If we<br>
> > > > > enforce scopes & new defaults in keystone, this will no longer work,<br>
> > > > > due to the default policy:<br>
> > > > ><br>
> > > > > identity:validate_token: (role:reader and system_scope:all) or<br>
> > > > > rule:service_role or rule:token_subject<br>
> > > > ><br>
> > > > > Now we could go and assign system-reader to all these users, but if<br>
> > > > > the end goal is to give them all the service role, and that allows<br>
> > > > > token validation, then to me that seems like a better path.<br>
> > > > ><br>
> > > > > Currently, we're creating the service role during deploy & upgrade,<br>
> > > > > then assigning it to users. Keystone is supposed to create the service<br>
> > > > > role in yoga, so we can eventually drop that part.<br>
> > > > ><br>
> > > > > Does this seem reasonable? Is keystone still on track to create the<br>
> > > > > service role in yoga?<br>
> > > ><br>
> > > > I think this is a reasonable plan and once we have service roles implemented<br>
> > > > in keystone as well as in all the services to request other service APIs then<br>
> > > > deployment project (Kolla here) can update them from system_reader to<br>
> > > > actual service role.<br>
> > ><br>
> > > To be clear, I am proposing to skip system-reader, and go straight to<br>
> > > the service role in yoga.<br>
> ><br>
> > But that would not be doable until services implement service roles which is<br>
> > Yoga cycle target for keystone and Z cyle target for other projects. Or you mean<br>
> > to re-consider to target the service role for all projects also in Yoga so that<br>
> > deployment projects can go with service role directly?<br>
> <br>
> Our current plan is to add the service role to all service users in<br>
> yoga. This will allow keystone token validation to work when keystone<br>
> drops the deprecated policies.<br>
> <br>
> We will not remove the admin role from service users in the service<br>
> project during yoga. This will allow projects other than keystone to<br>
> continue to work as before.<br>
> <br>
> At some later point, we will remove the admin role from service users<br>
> in the service project, hopefully relying on the service role for most<br>
> service-service communication. There may be other roles we need to<br>
> assign in order to drop admin, but we'll assess that as we go.<br>
> <br>
> Hopefully that's a bit more of a clear picture, and it seems sensible?<br>
<br>
+1, sounds good to me. Hopefully we will get in better shape by Z release<br>
when all (or maximum) services will be migrated to new RBAC. Till than<br>
your plan sounds reasonable. <br>
<br>
-gmann<br></blockquote><div><br></div><div>I'll follow the same approach in Puppet OpenStack and will add the project-scoped 'service' role</div><div>to each service user by default. IIUC This is consistent with the current devstack which assigns</div><div>the project-scoped service role to each service user, so I expect this approach will be tested</div><div>in dsvm jobs [1].</div><div> [1] <a href="https://github.com/openstack/devstack/blob/d5d0bed479497560489983ae1fc80444b44fe029/lib/keystone#L421">https://github.com/openstack/devstack/blob/d5d0bed479497560489983ae1fc80444b44fe029/lib/keystone#L421</a></div><div><br></div><div>The same was already implemented in TripleO by [2]</div><div> [2] <a href="https://review.opendev.org/c/openstack/tripleo-heat-templates/+/819250">https://review.opendev.org/c/openstack/tripleo-heat-templates/+/819250</a></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> <br>
> ><br>
> > -gmann<br>
> ><br>
> > ><br>
> > > ><br>
> > > > And yes that can be done for token validation as well as<br>
> > > > the service-to-service API calls for example nova to cinder or neutron to nova<br>
> > > > APIs call. I do not think we can migrate everything (service tokens) together for all<br>
> > > > the services in deployment projects until all these services are ready with the 'service'<br>
> > > > role implementation (implementation means changing their default roles<br>
> > > > to add 'service' role for service-to-service APIs).<br>
> > > ><br>
> > > > Regarding the keystone track on service role work in Yoga or not, I do not<br>
> > > > have clear answer may be Lance or keystone team can answer it. But Lance<br>
> > > > has spec up[1] but not yet merged.<br>
> > > ><br>
> > > > [1] <a href="https://review.opendev.org/c/openstack/keystone-specs/+/818616" rel="noreferrer" target="_blank">https://review.opendev.org/c/openstack/keystone-specs/+/818616</a><br>
> > > ><br>
> > > > -gmann<br>
> > > ><br>
> > > > ><br>
> > > > > ><br>
> > > > > > ><br>
> > > > > > > In anticipation of keystone setting enforce_scope=True and removing<br>
> > > > > > > old default policies (which I assume effectively removes<br>
> > > > > > > enforce_new_defaults?), we will set this in kolla-ansible, and try to<br>
> > > > > > > deal with any fallout. Hopefully the previous work will make this<br>
> > > > > > > minimal.<br>
> > > > > > ><br>
> > > > > > > How does that line up with other projects' approaches? What have we missed?<br>
> > > > > ><br>
> > > > > > Yeah, we want users/deployment projects/horizon etc to use the new policy from<br>
> > > > > > keystone as first and we will see feedback how they are (good, bad, really bad) from<br>
> > > > > > usage perspective. Why we choose keystone is, because new policy are there since<br>
> > > > > > many cycle and ready to use. Other projects needs to work their policy as per new<br>
> > > > > > SRBAC design/direction (for example nova needs to modify their policy before we ask<br>
> > > > > > users to use new policy and work is under progress[2]).<br>
> > > > > ><br>
> > > > > > I think trying in kolla will be good way to know if we can move to keystone's new policy<br>
> > > > > > completely in yoga.<br>
> > > > ><br>
> > > > > We have a scope-enforcing preview patch [1], and it's passing our base<br>
> > > > > set of tests. I have another that triggers all of the jobs.<br>
> > > > ><br>
> > > > > [1] <a href="https://review.opendev.org/c/openstack/kolla-ansible/+/825406" rel="noreferrer" target="_blank">https://review.opendev.org/c/openstack/kolla-ansible/+/825406</a><br>
> > > > > ><br>
> > > > > > [1] <a href="https://opendev.org/openstack/governance/src/branch/master/goals/selected/consistent-and-secure-rbac.rst#z-release-timeline" rel="noreferrer" target="_blank">https://opendev.org/openstack/governance/src/branch/master/goals/selected/consistent-and-secure-rbac.rst#z-release-timeline</a><br>
> > > > > > [2] <a href="https://blueprints.launchpad.net/nova/+spec/policy-defaults-refresh-2" rel="noreferrer" target="_blank">https://blueprints.launchpad.net/nova/+spec/policy-defaults-refresh-2</a><br>
> > > > > ><br>
> > > > > > -gmann<br>
> > > > > ><br>
> > > > > > ><br>
> > > > > > > Mark<br>
> > > > > > ><br>
> > > > > > > [1] <a href="https://opendev.org/openstack/governance/src/branch/master/goals/selected/consistent-and-secure-rbac.rst" rel="noreferrer" target="_blank">https://opendev.org/openstack/governance/src/branch/master/goals/selected/consistent-and-secure-rbac.rst</a><br>
> > > > > > > [2] <a href="https://opendev.org/openstack/governance/src/branch/master/goals/selected/consistent-and-secure-rbac.rst#yoga-timeline-7th-mar-2022" rel="noreferrer" target="_blank">https://opendev.org/openstack/governance/src/branch/master/goals/selected/consistent-and-secure-rbac.rst#yoga-timeline-7th-mar-2022</a><br>
> > > > > > > [3] <a href="https://opendev.org/openstack/kolla-ansible/commit/2e933dceb591c3505f35c2c1de924f3978fb81a7" rel="noreferrer" target="_blank">https://opendev.org/openstack/kolla-ansible/commit/2e933dceb591c3505f35c2c1de924f3978fb81a7</a><br>
> > > > > > > [4] <a href="https://review.opendev.org/c/openstack/kolla-ansible/+/815577" rel="noreferrer" target="_blank">https://review.opendev.org/c/openstack/kolla-ansible/+/815577</a><br>
> > > > > > > [5] <a href="https://etherpad.opendev.org/p/enabling-system-scope-in-kolla-ansible" rel="noreferrer" target="_blank">https://etherpad.opendev.org/p/enabling-system-scope-in-kolla-ansible</a><br>
> > > > > > ><br>
> > > > > > ><br>
> > > > ><br>
> > > > ><br>
> > ><br>
> > ><br>
> <br>
> <br>
<br>
</blockquote></div></div>