Open Stack

 ---- On Tue, 15 Mar 2022 10:45:05 -0500 Takashi Kajinami <tkajinam at redhat.com> wrote ----
 > 
 > 
 > On Wed, Mar 16, 2022 at 12:23 AM Ghanshyam Mann <gmann at ghanshyammann.com> wrote:
 >  ---- On Wed, 09 Mar 2022 19:28:03 -0600 Takashi Kajinami <tkajinam at redhat.com> wrote ----
 >  > 
 >  > 
 >  > On Thu, Mar 10, 2022 at 10:13 AM Ghanshyam Mann <gmann at ghanshyammann.com> wrote:
 >  >  ---- On Wed, 09 Mar 2022 17:14:02 -0600 Takashi Kajinami <tkajinam at redhat.com> wrote ----
 >  >  > 
 >  >  > 
 >  >  > On Thu, Mar 10, 2022 at 2:59 AM Ghanshyam Mann <gmann at ghanshyammann.com> wrote:
 >  >  >  ---- On Wed, 09 Mar 2022 06:43:03 -0600 Takashi Kajinami <tkajinam at redhat.com> wrote ----
 >  >  >  > Hi Rico.
 >  >  >  > Thank you for your insights.Please find my replies inline.
 >  >  >  > 
 >  >  >  > On Wed, Mar 9, 2022 at 6:22 PM Rico Lin <ricolin at ricolky.com> wrote:
 >  >  >  > 
 >  >  >  > 
 >  >  >  > On Wed, Mar 9, 2022 at 1:57 PM Takashi Kajinami <tkajinam at redhat.com> wrote:
 >  >  >  > >
 >  >  >  > > Hello,
 >  >  >  > >
 >  >  >  > >
 >  >  >  > > I've been working on updating policy rules in heat according to the latest
 >  >  >  > > guideline for SRBAC implementation.
 >  >  >  > 
 >  >  >  > Thanks so much for working on it, definitely +1 on enhancing security.
 >  >  >  > >
 >  >  >  > >  [1] https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html#direction-change
 >  >  >  > >
 >  >  >  > > However this change reveals one potential issue with system resources.
 >  >  >  > >
 >  >  >  > > In heat there are some resource types like OS::Nova::Flavor or OS::Keystone::User
 >  >  >  > > which is allowed for only system users. (I'll call these resources as "system resources" here)
 >  >  >  > We create those resources based on checking `role:admin`, so project admin is accepted.This means a user under that project with role:admin, can operate those resources.I'm not sure what system user means here, so correct me if I'm wrong.
 >  >  >  > The core problem is that backend services like nova enforces different policies from just role:admin whenSRBAC is enforced. I reported that a while ago as https://storyboard.openstack.org/#!/story/2009787 .
 >  >  >  > 
 >  >  >  > The main problem here is that heat currently supports resources which require different scopes of credentials.For example, once all deprecated policy rules are removed, nova allows project users to create instances(OS::Nova::Server)[1],while nova allows only system admins to create flavors(OS::Nova::Flavor)[2]. Currently Heat leverages the user credentialused to create a stack when Heat requests backend services to create its resources, and it's "impossible" to create a stackwith these two resources by a single credential (as each token can be scoped by project or system.
 >  >  >  > [1] https://github.com/openstack/nova/blob/d1b036fdb124d1ef48f5cf024d4677bfe5b481d7/nova/policies/servers.py#L210[2] https://github.com/openstack/nova/blob/d1b036fdb124d1ef48f5cf024d4677bfe5b481d7/nova/policies/flavor_manage.py#L28-L36
 >  >  >  > If we keep allowing system users to create stacks, then it would allow users to create system resourceslike flavors. But this never solves the problem with the "mixed" pattern which I described above.Hope this will explain these three options more clearly.
 >  >  >  >  
 >  >  >  > >
 >  >  >  > > Because heat uses user credentials to create resources in backend services, if we require
 >  >  >  > > project scope for stack/resource management then we are no longer able to create these system
 >  >  >  > > resources as part of stack. (*1)>
 >  >  >  > > There are some options I can think of at this moment.
 >  >  >  > >
 >  >  >  > > 1. Deprecate and remove all resources which require system scope
 >  >  >  > > Pros
 >  >  >  > > - This is most easy solution to be implemented
 >  >  >  > > Cons
 >  >  >  > > - This one has the biggest user impact
 >  >  >  > 
 >  >  >  > Don't :)
 >  >  > 
 >  >  > Yeah, not sure how removing will work.
 >  >  > 
 >  >  >  > > 2. Use service credential for resource creation
 >  >  >  > > Pros:
 >  >  >  > > - This might have smallest user impact
 >  >  >  > > Cons:
 >  >  >  > > - This requires a relatively big change in heat's architecture, which I've not yet evaluated.
 >  >  >  > > - We need to determine the project scope role to be allowed to do system operation.
 >  >  >  > >  
 >  >  >  > I think this is something we can discuss more.
 >  >  >  > > 3. Implement additional project-role in each service to allow creation via stack creation.
 >  >  >  > > Pros:
 >  >  >  > > - This would avoid user impact.
 >  >  >  > > Cons:
 >  >  >  > > - requires work in multiple components.
 >  >  >  > > - We need to determine the project scope role to be allowed to do system operation
 >  >  >  > 
 >  >  >  > 
 >  >  >  > Indeed current stack operation require `(role:admin and system_scope:all) OR (role:member and project_id:%(project_id)s)`.And if we remove the system part and keep `(role:member and project_id:%(project_id)s)`, will mean only a user in that project with a member role can operate stack, and if the user got also admin role, that user can manage User/Flavor resources through stack too.The main problem here is that nova will no longer allow project admins to create flavors,and it will require system admin when SRBAC is enforced(which means all deprecated rulesare disabled)
 >  >  >  >  
 >  >  >  > Also, note thatFor the case of Nova::Flavor, both policies from Nova and Heat resource will be enforced here, so it only accepts the intersection part of rules.
 >  >  >  > So if we go with removing the system admin, that means it's time to also remove some deprecated global scope operation.
 >  >  >  > IMO, to accept Project Admin or member make more reasonable (like you proposed in option 3)
 >  >  > 
 >  >  > This is a similar case that we discussed in the service role ML thread[1] or in the spec[2].
 >  >  > 
 >  >  > For such cases only, where services need to interact with another service we will have the 'service' as
 >  >  > a special role. In this case, the heat having the project scoped token will use the 'service' role token for
 >  >  > Nova Flavor APIs calls.  Now whether that 'service' role token needs to be system scoped or not is
 >  >  > open question Mark brought up in that discussion. IMO, we should use the scoped token even in case
 >  >  > of 'service' role and do not skip scope checks for 'service' role in oslo policy as it can be a valid role used
 >  >  > by any users at production. So with the new SRBAC, heat needs to use the system scoped 'service' role token.
 >  >  > 
 >  >  > But as 'service' role is not yet implemented by keystone[2], we can use the legacy admin token which is currently
 >  >  > used in heat or other service-to-service interaction. 
 >  >  > 
 >  >  > Thanks gmann for inputs.
 >  >  > HoweverI think we still need a different role here.
 >  >  > The role I've mentioned here is not the role assigned to the Heat service user but the project-roleassigned to users to allow them create stacks with system resources like nova flavor or keystone user.This role is assigned to actual cloud users so service would not be appropriate.
 >  >  > 
 >  >  > I thought of using the admin role initially but this means that we provide a way to "bypass" policy checkand expose system operations to project users.
 >  >  > 
 >  >  > As a side note, current heat does not use its own credential but user credentials used for stack API calls. 
 >  >  > We can introduce this mechanism and let heat manage system resources by its service user but that isthe completely new feature we need to implement.
 >  > 
 >  > Thanks for the detailed information. For service to service interaction, I think heat needs to start using the configured
 >  > service user token instead of using actual user token.
 >  > 
 >  > For example nova use for neutron and similarly for other service interaction
 >  > - https://zuul.opendev.org/t/openstack/build/654ccb626d0e41a69308db1ed3328824/log/controller/logs/etc/nova/nova_conf.txt#40-51
 >  > 
 >  > I'll look into that option but It might take some time because I've not yet evaluated the impact of that change.My current idea is to use service credentials for system resources only so that we can keep using the user credentialfor project resources. 
 >  > 
 >  > As heat allows only project users to manage stacks, we need to determine the project role to allow creatingstacks with such system resources. Does anyone have any suggestions about this ?
 >  > Do we agree we can allow project-admin to create stacks with flavors or keystone users as the previous policy rulesallow or should we implement a new role to restrict management of system resources ?
 > 
 > Currently, existing way of creating flavor with project user should work as Nova still support the legacy policy so yes
 > project admin can do that. And on the new role, I think that is what the 'service' role will be.
 > 
 > I think the question we are discussing here and on spec is what scope the project should use for 'service' role requests for
 > their service-to-service interaction? IMO, it can be the same scope that calling APIs has. For heat example, heat can create
 > a 'service' role with the 'system' scope and call Nova to create flavor.  Another option is we skip the scope check for 'service'
 > role as a special case but this can create issue when any production deployment has a role with the name of 'service'.
 > Any other thoughts on this?
 >  No. The current question is related to the scope used by the user creating a stack. 
 > 
 > "Stack" in heat is the resource belonging to individual projects, so it should be created by project users.
 > If we introduce the mechanism to make the heat service user create system resources like flavors on behalfof that project user, Heat would allow project users(eg. project admin) to create a flavor, while Nova doesn't allow itand requires system admin.We should consider what would be the appropriate project role required here.

Yeah, I understand the issue. I am not a Heat expert but from APIs point of view these kinds of APIs need more
granularity especially when an APIs for creating one resource create another resource in other service and user
might not be aware of that. If it cannot be made granular form Heat perspective then we may want to think of
a special role like you suggested.

I am not sure how many such examples (creating different resources in other services and with different scope operations)
we have like this for other service-to-service interactions (at least not in nova).

I am adding it for the keystone PTG etherpad and let's discuss this case along with the general 'service' role case.

-gmann

 >  -gmann
 > 
 > 
 >  > 
 >  >  -gmann
 >  > 
 >  >  >  [1] http://lists.openstack.org/pipermail/openstack-discuss/2022-January/026777.html
 >  >  > [2] https://review.opendev.org/c/openstack/keystone-specs/+/818616
 >  >  > 
 >  >  > -gmann
 >  >  > 
 >  >  >  > >
 >  >  >  > > Thank you,
 >  >  >  > > Takashi
 >  >  >  > >
 >  >  >  > 
 >  >  > 
 >  >  > 
 >  > 
 >  > 
 > 
 >