[heat][rbac] Management of "system resources" with SRBAC enforced
Hello,
I've been working on updating policy rules in heat according to the latest guideline for SRBAC implementation.
Based on the recent direction change[1], now only project users are allowed to create project-based resources like instances or networks. I'm updating policy rules in heat so that only project users can manage project resources in Heat like stacks or resources.
[1] https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rba...
However this change reveals one potential issue with system resources.
In heat there are some resource types like OS::Nova::Flavor or OS::Keystone::User which is allowed for only system users. (I'll call these resources as "system resources" here)
Because heat uses user credentials to create resources in backend services, if we require project scope for stack/resource management then we are no longer able to create these system resources as part of stack. (*1)
There are some options I can think of at this moment.
1. Deprecate and remove all resources which require system scope Pros - This is most easy solution to be implemented Cons - This one has the biggest user impact
2. Use service credential for resource creation Pros: - This might have smallest user impact Cons: - This requires a relatively big change in heat's architecture, which I've not yet evaluated. - We need to determine the project scope role to be allowed to do system operation.
3. Implement additional project-role in each service to allow creation via stack creation. Pros: - This would avoid user impact. Cons: - requires work in multiple components. - We need to determine the project scope role to be allowed to do system operation.
I'm not quite sure which one would be the best. If I can ask for some feedback about these options (or any other new options) from heat's perspective and rbac's perspective then it would be nice.
(*1) As of now, nova and neutron no longer allow creating system users to create project resources, so system users are no longer able to create stacks with project resources like instances or networks with default "new" policies.
Thank you, Takashi
On Wed, Mar 9, 2022 at 1:57 PM Takashi Kajinami tkajinam@redhat.com wrote:
Hello,
I've been working on updating policy rules in heat according to the latest guideline for SRBAC implementation.
Thanks so much for working on it, definitely +1 on enhancing security.
[1]
https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rba...
However this change reveals one potential issue with system resources.
In heat there are some resource types like OS::Nova::Flavor or
OS::Keystone::User
which is allowed for only system users. (I'll call these resources as
"system resources" here)
We create those resources based on checking `role:admin`, so project admin is accepted. This means a user under that project with role:admin, can operate those resources. I'm not sure what system user means here, so correct me if I'm wrong.
Because heat uses user credentials to create resources in backend
services, if we require
project scope for stack/resource management then we are no longer able to
create these system
resources as part of stack. (*1)
There are some options I can think of at this moment.
- Deprecate and remove all resources which require system scope
Pros
- This is most easy solution to be implemented
Cons
- This one has the biggest user impact
Don't :)
- Use service credential for resource creation
Pros:
- This might have smallest user impact
Cons:
- This requires a relatively big change in heat's architecture, which
I've not yet evaluated.
- We need to determine the project scope role to be allowed to do system
operation.
I think this is something we can discuss more.
- Implement additional project-role in each service to allow creation
via stack creation.
Pros:
- This would avoid user impact.
Cons:
- requires work in multiple components.
- We need to determine the project scope role to be allowed to do system
operation
Indeed current stack operation require `(role:admin and system_scope:all) OR (role:member and project_id:%(project_id)s)`. And if we remove the system part and keep `(role:member and project_id:%(project_id)s)`, will mean only a user in that project with a member role can operate stack, and if the user got also admin role, that user can manage User/Flavor resources through stack too.
Also, note that For the case of Nova::Flavor, both policies from Nova and Heat resource will be enforced here, so it only accepts the intersection part of rules.
So if we go with removing the system admin, that means it's time to also remove some deprecated global scope operation.
IMO, to accept Project Admin or member make more reasonable (like you proposed in option 3)
Thank you, Takashi
Hi Rico.
Thank you for your insights. Please find my replies inline.
On Wed, Mar 9, 2022 at 6:22 PM Rico Lin ricolin@ricolky.com wrote:
On Wed, Mar 9, 2022 at 1:57 PM Takashi Kajinami tkajinam@redhat.com wrote:
Hello,
I've been working on updating policy rules in heat according to the
latest
guideline for SRBAC implementation.
Thanks so much for working on it, definitely +1 on enhancing security.
[1]
https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rba...
However this change reveals one potential issue with system resources.
In heat there are some resource types like OS::Nova::Flavor or
OS::Keystone::User
which is allowed for only system users. (I'll call these resources as
"system resources" here)
We create those resources based on checking `role:admin`, so project admin is accepted. This means a user under that project with role:admin, can operate those resources. I'm not sure what system user means here, so correct me if I'm wrong.
The core problem is that backend services like nova enforces different policies from just role:admin when SRBAC is enforced. I reported that a while ago as https://storyboard.openstack.org/#!/story/2009787 .
The main problem here is that heat currently supports resources which require different scopes of credentials. For example, once all deprecated policy rules are removed, nova allows project users to create instances(OS::Nova::Server)[1], while nova allows only system admins to create flavors(OS::Nova::Flavor)[2]. Currently Heat leverages the user credential used to create a stack when Heat requests backend services to create its resources, and it's "impossible" to create a stack with these two resources by a single credential (as each token can be scoped by project or system.
[1] https://github.com/openstack/nova/blob/d1b036fdb124d1ef48f5cf024d4677bfe5b48... [2] https://github.com/openstack/nova/blob/d1b036fdb124d1ef48f5cf024d4677bfe5b48...
If we keep allowing system users to create stacks, then it would allow users to create system resources like flavors. But this never solves the problem with the "mixed" pattern which I described above. Hope this will explain these three options more clearly.
Because heat uses user credentials to create resources in backend
services, if we require
project scope for stack/resource management then we are no longer able
to create these system
resources as part of stack. (*1)
There are some options I can think of at this moment.
- Deprecate and remove all resources which require system scope
Pros
- This is most easy solution to be implemented
Cons
- This one has the biggest user impact
Don't :)
- Use service credential for resource creation
Pros:
- This might have smallest user impact
Cons:
- This requires a relatively big change in heat's architecture, which
I've not yet evaluated.
- We need to determine the project scope role to be allowed to do system
operation.
I think this is something we can discuss more.
- Implement additional project-role in each service to allow creation
via stack creation.
Pros:
- This would avoid user impact.
Cons:
- requires work in multiple components.
- We need to determine the project scope role to be allowed to do system
operation
Indeed current stack operation require `(role:admin and system_scope:all) OR (role:member and project_id:%(project_id)s)`. And if we remove the system part and keep `(role:member and project_id:%(project_id)s)`, will mean only a user in that project with a member role can operate stack, and if the user got also admin role, that user can manage User/Flavor resources through stack too.
The main problem here is that nova will no longer allow project admins to create flavors, and it will require system admin when SRBAC is enforced(which means all deprecated rules are disabled)
Also, note that For the case of Nova::Flavor, both policies from Nova and Heat resource will be enforced here, so it only accepts the intersection part of rules.
So if we go with removing the system admin, that means it's time to also remove some deprecated global scope operation.
IMO, to accept Project Admin or member make more reasonable (like you proposed in option 3)
Thank you, Takashi
---- On Wed, 09 Mar 2022 06:43:03 -0600 Takashi Kajinami tkajinam@redhat.com wrote ----
Hi Rico. Thank you for your insights.Please find my replies inline.
On Wed, Mar 9, 2022 at 6:22 PM Rico Lin ricolin@ricolky.com wrote:
On Wed, Mar 9, 2022 at 1:57 PM Takashi Kajinami tkajinam@redhat.com wrote:
Hello,
I've been working on updating policy rules in heat according to the latest guideline for SRBAC implementation.
Thanks so much for working on it, definitely +1 on enhancing security.
[1] https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rba...
However this change reveals one potential issue with system resources.
In heat there are some resource types like OS::Nova::Flavor or OS::Keystone::User which is allowed for only system users. (I'll call these resources as "system resources" here)
We create those resources based on checking `role:admin`, so project admin is accepted.This means a user under that project with role:admin, can operate those resources.I'm not sure what system user means here, so correct me if I'm wrong. The core problem is that backend services like nova enforces different policies from just role:admin whenSRBAC is enforced. I reported that a while ago as https://storyboard.openstack.org/#!/story/2009787 .
The main problem here is that heat currently supports resources which require different scopes of credentials.For example, once all deprecated policy rules are removed, nova allows project users to create instances(OS::Nova::Server)[1],while nova allows only system admins to create flavors(OS::Nova::Flavor)[2]. Currently Heat leverages the user credentialused to create a stack when Heat requests backend services to create its resources, and it's "impossible" to create a stackwith these two resources by a single credential (as each token can be scoped by project or system. [1] https://github.com/openstack/nova/blob/d1b036fdb124d1ef48f5cf024d4677bfe5b48...] https://github.com/openstack/nova/blob/d1b036fdb124d1ef48f5cf024d4677bfe5b48... If we keep allowing system users to create stacks, then it would allow users to create system resourceslike flavors. But this never solves the problem with the "mixed" pattern which I described above.Hope this will explain these three options more clearly.
Because heat uses user credentials to create resources in backend services, if we require project scope for stack/resource management then we are no longer able to create these system resources as part of stack. (*1)> There are some options I can think of at this moment.
- Deprecate and remove all resources which require system scope
Pros
- This is most easy solution to be implemented
Cons
- This one has the biggest user impact
Don't :)
Yeah, not sure how removing will work.
- Use service credential for resource creation
Pros:
- This might have smallest user impact
Cons:
- This requires a relatively big change in heat's architecture, which I've not yet evaluated.
- We need to determine the project scope role to be allowed to do system operation.
I think this is something we can discuss more.
- Implement additional project-role in each service to allow creation via stack creation.
Pros:
- This would avoid user impact.
Cons:
- requires work in multiple components.
- We need to determine the project scope role to be allowed to do system operation
Indeed current stack operation require `(role:admin and system_scope:all) OR (role:member and project_id:%(project_id)s)`.And if we remove the system part and keep `(role:member and project_id:%(project_id)s)`, will mean only a user in that project with a member role can operate stack, and if the user got also admin role, that user can manage User/Flavor resources through stack too.The main problem here is that nova will no longer allow project admins to create flavors,and it will require system admin when SRBAC is enforced(which means all deprecated rulesare disabled)
Also, note thatFor the case of Nova::Flavor, both policies from Nova and Heat resource will be enforced here, so it only accepts the intersection part of rules. So if we go with removing the system admin, that means it's time to also remove some deprecated global scope operation. IMO, to accept Project Admin or member make more reasonable (like you proposed in option 3)
This is a similar case that we discussed in the service role ML thread[1] or in the spec[2].
For such cases only, where services need to interact with another service we will have the 'service' as a special role. In this case, the heat having the project scoped token will use the 'service' role token for Nova Flavor APIs calls. Now whether that 'service' role token needs to be system scoped or not is open question Mark brought up in that discussion. IMO, we should use the scoped token even in case of 'service' role and do not skip scope checks for 'service' role in oslo policy as it can be a valid role used by any users at production. So with the new SRBAC, heat needs to use the system scoped 'service' role token.
But as 'service' role is not yet implemented by keystone[2], we can use the legacy admin token which is currently used in heat or other service-to-service interaction.
[1] http://lists.openstack.org/pipermail/openstack-discuss/2022-January/026777.h... [2] https://review.opendev.org/c/openstack/keystone-specs/+/818616
-gmann
Thank you, Takashi
On Thu, Mar 10, 2022 at 2:59 AM Ghanshyam Mann gmann@ghanshyammann.com wrote:
---- On Wed, 09 Mar 2022 06:43:03 -0600 Takashi Kajinami < tkajinam@redhat.com> wrote ----
Hi Rico. Thank you for your insights.Please find my replies inline.
On Wed, Mar 9, 2022 at 6:22 PM Rico Lin ricolin@ricolky.com wrote:
On Wed, Mar 9, 2022 at 1:57 PM Takashi Kajinami tkajinam@redhat.com
wrote:
Hello,
I've been working on updating policy rules in heat according to the
latest
guideline for SRBAC implementation.
Thanks so much for working on it, definitely +1 on enhancing security.
[1]
https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rba...
However this change reveals one potential issue with system resources.
In heat there are some resource types like OS::Nova::Flavor or
OS::Keystone::User
which is allowed for only system users. (I'll call these resources as
"system resources" here)
We create those resources based on checking `role:admin`, so project
admin is accepted.This means a user under that project with role:admin, can operate those resources.I'm not sure what system user means here, so correct me if I'm wrong.
The core problem is that backend services like nova enforces different
policies from just role:admin whenSRBAC is enforced. I reported that a while ago as https://storyboard.openstack.org/#!/story/2009787 .
The main problem here is that heat currently supports resources which
require different scopes of credentials.For example, once all deprecated policy rules are removed, nova allows project users to create instances(OS::Nova::Server)[1],while nova allows only system admins to create flavors(OS::Nova::Flavor)[2]. Currently Heat leverages the user credentialused to create a stack when Heat requests backend services to create its resources, and it's "impossible" to create a stackwith these two resources by a single credential (as each token can be scoped by project or system.
[1]
https://github.com/openstack/nova/blob/d1b036fdb124d1ef48f5cf024d4677bfe5b48...] https://github.com/openstack/nova/blob/d1b036fdb124d1ef48f5cf024d4677bfe5b48...
If we keep allowing system users to create stacks, then it would allow
users to create system resourceslike flavors. But this never solves the problem with the "mixed" pattern which I described above.Hope this will explain these three options more clearly.
Because heat uses user credentials to create resources in backend
services, if we require
project scope for stack/resource management then we are no longer
able to create these system
resources as part of stack. (*1)> There are some options I can think of at this moment.
- Deprecate and remove all resources which require system scope
Pros
- This is most easy solution to be implemented
Cons
- This one has the biggest user impact
Don't :)
Yeah, not sure how removing will work.
- Use service credential for resource creation
Pros:
- This might have smallest user impact
Cons:
- This requires a relatively big change in heat's architecture, which
I've not yet evaluated.
- We need to determine the project scope role to be allowed to do
system operation.
I think this is something we can discuss more.
- Implement additional project-role in each service to allow
creation via stack creation.
Pros:
- This would avoid user impact.
Cons:
- requires work in multiple components.
- We need to determine the project scope role to be allowed to do
system operation
Indeed current stack operation require `(role:admin and
system_scope:all) OR (role:member and project_id:%(project_id)s)`.And if we remove the system part and keep `(role:member and project_id:%(project_id)s)`, will mean only a user in that project with a member role can operate stack, and if the user got also admin role, that user can manage User/Flavor resources through stack too.The main problem here is that nova will no longer allow project admins to create flavors,and it will require system admin when SRBAC is enforced(which means all deprecated rulesare disabled)
Also, note thatFor the case of Nova::Flavor, both policies from Nova
and Heat resource will be enforced here, so it only accepts the intersection part of rules.
So if we go with removing the system admin, that means it's time to
also remove some deprecated global scope operation.
IMO, to accept Project Admin or member make more reasonable (like you
proposed in option 3)
This is a similar case that we discussed in the service role ML thread[1] or in the spec[2].
For such cases only, where services need to interact with another service we will have the 'service' as a special role. In this case, the heat having the project scoped token will use the 'service' role token for Nova Flavor APIs calls. Now whether that 'service' role token needs to be system scoped or not is open question Mark brought up in that discussion. IMO, we should use the scoped token even in case of 'service' role and do not skip scope checks for 'service' role in oslo policy as it can be a valid role used by any users at production. So with the new SRBAC, heat needs to use the system scoped 'service' role token.
But as 'service' role is not yet implemented by keystone[2], we can use the legacy admin token which is currently used in heat or other service-to-service interaction.
Thanks gmann for inputs.
HoweverI think we still need a different role here.
The role I've mentioned here is not the role assigned to the Heat service user but the project-role assigned to users to allow them create stacks with system resources like nova flavor or keystone user. This role is assigned to actual cloud users so service would not be appropriate.
I thought of using the admin role initially but this means that we provide a way to "bypass" policy check and expose system operations to project users.
As a side note, current heat does not use its own credential but user credentials used for stack API calls. We can introduce this mechanism and let heat manage system resources by its service user but that is the completely new feature we need to implement.
[1] http://lists.openstack.org/pipermail/openstack-discuss/2022-January/026777.h... [2] https://review.opendev.org/c/openstack/keystone-specs/+/818616
-gmann
Thank you, Takashi
---- On Wed, 09 Mar 2022 17:14:02 -0600 Takashi Kajinami tkajinam@redhat.com wrote ----
On Thu, Mar 10, 2022 at 2:59 AM Ghanshyam Mann gmann@ghanshyammann.com wrote: ---- On Wed, 09 Mar 2022 06:43:03 -0600 Takashi Kajinami tkajinam@redhat.com wrote ----
Hi Rico. Thank you for your insights.Please find my replies inline.
On Wed, Mar 9, 2022 at 6:22 PM Rico Lin ricolin@ricolky.com wrote:
On Wed, Mar 9, 2022 at 1:57 PM Takashi Kajinami tkajinam@redhat.com wrote:
Hello,
I've been working on updating policy rules in heat according to the latest guideline for SRBAC implementation.
Thanks so much for working on it, definitely +1 on enhancing security.
[1] https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rba...
However this change reveals one potential issue with system resources.
In heat there are some resource types like OS::Nova::Flavor or OS::Keystone::User which is allowed for only system users. (I'll call these resources as "system resources" here)
We create those resources based on checking `role:admin`, so project admin is accepted.This means a user under that project with role:admin, can operate those resources.I'm not sure what system user means here, so correct me if I'm wrong. The core problem is that backend services like nova enforces different policies from just role:admin whenSRBAC is enforced. I reported that a while ago as https://storyboard.openstack.org/#!/story/2009787 .
The main problem here is that heat currently supports resources which require different scopes of credentials.For example, once all deprecated policy rules are removed, nova allows project users to create instances(OS::Nova::Server)[1],while nova allows only system admins to create flavors(OS::Nova::Flavor)[2]. Currently Heat leverages the user credentialused to create a stack when Heat requests backend services to create its resources, and it's "impossible" to create a stackwith these two resources by a single credential (as each token can be scoped by project or system. [1] https://github.com/openstack/nova/blob/d1b036fdb124d1ef48f5cf024d4677bfe5b48...] https://github.com/openstack/nova/blob/d1b036fdb124d1ef48f5cf024d4677bfe5b48... If we keep allowing system users to create stacks, then it would allow users to create system resourceslike flavors. But this never solves the problem with the "mixed" pattern which I described above.Hope this will explain these three options more clearly.
Because heat uses user credentials to create resources in backend services, if we require project scope for stack/resource management then we are no longer able to create these system resources as part of stack. (*1)> There are some options I can think of at this moment.
- Deprecate and remove all resources which require system scope
Pros
- This is most easy solution to be implemented
Cons
- This one has the biggest user impact
Don't :)
Yeah, not sure how removing will work.
- Use service credential for resource creation
Pros:
- This might have smallest user impact
Cons:
- This requires a relatively big change in heat's architecture, which I've not yet evaluated.
- We need to determine the project scope role to be allowed to do system operation.
I think this is something we can discuss more.
- Implement additional project-role in each service to allow creation via stack creation.
Pros:
- This would avoid user impact.
Cons:
- requires work in multiple components.
- We need to determine the project scope role to be allowed to do system operation
Indeed current stack operation require `(role:admin and system_scope:all) OR (role:member and project_id:%(project_id)s)`.And if we remove the system part and keep `(role:member and project_id:%(project_id)s)`, will mean only a user in that project with a member role can operate stack, and if the user got also admin role, that user can manage User/Flavor resources through stack too.The main problem here is that nova will no longer allow project admins to create flavors,and it will require system admin when SRBAC is enforced(which means all deprecated rulesare disabled)
Also, note thatFor the case of Nova::Flavor, both policies from Nova and Heat resource will be enforced here, so it only accepts the intersection part of rules. So if we go with removing the system admin, that means it's time to also remove some deprecated global scope operation. IMO, to accept Project Admin or member make more reasonable (like you proposed in option 3)
This is a similar case that we discussed in the service role ML thread[1] or in the spec[2].
For such cases only, where services need to interact with another service we will have the 'service' as a special role. In this case, the heat having the project scoped token will use the 'service' role token for Nova Flavor APIs calls. Now whether that 'service' role token needs to be system scoped or not is open question Mark brought up in that discussion. IMO, we should use the scoped token even in case of 'service' role and do not skip scope checks for 'service' role in oslo policy as it can be a valid role used by any users at production. So with the new SRBAC, heat needs to use the system scoped 'service' role token.
But as 'service' role is not yet implemented by keystone[2], we can use the legacy admin token which is currently used in heat or other service-to-service interaction.
Thanks gmann for inputs. HoweverI think we still need a different role here. The role I've mentioned here is not the role assigned to the Heat service user but the project-roleassigned to users to allow them create stacks with system resources like nova flavor or keystone user.This role is assigned to actual cloud users so service would not be appropriate.
I thought of using the admin role initially but this means that we provide a way to "bypass" policy checkand expose system operations to project users.
As a side note, current heat does not use its own credential but user credentials used for stack API calls. We can introduce this mechanism and let heat manage system resources by its service user but that isthe completely new feature we need to implement.
Thanks for the detailed information. For service to service interaction, I think heat needs to start using the configured service user token instead of using actual user token.
For example nova use for neutron and similarly for other service interaction - https://zuul.opendev.org/t/openstack/build/654ccb626d0e41a69308db1ed3328824/...
-gmann
[1] http://lists.openstack.org/pipermail/openstack-discuss/2022-January/026777.h... [2] https://review.opendev.org/c/openstack/keystone-specs/+/818616
-gmann
Thank you, Takashi
On Thu, Mar 10, 2022 at 10:13 AM Ghanshyam Mann gmann@ghanshyammann.com wrote:
---- On Wed, 09 Mar 2022 17:14:02 -0600 Takashi Kajinami < tkajinam@redhat.com> wrote ----
On Thu, Mar 10, 2022 at 2:59 AM Ghanshyam Mann gmann@ghanshyammann.com
wrote:
---- On Wed, 09 Mar 2022 06:43:03 -0600 Takashi Kajinami <
tkajinam@redhat.com> wrote ----
Hi Rico. Thank you for your insights.Please find my replies inline.
On Wed, Mar 9, 2022 at 6:22 PM Rico Lin ricolin@ricolky.com wrote:
On Wed, Mar 9, 2022 at 1:57 PM Takashi Kajinami tkajinam@redhat.com
wrote:
Hello,
I've been working on updating policy rules in heat according to
the latest
guideline for SRBAC implementation.
Thanks so much for working on it, definitely +1 on enhancing
security.
[1]
https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rba...
However this change reveals one potential issue with system
resources.
In heat there are some resource types like OS::Nova::Flavor or
OS::Keystone::User
which is allowed for only system users. (I'll call these resources
as "system resources" here)
We create those resources based on checking `role:admin`, so project
admin is accepted.This means a user under that project with role:admin, can operate those resources.I'm not sure what system user means here, so correct me if I'm wrong.
The core problem is that backend services like nova enforces
different policies from just role:admin whenSRBAC is enforced. I reported that a while ago as https://storyboard.openstack.org/#!/story/2009787 .
The main problem here is that heat currently supports resources
which require different scopes of credentials.For example, once all deprecated policy rules are removed, nova allows project users to create instances(OS::Nova::Server)[1],while nova allows only system admins to create flavors(OS::Nova::Flavor)[2]. Currently Heat leverages the user credentialused to create a stack when Heat requests backend services to create its resources, and it's "impossible" to create a stackwith these two resources by a single credential (as each token can be scoped by project or system.
[1]
https://github.com/openstack/nova/blob/d1b036fdb124d1ef48f5cf024d4677bfe5b48...] https://github.com/openstack/nova/blob/d1b036fdb124d1ef48f5cf024d4677bfe5b48...
If we keep allowing system users to create stacks, then it would
allow users to create system resourceslike flavors. But this never solves the problem with the "mixed" pattern which I described above.Hope this will explain these three options more clearly.
Because heat uses user credentials to create resources in backend
services, if we require
project scope for stack/resource management then we are no longer
able to create these system
resources as part of stack. (*1)> There are some options I can think of at this moment.
- Deprecate and remove all resources which require system scope
Pros
- This is most easy solution to be implemented
Cons
- This one has the biggest user impact
Don't :)
Yeah, not sure how removing will work.
- Use service credential for resource creation
Pros:
- This might have smallest user impact
Cons:
- This requires a relatively big change in heat's architecture,
which I've not yet evaluated.
- We need to determine the project scope role to be allowed to do
system operation.
I think this is something we can discuss more.
- Implement additional project-role in each service to allow
creation via stack creation.
Pros:
- This would avoid user impact.
Cons:
- requires work in multiple components.
- We need to determine the project scope role to be allowed to do
system operation
Indeed current stack operation require `(role:admin and
system_scope:all) OR (role:member and project_id:%(project_id)s)`.And if we remove the system part and keep `(role:member and project_id:%(project_id)s)`, will mean only a user in that project with a member role can operate stack, and if the user got also admin role, that user can manage User/Flavor resources through stack too.The main problem here is that nova will no longer allow project admins to create flavors,and it will require system admin when SRBAC is enforced(which means all deprecated rulesare disabled)
Also, note thatFor the case of Nova::Flavor, both policies from Nova
and Heat resource will be enforced here, so it only accepts the intersection part of rules.
So if we go with removing the system admin, that means it's time to
also remove some deprecated global scope operation.
IMO, to accept Project Admin or member make more reasonable (like
you proposed in option 3)
This is a similar case that we discussed in the service role ML
thread[1] or in the spec[2].
For such cases only, where services need to interact with another
service we will have the 'service' as
a special role. In this case, the heat having the project scoped token
will use the 'service' role token for
Nova Flavor APIs calls. Now whether that 'service' role token needs to
be system scoped or not is
open question Mark brought up in that discussion. IMO, we should use
the scoped token even in case
of 'service' role and do not skip scope checks for 'service' role in
oslo policy as it can be a valid role used
by any users at production. So with the new SRBAC, heat needs to use
the system scoped 'service' role token.
But as 'service' role is not yet implemented by keystone[2], we can use
the legacy admin token which is currently
used in heat or other service-to-service interaction.
Thanks gmann for inputs. HoweverI think we still need a different role here. The role I've mentioned here is not the role assigned to the Heat
service user but the project-roleassigned to users to allow them create stacks with system resources like nova flavor or keystone user.This role is assigned to actual cloud users so service would not be appropriate.
I thought of using the admin role initially but this means that we
provide a way to "bypass" policy checkand expose system operations to project users.
As a side note, current heat does not use its own credential but user
credentials used for stack API calls.
We can introduce this mechanism and let heat manage system resources by
its service user but that isthe completely new feature we need to implement.
Thanks for the detailed information. For service to service interaction, I think heat needs to start using the configured service user token instead of using actual user token.
For example nova use for neutron and similarly for other service interaction
https://zuul.opendev.org/t/openstack/build/654ccb626d0e41a69308db1ed3328824/...
I'll look into that option but It might take some time because I've not yet evaluated the impact of that change. My current idea is to use service credentials for system resources only so that we can keep using the user credential for project resources.
As heat allows only project users to manage stacks, we need to determine the project role to allow creating stacks with such system resources. Does anyone have any suggestions about this ?
Do we agree we can allow project-admin to create stacks with flavors or keystone users as the previous policy rules allow or should we implement a new role to restrict management of system resources ?
-gmann
[1]
http://lists.openstack.org/pipermail/openstack-discuss/2022-January/026777.h...
[2] https://review.opendev.org/c/openstack/keystone-specs/+/818616
-gmann
Thank you, Takashi
---- On Wed, 09 Mar 2022 19:28:03 -0600 Takashi Kajinami tkajinam@redhat.com wrote ----
On Thu, Mar 10, 2022 at 10:13 AM Ghanshyam Mann gmann@ghanshyammann.com wrote: ---- On Wed, 09 Mar 2022 17:14:02 -0600 Takashi Kajinami tkajinam@redhat.com wrote ----
On Thu, Mar 10, 2022 at 2:59 AM Ghanshyam Mann gmann@ghanshyammann.com wrote: ---- On Wed, 09 Mar 2022 06:43:03 -0600 Takashi Kajinami tkajinam@redhat.com wrote ----
Hi Rico. Thank you for your insights.Please find my replies inline.
On Wed, Mar 9, 2022 at 6:22 PM Rico Lin ricolin@ricolky.com wrote:
On Wed, Mar 9, 2022 at 1:57 PM Takashi Kajinami tkajinam@redhat.com wrote:
Hello,
I've been working on updating policy rules in heat according to the latest guideline for SRBAC implementation.
Thanks so much for working on it, definitely +1 on enhancing security.
[1] https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rba...
However this change reveals one potential issue with system resources.
In heat there are some resource types like OS::Nova::Flavor or OS::Keystone::User which is allowed for only system users. (I'll call these resources as "system resources" here)
We create those resources based on checking `role:admin`, so project admin is accepted.This means a user under that project with role:admin, can operate those resources.I'm not sure what system user means here, so correct me if I'm wrong. The core problem is that backend services like nova enforces different policies from just role:admin whenSRBAC is enforced. I reported that a while ago as https://storyboard.openstack.org/#!/story/2009787 .
The main problem here is that heat currently supports resources which require different scopes of credentials.For example, once all deprecated policy rules are removed, nova allows project users to create instances(OS::Nova::Server)[1],while nova allows only system admins to create flavors(OS::Nova::Flavor)[2]. Currently Heat leverages the user credentialused to create a stack when Heat requests backend services to create its resources, and it's "impossible" to create a stackwith these two resources by a single credential (as each token can be scoped by project or system. [1] https://github.com/openstack/nova/blob/d1b036fdb124d1ef48f5cf024d4677bfe5b48...] https://github.com/openstack/nova/blob/d1b036fdb124d1ef48f5cf024d4677bfe5b48... If we keep allowing system users to create stacks, then it would allow users to create system resourceslike flavors. But this never solves the problem with the "mixed" pattern which I described above.Hope this will explain these three options more clearly.
Because heat uses user credentials to create resources in backend services, if we require project scope for stack/resource management then we are no longer able to create these system resources as part of stack. (*1)> There are some options I can think of at this moment.
- Deprecate and remove all resources which require system scope
Pros
- This is most easy solution to be implemented
Cons
- This one has the biggest user impact
Don't :)
Yeah, not sure how removing will work.
- Use service credential for resource creation
Pros:
- This might have smallest user impact
Cons:
- This requires a relatively big change in heat's architecture, which I've not yet evaluated.
- We need to determine the project scope role to be allowed to do system operation.
I think this is something we can discuss more.
- Implement additional project-role in each service to allow creation via stack creation.
Pros:
- This would avoid user impact.
Cons:
- requires work in multiple components.
- We need to determine the project scope role to be allowed to do system operation
Indeed current stack operation require `(role:admin and system_scope:all) OR (role:member and project_id:%(project_id)s)`.And if we remove the system part and keep `(role:member and project_id:%(project_id)s)`, will mean only a user in that project with a member role can operate stack, and if the user got also admin role, that user can manage User/Flavor resources through stack too.The main problem here is that nova will no longer allow project admins to create flavors,and it will require system admin when SRBAC is enforced(which means all deprecated rulesare disabled)
Also, note thatFor the case of Nova::Flavor, both policies from Nova and Heat resource will be enforced here, so it only accepts the intersection part of rules. So if we go with removing the system admin, that means it's time to also remove some deprecated global scope operation. IMO, to accept Project Admin or member make more reasonable (like you proposed in option 3)
This is a similar case that we discussed in the service role ML thread[1] or in the spec[2].
For such cases only, where services need to interact with another service we will have the 'service' as a special role. In this case, the heat having the project scoped token will use the 'service' role token for Nova Flavor APIs calls. Now whether that 'service' role token needs to be system scoped or not is open question Mark brought up in that discussion. IMO, we should use the scoped token even in case of 'service' role and do not skip scope checks for 'service' role in oslo policy as it can be a valid role used by any users at production. So with the new SRBAC, heat needs to use the system scoped 'service' role token.
But as 'service' role is not yet implemented by keystone[2], we can use the legacy admin token which is currently used in heat or other service-to-service interaction.
Thanks gmann for inputs. HoweverI think we still need a different role here. The role I've mentioned here is not the role assigned to the Heat service user but the project-roleassigned to users to allow them create stacks with system resources like nova flavor or keystone user.This role is assigned to actual cloud users so service would not be appropriate.
I thought of using the admin role initially but this means that we provide a way to "bypass" policy checkand expose system operations to project users.
As a side note, current heat does not use its own credential but user credentials used for stack API calls. We can introduce this mechanism and let heat manage system resources by its service user but that isthe completely new feature we need to implement.
Thanks for the detailed information. For service to service interaction, I think heat needs to start using the configured service user token instead of using actual user token.
For example nova use for neutron and similarly for other service interaction
I'll look into that option but It might take some time because I've not yet evaluated the impact of that change.My current idea is to use service credentials for system resources only so that we can keep using the user credentialfor project resources.
As heat allows only project users to manage stacks, we need to determine the project role to allow creatingstacks with such system resources. Does anyone have any suggestions about this ? Do we agree we can allow project-admin to create stacks with flavors or keystone users as the previous policy rulesallow or should we implement a new role to restrict management of system resources ?
Currently, existing way of creating flavor with project user should work as Nova still support the legacy policy so yes project admin can do that. And on the new role, I think that is what the 'service' role will be.
I think the question we are discussing here and on spec is what scope the project should use for 'service' role requests for their service-to-service interaction? IMO, it can be the same scope that calling APIs has. For heat example, heat can create a 'service' role with the 'system' scope and call Nova to create flavor. Another option is we skip the scope check for 'service' role as a special case but this can create issue when any production deployment has a role with the name of 'service'. Any other thoughts on this?
-gmann
-gmann
[1] http://lists.openstack.org/pipermail/openstack-discuss/2022-January/026777.h... [2] https://review.opendev.org/c/openstack/keystone-specs/+/818616
-gmann
Thank you, Takashi
On Wed, Mar 16, 2022 at 12:23 AM Ghanshyam Mann gmann@ghanshyammann.com wrote:
---- On Wed, 09 Mar 2022 19:28:03 -0600 Takashi Kajinami < tkajinam@redhat.com> wrote ----
On Thu, Mar 10, 2022 at 10:13 AM Ghanshyam Mann <
gmann@ghanshyammann.com> wrote:
---- On Wed, 09 Mar 2022 17:14:02 -0600 Takashi Kajinami <
tkajinam@redhat.com> wrote ----
On Thu, Mar 10, 2022 at 2:59 AM Ghanshyam Mann <
gmann@ghanshyammann.com> wrote:
---- On Wed, 09 Mar 2022 06:43:03 -0600 Takashi Kajinami <
tkajinam@redhat.com> wrote ----
Hi Rico. Thank you for your insights.Please find my replies inline.
On Wed, Mar 9, 2022 at 6:22 PM Rico Lin ricolin@ricolky.com
wrote:
On Wed, Mar 9, 2022 at 1:57 PM Takashi Kajinami <
tkajinam@redhat.com> wrote:
Hello,
I've been working on updating policy rules in heat according to
the latest
guideline for SRBAC implementation.
Thanks so much for working on it, definitely +1 on enhancing
security.
[1]
https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rba...
However this change reveals one potential issue with system
resources.
In heat there are some resource types like OS::Nova::Flavor or
OS::Keystone::User
which is allowed for only system users. (I'll call these
resources as "system resources" here)
We create those resources based on checking `role:admin`, so
project admin is accepted.This means a user under that project with role:admin, can operate those resources.I'm not sure what system user means here, so correct me if I'm wrong.
The core problem is that backend services like nova enforces
different policies from just role:admin whenSRBAC is enforced. I reported that a while ago as https://storyboard.openstack.org/#!/story/2009787 .
The main problem here is that heat currently supports resources
which require different scopes of credentials.For example, once all deprecated policy rules are removed, nova allows project users to create instances(OS::Nova::Server)[1],while nova allows only system admins to create flavors(OS::Nova::Flavor)[2]. Currently Heat leverages the user credentialused to create a stack when Heat requests backend services to create its resources, and it's "impossible" to create a stackwith these two resources by a single credential (as each token can be scoped by project or system.
[1]
https://github.com/openstack/nova/blob/d1b036fdb124d1ef48f5cf024d4677bfe5b48...] https://github.com/openstack/nova/blob/d1b036fdb124d1ef48f5cf024d4677bfe5b48...
If we keep allowing system users to create stacks, then it would
allow users to create system resourceslike flavors. But this never solves the problem with the "mixed" pattern which I described above.Hope this will explain these three options more clearly.
Because heat uses user credentials to create resources in
backend services, if we require
project scope for stack/resource management then we are no
longer able to create these system
resources as part of stack. (*1)> There are some options I can think of at this moment.
- Deprecate and remove all resources which require system scope
Pros
- This is most easy solution to be implemented
Cons
- This one has the biggest user impact
Don't :)
Yeah, not sure how removing will work.
- Use service credential for resource creation
Pros:
- This might have smallest user impact
Cons:
- This requires a relatively big change in heat's architecture,
which I've not yet evaluated.
- We need to determine the project scope role to be allowed to
do system operation.
I think this is something we can discuss more.
- Implement additional project-role in each service to allow
creation via stack creation.
Pros:
- This would avoid user impact.
Cons:
- requires work in multiple components.
- We need to determine the project scope role to be allowed to
do system operation
Indeed current stack operation require `(role:admin and
system_scope:all) OR (role:member and project_id:%(project_id)s)`.And if we remove the system part and keep `(role:member and project_id:%(project_id)s)`, will mean only a user in that project with a member role can operate stack, and if the user got also admin role, that user can manage User/Flavor resources through stack too.The main problem here is that nova will no longer allow project admins to create flavors,and it will require system admin when SRBAC is enforced(which means all deprecated rulesare disabled)
Also, note thatFor the case of Nova::Flavor, both policies from
Nova and Heat resource will be enforced here, so it only accepts the intersection part of rules.
So if we go with removing the system admin, that means it's time
to also remove some deprecated global scope operation.
IMO, to accept Project Admin or member make more reasonable (like
you proposed in option 3)
This is a similar case that we discussed in the service role ML
thread[1] or in the spec[2].
For such cases only, where services need to interact with another
service we will have the 'service' as
a special role. In this case, the heat having the project scoped
token will use the 'service' role token for
Nova Flavor APIs calls. Now whether that 'service' role token needs
to be system scoped or not is
open question Mark brought up in that discussion. IMO, we should use
the scoped token even in case
of 'service' role and do not skip scope checks for 'service' role in
oslo policy as it can be a valid role used
by any users at production. So with the new SRBAC, heat needs to use
the system scoped 'service' role token.
But as 'service' role is not yet implemented by keystone[2], we can
use the legacy admin token which is currently
used in heat or other service-to-service interaction.
Thanks gmann for inputs. HoweverI think we still need a different role here. The role I've mentioned here is not the role assigned to the Heat
service user but the project-roleassigned to users to allow them create stacks with system resources like nova flavor or keystone user.This role is assigned to actual cloud users so service would not be appropriate.
I thought of using the admin role initially but this means that we
provide a way to "bypass" policy checkand expose system operations to project users.
As a side note, current heat does not use its own credential but
user credentials used for stack API calls.
We can introduce this mechanism and let heat manage system resources
by its service user but that isthe completely new feature we need to implement.
Thanks for the detailed information. For service to service
interaction, I think heat needs to start using the configured
service user token instead of using actual user token.
For example nova use for neutron and similarly for other service
interaction
https://zuul.opendev.org/t/openstack/build/654ccb626d0e41a69308db1ed3328824/...
I'll look into that option but It might take some time because I've not
yet evaluated the impact of that change.My current idea is to use service credentials for system resources only so that we can keep using the user credentialfor project resources.
As heat allows only project users to manage stacks, we need to
determine the project role to allow creatingstacks with such system resources. Does anyone have any suggestions about this ?
Do we agree we can allow project-admin to create stacks with flavors or
keystone users as the previous policy rulesallow or should we implement a new role to restrict management of system resources ?
Currently, existing way of creating flavor with project user should work as Nova still support the legacy policy so yes project admin can do that. And on the new role, I think that is what the 'service' role will be.
I think the question we are discussing here and on spec is what scope the project should use for 'service' role requests for their service-to-service interaction? IMO, it can be the same scope that calling APIs has. For heat example, heat can create a 'service' role with the 'system' scope and call Nova to create flavor. Another option is we skip the scope check for 'service' role as a special case but this can create issue when any production deployment has a role with the name of 'service'. Any other thoughts on this?
No. The current question is related to the scope used by the user creating a stack.
"Stack" in heat is the resource belonging to individual projects, so it should be created by project users.
If we introduce the mechanism to make the heat service user create system resources like flavors on behalf of that project user, Heat would allow project users(eg. project admin) to create a flavor, while Nova doesn't allow it and requires system admin. We should consider what would be the appropriate project role required here.
-gmann
-gmann
[1]
http://lists.openstack.org/pipermail/openstack-discuss/2022-January/026777.h...
[2] https://review.opendev.org/c/openstack/keystone-specs/+/818616
-gmann
Thank you, Takashi
---- On Tue, 15 Mar 2022 10:45:05 -0500 Takashi Kajinami tkajinam@redhat.com wrote ----
On Wed, Mar 16, 2022 at 12:23 AM Ghanshyam Mann gmann@ghanshyammann.com wrote: ---- On Wed, 09 Mar 2022 19:28:03 -0600 Takashi Kajinami tkajinam@redhat.com wrote ----
On Thu, Mar 10, 2022 at 10:13 AM Ghanshyam Mann gmann@ghanshyammann.com wrote: ---- On Wed, 09 Mar 2022 17:14:02 -0600 Takashi Kajinami tkajinam@redhat.com wrote ----
On Thu, Mar 10, 2022 at 2:59 AM Ghanshyam Mann gmann@ghanshyammann.com wrote: ---- On Wed, 09 Mar 2022 06:43:03 -0600 Takashi Kajinami tkajinam@redhat.com wrote ----
Hi Rico. Thank you for your insights.Please find my replies inline.
On Wed, Mar 9, 2022 at 6:22 PM Rico Lin ricolin@ricolky.com wrote:
On Wed, Mar 9, 2022 at 1:57 PM Takashi Kajinami tkajinam@redhat.com wrote:
Hello,
I've been working on updating policy rules in heat according to the latest guideline for SRBAC implementation.
Thanks so much for working on it, definitely +1 on enhancing security.
[1] https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rba...
However this change reveals one potential issue with system resources.
In heat there are some resource types like OS::Nova::Flavor or OS::Keystone::User which is allowed for only system users. (I'll call these resources as "system resources" here)
We create those resources based on checking `role:admin`, so project admin is accepted.This means a user under that project with role:admin, can operate those resources.I'm not sure what system user means here, so correct me if I'm wrong. The core problem is that backend services like nova enforces different policies from just role:admin whenSRBAC is enforced. I reported that a while ago as https://storyboard.openstack.org/#!/story/2009787 .
The main problem here is that heat currently supports resources which require different scopes of credentials.For example, once all deprecated policy rules are removed, nova allows project users to create instances(OS::Nova::Server)[1],while nova allows only system admins to create flavors(OS::Nova::Flavor)[2]. Currently Heat leverages the user credentialused to create a stack when Heat requests backend services to create its resources, and it's "impossible" to create a stackwith these two resources by a single credential (as each token can be scoped by project or system. [1] https://github.com/openstack/nova/blob/d1b036fdb124d1ef48f5cf024d4677bfe5b48...] https://github.com/openstack/nova/blob/d1b036fdb124d1ef48f5cf024d4677bfe5b48... If we keep allowing system users to create stacks, then it would allow users to create system resourceslike flavors. But this never solves the problem with the "mixed" pattern which I described above.Hope this will explain these three options more clearly.
Because heat uses user credentials to create resources in backend services, if we require project scope for stack/resource management then we are no longer able to create these system resources as part of stack. (*1)> There are some options I can think of at this moment.
- Deprecate and remove all resources which require system scope
Pros
- This is most easy solution to be implemented
Cons
- This one has the biggest user impact
Don't :)
Yeah, not sure how removing will work.
- Use service credential for resource creation
Pros:
- This might have smallest user impact
Cons:
- This requires a relatively big change in heat's architecture, which I've not yet evaluated.
- We need to determine the project scope role to be allowed to do system operation.
I think this is something we can discuss more.
- Implement additional project-role in each service to allow creation via stack creation.
Pros:
- This would avoid user impact.
Cons:
- requires work in multiple components.
- We need to determine the project scope role to be allowed to do system operation
Indeed current stack operation require `(role:admin and system_scope:all) OR (role:member and project_id:%(project_id)s)`.And if we remove the system part and keep `(role:member and project_id:%(project_id)s)`, will mean only a user in that project with a member role can operate stack, and if the user got also admin role, that user can manage User/Flavor resources through stack too.The main problem here is that nova will no longer allow project admins to create flavors,and it will require system admin when SRBAC is enforced(which means all deprecated rulesare disabled)
Also, note thatFor the case of Nova::Flavor, both policies from Nova and Heat resource will be enforced here, so it only accepts the intersection part of rules. So if we go with removing the system admin, that means it's time to also remove some deprecated global scope operation. IMO, to accept Project Admin or member make more reasonable (like you proposed in option 3)
This is a similar case that we discussed in the service role ML thread[1] or in the spec[2].
For such cases only, where services need to interact with another service we will have the 'service' as a special role. In this case, the heat having the project scoped token will use the 'service' role token for Nova Flavor APIs calls. Now whether that 'service' role token needs to be system scoped or not is open question Mark brought up in that discussion. IMO, we should use the scoped token even in case of 'service' role and do not skip scope checks for 'service' role in oslo policy as it can be a valid role used by any users at production. So with the new SRBAC, heat needs to use the system scoped 'service' role token.
But as 'service' role is not yet implemented by keystone[2], we can use the legacy admin token which is currently used in heat or other service-to-service interaction.
Thanks gmann for inputs. HoweverI think we still need a different role here. The role I've mentioned here is not the role assigned to the Heat service user but the project-roleassigned to users to allow them create stacks with system resources like nova flavor or keystone user.This role is assigned to actual cloud users so service would not be appropriate.
I thought of using the admin role initially but this means that we provide a way to "bypass" policy checkand expose system operations to project users.
As a side note, current heat does not use its own credential but user credentials used for stack API calls. We can introduce this mechanism and let heat manage system resources by its service user but that isthe completely new feature we need to implement.
Thanks for the detailed information. For service to service interaction, I think heat needs to start using the configured service user token instead of using actual user token.
For example nova use for neutron and similarly for other service interaction
I'll look into that option but It might take some time because I've not yet evaluated the impact of that change.My current idea is to use service credentials for system resources only so that we can keep using the user credentialfor project resources.
As heat allows only project users to manage stacks, we need to determine the project role to allow creatingstacks with such system resources. Does anyone have any suggestions about this ? Do we agree we can allow project-admin to create stacks with flavors or keystone users as the previous policy rulesallow or should we implement a new role to restrict management of system resources ?
Currently, existing way of creating flavor with project user should work as Nova still support the legacy policy so yes project admin can do that. And on the new role, I think that is what the 'service' role will be.
I think the question we are discussing here and on spec is what scope the project should use for 'service' role requests for their service-to-service interaction? IMO, it can be the same scope that calling APIs has. For heat example, heat can create a 'service' role with the 'system' scope and call Nova to create flavor. Another option is we skip the scope check for 'service' role as a special case but this can create issue when any production deployment has a role with the name of 'service'. Any other thoughts on this? No. The current question is related to the scope used by the user creating a stack.
"Stack" in heat is the resource belonging to individual projects, so it should be created by project users. If we introduce the mechanism to make the heat service user create system resources like flavors on behalfof that project user, Heat would allow project users(eg. project admin) to create a flavor, while Nova doesn't allow itand requires system admin.We should consider what would be the appropriate project role required here.
Yeah, I understand the issue. I am not a Heat expert but from APIs point of view these kinds of APIs need more granularity especially when an APIs for creating one resource create another resource in other service and user might not be aware of that. If it cannot be made granular form Heat perspective then we may want to think of a special role like you suggested.
I am not sure how many such examples (creating different resources in other services and with different scope operations) we have like this for other service-to-service interactions (at least not in nova).
I am adding it for the keystone PTG etherpad and let's discuss this case along with the general 'service' role case.
-gmann
-gmann
-gmann
[1] http://lists.openstack.org/pipermail/openstack-discuss/2022-January/026777.h... [2] https://review.opendev.org/c/openstack/keystone-specs/+/818616
-gmann
Thank you, Takashi
participants (3)
-
Ghanshyam Mann
-
Rico Lin
-
Takashi Kajinami