[nova][cinder][barbican] storing secrets in service project
Hi all, When Barbican is involved, and services like Nova and Cinder create secrets behind the scenes themselves, this results in problems deleting or otherwise managing some resources by cloud admins. For example as described in https://bugs.launchpad.net/cinder/+bug/1775372, admin can not delete an encrypted volume created by another user, and needs to resort to adding herself to the project first with appropriate roles. This stems IMO from the difference in default resource access policies between Barbican and other services that consume it. For example, while in Nova or Cinder, cloud admin by default can delete servers or volumes in other projects, in Barbican by default only secret creator or admin "in the same project" can delete a secret. I came up with at least 3 solutions (already mentioned in the issue above) but would like to discuss their validity, pros and cons, and especially how they affect the security model and the use cases that this Nova-Cinder-Barbican integration aims to solve. 1. Just ignore failure to delete a secret and log appropriate warning that a secret may be left orphaned. Obviously not the best solution, but rather simple one, also will protect from cases when for whatever reason a secret will be deleted from Barbican directly (e.g. by mistake), rendering volume undeletable for anyone. 2. Change default barbican policies to allow secret deletion for cloud admin (system scope?). Naturally has implications for the Barbican model for resources access, so may be not that appropriate. 3. Store the secrets in the service project instead of user's. Currently those secrets are generated by Nova or Cinder themselves, and user quite probably is not even aware that there was something created in Barbican on her behalf. The user also has no option to pass her own secrets to any of the involved API calls, so this is something happening completely behind the scenes. So do we really need to store these secrets in the user's project? Wouldn't it be better to store them in the service project, have the same service user defined in Nova and Cinder, so it can operate and access those secrets regardless of who manages the resource which depends on access to this secret? This also may solve the problem of encrypted local instance images in Nova, where upon host restart, Nova is not able to bring those instances up because it lacks the context capable of accessing those secrets. I think the third option is the best one, but I may be missing the actual use case meant behind this feature. Is it "keep the data encrypted at rest and in transit" or more strictly "encrypt data of different users with different keys and limit access to those keys" (for the case of breach etc) ? In the first case it seems storing all the auto-generated secrets in a service project would suffice... Please help me understand the best approach to take. Best regards, Pavlo.
Hi all,
When Barbican is involved, and services like Nova and Cinder create secrets behind the scenes themselves, this results in problems deleting or otherwise managing some resources by cloud admins.
On Tue, 2021-08-31 at 16:46 +0300, Pavlo Shchelokovskyy wrote: this is mostly by design. cloud admin shoudl not be able to access tenat secret or teants cannot trust the cloud.
For example as described in https://bugs.launchpad.net/cinder/+bug/1775372, admin can not delete an encrypted volume created by another user, and needs to resort to adding herself to the project first with appropriate roles.
yep although i woudl argue that unless that is an abuse of admin privialges. there are some case where the admin woudl have to intervient like this but its generally agaisnt the self service model for admin to be doing this.
This stems IMO from the difference in default resource access policies between Barbican and other services that consume it. For example, while in Nova or Cinder, cloud admin by default can delete servers or volumes in other projects, in Barbican by default only secret creator or admin "in the same project" can delete a secret.
I came up with at least 3 solutions (already mentioned in the issue above) but would like to discuss their validity, pros and cons, and especially how they affect the security model and the use cases that this Nova-Cinder-Barbican integration aims to solve.
1. Just ignore failure to delete a secret and log appropriate warning that a secret may be left orphaned. Obviously not the best solution, but rather simple one, also will protect from cases when for whatever reason a secret will be deleted from Barbican directly (e.g. by mistake), rendering volume undeletable for anyone.
2. Change default barbican policies to allow secret deletion for cloud admin (system scope?). Naturally has implications for the Barbican model for resources access, so may be not that appropriate.
3. Store the secrets in the service project instead of user's. Currently those secrets are generated by Nova or Cinder themselves, and user quite probably is not even aware that there was something created in Barbican on her behalf. The user also has no option to pass her own secrets to any of the involved API calls, so this is something happening completely behind the scenes. So do we really need to store these secrets in the user's project? Wouldn't it be better to store them in the service project, have the same service user defined in Nova and Cinder, so it can operate and access those secrets regardless of who manages the resource which depends on access to this secret? This also may solve the problem of encrypted local instance images in Nova, where upon host restart, Nova is not able to bring those instances up because it lacks the context capable of accessing those secrets.
I think the third option is the best one, but I may be missing the actual use case meant behind this feature.
i am not sure i agree. we intentionally do not allow admin to start instance that have encypted storage as the admin should not be able to access the decrypted data. yes they could always hack around this but to me this is intentioaly not supported and not a bug.
Is it "keep the data encrypted at rest and in transit" or more strictly "encrypt data of different users with different keys and limit access to those keys" (for the case of breach etc) ? In the first case it seems storing all the auto-generated secrets in a service project would suffice... its both the intent is to keep the data encyprted at rest and in transit and ensur that admin cannot decypt it. deletetion of hte resouce is perhaps somethign that could be allowable with a sytem admin token retrivial of the secreate e.g. to start the vm or do a snapthot for exmaple i think would be a regression in security and not something we shoudl do.
and yes im aware that admin can already change default plicy in barbican to allow them to retirve the securet or add themsevles to the project to circumvent some of the protections. i dont think its in scope of nova ot manage any user secreate or user data in general. moving this into the proejct like nova and cinder changes the security model and risk assment for operators and devleopers of thoes services. we woudl be adding user data that need to be managed in a way that is complient with strict standard where as by using barbican we can cerntrailse that to a singel service which has one role, manage secrets.
Please help me understand the best approach to take.
Best regards, Pavlo.
On 8/31/21 4:17 PM, Sean Mooney wrote:
Hi all,
When Barbican is involved, and services like Nova and Cinder create secrets behind the scenes themselves, this results in problems deleting or otherwise managing some resources by cloud admins.
On Tue, 2021-08-31 at 16:46 +0300, Pavlo Shchelokovskyy wrote: this is mostly by design. cloud admin shoudl not be able to access tenat secret or teants cannot trust the cloud.
access the secret != delete a secret ...
For example as described in https://bugs.launchpad.net/cinder/+bug/1775372, admin can not delete an encrypted volume created by another user, and needs to resort to adding herself to the project first with appropriate roles. yep although i woudl argue that unless that is an abuse of admin privialges. there are some case where the admin woudl have to intervient like this but its generally agaisnt the self service model for admin to be doing this.
It's perfectly legitimate for an admin to delete *ANY* resource if he feels he needs to. For example on a public cloud: after the grace period expired. Example for an internal cloud: when an employee leaves.
I think the third option is the best one, but I may be missing the actual use case meant behind this feature. i am not sure i agree.
we intentionally do not allow admin to start instance that have encypted storage as the admin should not be able to access the decrypted data. yes they could always hack around this but to me this is intentioaly not supported and not a bug.
It is a bug that we can't live-migrate LUKS volumes. Please do recognize this as an OpenStack defect, because if you don't, you wont recognize we need to fix the situation, which is kind of super bad.
Is it "keep the data encrypted at rest and in transit" or more strictly "encrypt data of different users with different keys and limit access to those keys" (for the case of breach etc) ? In the first case it seems storing all the auto-generated secrets in a service project would suffice... its both the intent is to keep the data encyprted at rest and in transit and ensur that admin cannot decypt it. deletetion of hte resouce is perhaps somethign that could be allowable with a sytem admin token retrivial of the secreate e.g. to start the vm or do a snapthot for exmaple i think would be a regression in security and not something we shoudl do.
We you can't start the VM, it means you can't live-migrate. So there's a big issue to tackle here. I don't agree that it's a big issue that an admin can access secrets, because it's already the case (ie: by adding the creator role in that project). If you don't trust your admin: go away, it's game over anyways. As for LUKS devices, it'd be trivial for an admin to dump the VM's memory and access the decryption key anyways... So you're only being annoying to the admins, and not protecting anyone here. Cheers, Thomas Goirand (zigo)
On 8/31/21 4:17 PM, Sean Mooney wrote:
Hi all,
When Barbican is involved, and services like Nova and Cinder create secrets behind the scenes themselves, this results in problems deleting or otherwise managing some resources by cloud admins.
On Tue, 2021-08-31 at 16:46 +0300, Pavlo Shchelokovskyy wrote: this is mostly by design. cloud admin shoudl not be able to access tenat secret or teants cannot trust the cloud.
access the secret != delete a secret ...
For example as described in https://bugs.launchpad.net/cinder/+bug/1775372, admin can not delete an encrypted volume created by another user, and needs to resort to adding herself to the project first with appropriate roles. yep although i woudl argue that unless that is an abuse of admin privialges. there are some case where the admin woudl have to intervient like this but its generally agaisnt the self service model for admin to be doing this.
It's perfectly legitimate for an admin to delete *ANY* resource if he feels he needs to. For example on a public cloud: after the grace period expired. Example for an internal cloud: when an employee leaves.
I think the third option is the best one, but I may be missing the actual use case meant behind this feature. i am not sure i agree.
we intentionally do not allow admin to start instance that have encypted storage as the admin should not be able to access the decrypted data. yes they could always hack around this but to me this is intentioaly not supported and not a bug.
It is a bug that we can't live-migrate LUKS volumes. Please do recognize this as an OpenStack defect, because if you don't, you wont recognize we need to fix the situation, which is kind of super bad.
On Thu, 2021-09-02 at 13:25 +0200, Thomas Goirand wrote: live-migration should actully work i belive for ceph. i think its just if the volume is host mounted like we do with isci that it fails. cold migration will partly work you just wont be able to start the instance so you can cold migrate a shut off instance. same for evacuate.
Is it "keep the data encrypted at rest and in transit" or more strictly "encrypt data of different users with different keys and limit access to those keys" (for the case of breach etc) ? In the first case it seems storing all the auto-generated secrets in a service project would suffice... its both the intent is to keep the data encyprted at rest and in transit and ensur that admin cannot decypt it. deletetion of hte resouce is perhaps somethign that could be allowable with a sytem admin token retrivial of the secreate e.g. to start the vm or do a snapthot for exmaple i think would be a regression in security and not something we shoudl do.
We you can't start the VM, it means you can't live-migrate.
sure but if its not running you can cold migrate it.
So there's a big issue to tackle here.
not really you should not be starting the instance in this case. if its off you should just cold migrate. that work in more cases the live migrate and is more effcinet since we wont have to copy any guest memory.
I don't agree that it's a big issue that an admin can access secrets, because it's already the case (ie: by adding the creator role in that project).
If you don't trust your admin: go away, it's game over anyways.
As for LUKS devices, it'd be trivial for an admin to dump the VM's memory and access the decryption key anyways... So you're only being annoying to the admins, and not protecting anyone here.
this is hte how point of secure comptueing and the sev/image incryption work that peopel have been tryign to add to nova. the ablity to upload encypted images and use them to create encypted volumes that the admin cant access which can be verifyed using vtpm and other methods is becoming more and more a requirement for many government customers. this is why SEV support was intoduced to prevent admin form doing that. its not about being annoying to admins its about ensureing we can actuly use openstack in secure enviroment so that we can continue to support our customer that need a miniuma or zero trust solution.
Cheers,
Thomas Goirand (zigo)
On Tue, 31 Aug 2021 at 14:50, Pavlo Shchelokovskyy <pshchelokovskyy@mirantis.com> wrote:
Hi all,
When Barbican is involved, and services like Nova and Cinder create secrets behind the scenes themselves, this results in problems deleting or otherwise managing some resources by cloud admins.
For example as described in https://bugs.launchpad.net/cinder/+bug/1775372, admin can not delete an encrypted volume created by another user, and needs to resort to adding herself to the project first with appropriate roles.
This stems IMO from the difference in default resource access policies between Barbican and other services that consume it. For example, while in Nova or Cinder, cloud admin by default can delete servers or volumes in other projects, in Barbican by default only secret creator or admin "in the same project" can delete a secret.
I came up with at least 3 solutions (already mentioned in the issue above) but would like to discuss their validity, pros and cons, and especially how they affect the security model and the use cases that this Nova-Cinder-Barbican integration aims to solve.
1. Just ignore failure to delete a secret and log appropriate warning that a secret may be left orphaned. Obviously not the best solution, but rather simple one, also will protect from cases when for whatever reason a secret will be deleted from Barbican directly (e.g. by mistake), rendering volume undeletable for anyone.
2. Change default barbican policies to allow secret deletion for cloud admin (system scope?). Naturally has implications for the Barbican model for resources access, so may be not that appropriate.
3. Store the secrets in the service project instead of user's. Currently those secrets are generated by Nova or Cinder themselves, and user quite probably is not even aware that there was something created in Barbican on her behalf. The user also has no option to pass her own secrets to any of the involved API calls, so this is something happening completely behind the scenes. So do we really need to store these secrets in the user's project? Wouldn't it be better to store them in the service project, have the same service user defined in Nova and Cinder, so it can operate and access those secrets regardless of who manages the resource which depends on access to this secret? This also may solve the problem of encrypted local instance images in Nova, where upon host restart, Nova is not able to bring those instances up because it lacks the context capable of accessing those secrets.
I think the third option is the best one, but I may be missing the actual use case meant behind this feature. Is it "keep the data encrypted at rest and in transit" or more strictly "encrypt data of different users with different keys and limit access to those keys" (for the case of breach etc) ? In the first case it seems storing all the auto-generated secrets in a service project would suffice...
Please help me understand the best approach to take.
Some loosely-related context: There are certainly some operational issues when using encrypted volumes. In a recent project we used custom policy to meet the requirements to avoid some issues. We configured nova & cinder to send a service token to barbican, and configured the barbican policy to only allow secret decryption for certain combinations of user roles and service roles. This allows an operator with a particular role to migrate another user's instance, but prevents them from reading the secret directly since they wouldn't have the service role. Unfortunately it was not possible to do the same for barbican secrets since there is a hard coded project check in the barbican DB layer which bypasses policy. Some related patches: * https://review.opendev.org/c/openstack/castellan/+/769560 * https://review.opendev.org/c/openstack/nova/+/781079 Other patches are yet to make it out of the environment.
Best regards, Pavlo.
participants (4)
-
Mark Goddard
-
Pavlo Shchelokovskyy
-
Sean Mooney
-
Thomas Goirand