Unshelve in nova, cinder initialize_connection with user context
Hello, I have encountered what appears to be an edge-case instance shelving situation. I have to allow access to an internal API call to enable an "unprivileged" user to perform an instance operation. Specifically, I have a user with a "restricted permission" role in a project, who should only be able to start/stop/shelve/unshelve an existing instance. With sensible roles/actions in the nova policy, they are able to perform all actions, except /un/shelving. This should be controlled by a nova policy, however the user is unable to unshelve without permission to access the /cinder/ API call "initialize_connection() <https://developer.openstack.org/api-ref/block-storage/v3/?expanded=initialize-volume-attachment-detail#initialize-volume-attachment>". Without granting permission to the user, cinder rejects nova's request with an HTTP 403 status message. Outcomes of this situation are, curious and potentially unwanted. Curious being the user is able to /shelve/. Which leads to a point of no return for the user. But of particular concern is this API call exposes the backend storage driver's connection info to users. The RBD driver, as we use, does not even require a populated "connector" object before it replies. I was looking for advice/opinions on this issue, and hope someone here can help or direct me to the right place. My thoughts were that perhaps this is a bug or there's a configuration issue with our nova/cinder (note, i'm using mitaka, so maybe it's fixed upstream!). I am unable to check whether this is just a bug in nova, due to my lack of experience. But my rationale is nova is using the /user//'s/ context instead of an internal nova /admin/-like context to call the cinder API. As for configuration, I've been unable to pull up anything that seems relevant to this issue looking through /etc/{nova,cinder}/{nova,cinder}.conf, but maybe there's an option we've enabled that's led to this behavior. I'm happy to answer questions if anything is unclear. Sorry for verbosity, vagueness or inaccuracies -- I'm very new to Openstack, having started at zero a month ago and am using this task as a way to learn more. Any help is much appreciated :) Jordan
Hello,
I have encountered what appears to be an edge-case instance shelving situation. I have to allow access to an internal API call to enable an "unprivileged" user to perform an instance operation.
Specifically, I have a user with a "restricted permission" role in a project, who should only be able to start/stop/shelve/unshelve an existing instance. With sensible roles/actions in the nova policy, they are able to perform all actions, except /un/shelving. This should be controlled by a nova policy, however the user is unable to unshelve without permission to access the /cinder/ API call "initialize_connection() < https://developer.openstack.org/api-ref/block-storage/v3/?expanded=initializ...
". Without granting permission to the user, cinder rejects nova's request with an HTTP 403 status message.
Outcomes of this situation are, curious and potentially unwanted. Curious being the user is able to /shelve/. Which leads to a point of no return for the user. But of particular concern is this API call exposes the backend storage driver's connection info to users. The RBD driver, as we use, does not even require a populated "connector" object before it replies.
I was looking for advice/opinions on this issue, and hope someone here can help or direct me to the right place.
My thoughts were that perhaps this is a bug or there's a configuration issue with our nova/cinder (note, i'm using mitaka, so maybe it's fixed upstream!). I am unable to check whether this is just a bug in nova, due to my lack of experience. But my rationale is nova is using the /user//'s/ context instead of an internal nova /admin/-like context to call the cinder API. As for configuration, I've been unable to pull up anything that seems relevant to this issue looking through /etc/{nova,cinder}/{nova,cinder}.conf, but maybe there's an option we've enabled that's led to this behavior.
On Thu, 2019-04-11 at 16:04 +1200, Jordan Ansell wrote: this is an interesting usecase and while this is not a part of the code i useually deal with i be live your understand is correct in that we try intentionally to use the users auth context wehre ever possible. nova has/can have admin credential in its config for some service like cinder and neutron however for security reasons(primarily try to not store admin credentials on compute nodes) we try to not use the admin credentials. i dont know if we have a config knob you can change to modify this behavior. always using the admin credentials would be considerd an anti patteren but i also know that some work has been done in keystone with system tokens? servier tokens? something like that may facilitate this usecase in the future. hopefuly someone more knoladgeable in this area will chime in but in no surprised that you have encountered issues with this configuration as it is not tested in the gate. the default rule applie if its not tested assume its broken althohgh untested feature do work more often then you would expect.
I'm happy to answer questions if anything is unclear. Sorry for verbosity, vagueness or inaccuracies -- I'm very new to Openstack, having started at zero a month ago and am using this task as a way to learn more.
Any help is much appreciated :) Jordan
Hello, On Thu, Apr 11, 2019, at 01:25, Sean Mooney wrote:
Hello,
I have encountered what appears to be an edge-case instance shelving situation. I have to allow access to an internal API call to enable an "unprivileged" user to perform an instance operation.
Specifically, I have a user with a "restricted permission" role in a project, who should only be able to start/stop/shelve/unshelve an existing instance. With sensible roles/actions in the nova policy, they are able to perform all actions, except /un/shelving. This should be controlled by a nova policy, however the user is unable to unshelve without permission to access the /cinder/ API call "initialize_connection() < https://developer.openstack.org/api-ref/block-storage/v3/?expanded=initializ...
". Without granting permission to the user, cinder rejects nova's request with an HTTP 403 status message.
Outcomes of this situation are, curious and potentially unwanted. Curious being the user is able to /shelve/. Which leads to a point of no return for the user. But of particular concern is this API call exposes the backend storage driver's connection info to users. The RBD driver, as we use, does not even require a populated "connector" object before it replies.
I was looking for advice/opinions on this issue, and hope someone here can help or direct me to the right place.
My thoughts were that perhaps this is a bug or there's a configuration issue with our nova/cinder (note, i'm using mitaka, so maybe it's fixed upstream!). I am unable to check whether this is just a bug in nova, due to my lack of experience. But my rationale is nova is using the /user//'s/ context instead of an internal nova /admin/-like context to call the cinder API. As for configuration, I've been unable to pull up anything that seems relevant to this issue looking through /etc/{nova,cinder}/{nova,cinder}.conf, but maybe there's an option we've enabled that's led to this behavior.
On Thu, 2019-04-11 at 16:04 +1200, Jordan Ansell wrote: this is an interesting usecase and while this is not a part of the code i useually deal with i be live your understand is correct in that we try intentionally to use the users auth context wehre ever possible.
nova has/can have admin credential in its config for some service like cinder and neutron however for security reasons(primarily try to not store admin credentials on compute nodes) we try to not use the admin credentials.
i dont know if we have a config knob you can change to modify this behavior. always using the admin credentials would be considerd an anti patteren but i also know that some work has been done in keystone with system tokens? servier tokens? something like that may facilitate this usecase in the future.
hopefuly someone more knoladgeable in this area will chime in but in no surprised that you have encountered issues with this configuration as it is not tested in the gate. the default rule applie if its not tested assume its broken althohgh untested feature do work more often then you would expect.
Forgive my unfamiliarity with the nova and cinder APIs but it sounds like the shelve operation in nova is a purely nova-side API, while the unshelve operation calls cinder's volume-attach API if the instance had a mounted volume before it was shelved. Since the operation involves cinder, you'd have to update the cinder policy to allow this unprivileged user to perform the volume-attach action. It's not really a bug, it's just fallout from having policies managed separately for separate services. Keystone's work with system scope[1] wouldn't really apply here, if I understand the situation correctly, because attaching a volume is an action that is specific to a resource within a project (tenant), it's not an action that affects the deployment as a whole or spans across multiple projects. Hope this helps. [1] https://docs.openstack.org/keystone/latest/contributor/services.html#system-... Colleen
I'm happy to answer questions if anything is unclear. Sorry for verbosity, vagueness or inaccuracies -- I'm very new to Openstack, having started at zero a month ago and am using this task as a way to learn more.
Any help is much appreciated :) Jordan
On 4/11/2019 10:52 AM, Colleen Murphy wrote:
Forgive my unfamiliarity with the nova and cinder APIs but it sounds like the shelve operation in nova is a purely nova-side API, while the unshelve operation calls cinder's volume-attach API if the instance had a mounted volume before it was shelved. Since the operation involves cinder, you'd have to update the cinder policy to allow this unprivileged user to perform the volume-attach action. It's not really a bug, it's just fallout from having policies managed separately for separate services.
Correct, the user has to also be able to attach/detach volumes to/from their server since shelve/unshelve does that under the covers more or less. The same actually happens with ports, but you have to configure nova to have access to neutron's port binding API which is elevated (admin or service user token) beyond the normal user auth. Mitaka doesn't have configuration like that for cinder in nova, it was added a bit later but there is a patch being backported which you could also try to backport to mitaka - but it might be a bit messy (the keystoneauth stuff wasn't being used by nova in mitaka): https://review.openstack.org/#/c/603044/ -- Thanks, Matt
On 12/04/19 2:14 PM, Matt Riedemann wrote:
On 4/11/2019 10:52 AM, Colleen Murphy wrote:
Forgive my unfamiliarity with the nova and cinder APIs but it sounds like the shelve operation in nova is a purely nova-side API, while the unshelve operation calls cinder's volume-attach API if the instance had a mounted volume before it was shelved. Since the operation involves cinder, you'd have to update the cinder policy to allow this unprivileged user to perform the volume-attach action. It's not really a bug, it's just fallout from having policies managed separately for separate services.
Correct, the user has to also be able to attach/detach volumes to/from their server since shelve/unshelve does that under the covers more or less.
The same actually happens with ports, but you have to configure nova to have access to neutron's port binding API which is elevated (admin or service user token) beyond the normal user auth.
Mitaka doesn't have configuration like that for cinder in nova, it was added a bit later but there is a patch being backported which you could also try to backport to mitaka - but it might be a bit messy (the keystoneauth stuff wasn't being used by nova in mitaka):
Thank you for that, I'll have a look into it and see if this patch is something we could use :) And thanks too to Colleen and Sean for your comments earlier and link to information about permission scope in keystone. Your comments, together with what Matt is saying here tidies up my understanding of just what is happening. For now we have gone with a fix along the same vein as what Colleen suggested, allowing the cinder os-initialize_connection action to be performed by the restricted user. Interesting to me is why nova (using the restricted user's context) is able to perform the attach volume operation in full (starting a stopped instance) but not the sub-operation of attaching a volume, just setting up the volume connection (as with unshelving). Also interesting is the connection information returned by the cinder action we're discussing, as this "SAN" information of no /practical/ use to a user and sensitive in some cases. It strengthens my belief that something akin to the patch Matt has shared ought to be something we consider. Regards, Jordan
On 4/11/2019 10:12 PM, Jordan Ansell wrote:
Also interesting is the connection information returned by the cinder action we're discussing, as this "SAN" information of no /practical/ use to a user and sensitive in some cases. It strengthens my belief that something akin to the patch Matt has shared ought to be something we consider.
I believe the issue you're talking about is this bug: https://bugs.launchpad.net/cinder/+bug/1736773 -- Thanks, Matt
participants (4)
-
Colleen Murphy
-
Jordan Ansell
-
Matt Riedemann
-
Sean Mooney