[nova][glance] Granting image member access for snapshots (bug 1675791)
I have a fix proposed for a pretty old bug (1675791 [1]). This originally came up because of a scenario where an admin shelves a server and then the owner of the shelved server cannot unshelve it since they do not have access to the shelve snapshot image. The same is true for normal snapshot and backup operations though, see this proposed spec for Stein [2]. It also came up during the cross-cell resize spec review [3] since that solution depends on snapshot to get the root disk from one cell to another. In a nutshell, when creating a snapshot now, the compute API will check if the project creating the snapshot is the same as the project owner of the server. If not, the image is created with visibility=shared and the project owner of the instance is granted member access to the image, which allows them to GET the image directly via the ID, but not list it by default (the tenant user has to accept the pending membership for that). I have tested this out in devstack today and everything seems to work well. I am posting this to (a) raise awareness of the bug and proposed fix since it is sort of a behavior change in the createImage/createBackup/shelve APIs and (b) to make sure the glance team is aware and acknowledges this is an OK thing to do, i.e. are there any kind of unforeseen side effects of automatically granting image membership like this (I would think not since the owner of the instance has access to the root disk of the server anyway - it is their data). Also note that some really crusty legacy code in most of the in-tree virt drivers had to be removed (some virt drivers would change the image visibility back to private during the actual data upload to glance) which could mean out of tree drivers have the same issue. [1] https://bugs.launchpad.net/nova/+bug/1675791 [2] https://review.openstack.org/#/c/616843/ [3] https://review.openstack.org/#/c/616037/3/specs/stein/approved/cross-cell-re... -- Thanks, Matt
On 1/14/2019 6:04 PM, Matt Riedemann wrote:
I have a fix proposed
The proposed fix is here: https://review.openstack.org/#/c/630769/
for a pretty old bug (1675791 [1]). This originally came up because of a scenario where an admin shelves a server and then the owner of the shelved server cannot unshelve it since they do not have access to the shelve snapshot image.
The same is true for normal snapshot and backup operations though, see this proposed spec for Stein [2].
It also came up during the cross-cell resize spec review [3] since that solution depends on snapshot to get the root disk from one cell to another.
In a nutshell, when creating a snapshot now, the compute API will check if the project creating the snapshot is the same as the project owner of the server. If not, the image is created with visibility=shared and the project owner of the instance is granted member access to the image, which allows them to GET the image directly via the ID, but not list it by default (the tenant user has to accept the pending membership for that). I have tested this out in devstack today and everything seems to work well.
I am posting this to (a) raise awareness of the bug and proposed fix since it is sort of a behavior change in the createImage/createBackup/shelve APIs and (b) to make sure the glance team is aware and acknowledges this is an OK thing to do, i.e. are there any kind of unforeseen side effects of automatically granting image membership like this (I would think not since the owner of the instance has access to the root disk of the server anyway - it is their data).
Also note that some really crusty legacy code in most of the in-tree virt drivers had to be removed (some virt drivers would change the image visibility back to private during the actual data upload to glance) which could mean out of tree drivers have the same issue.
[1] https://bugs.launchpad.net/nova/+bug/1675791 [2] https://review.openstack.org/#/c/616843/ [3] https://review.openstack.org/#/c/616037/3/specs/stein/approved/cross-cell-re...
On 1/15/19 8:40 AM, Eric Fried wrote:
On 1/14/2019 6:04 PM, Matt Riedemann wrote:
I have a fix proposed
The proposed fix is here: https://review.openstack.org/#/c/630769/
for a pretty old bug (1675791 [1]). This originally came up because of a scenario where an admin shelves a server and then the owner of the shelved server cannot unshelve it since they do not have access to the shelve snapshot image.
The same is true for normal snapshot and backup operations though, see this proposed spec for Stein [2].
It also came up during the cross-cell resize spec review [3] since that solution depends on snapshot to get the root disk from one cell to another.
In a nutshell, when creating a snapshot now, the compute API will check if the project creating the snapshot is the same as the project owner of the server. If not, the image is created with visibility=shared and the project owner of the instance is granted member access to the image, which allows them to GET the image directly via the ID, but not list it by default (the tenant user has to accept the pending membership for that). I have tested this out in devstack today and everything seems to work well.
I am posting this to (a) raise awareness of the bug and proposed fix since it is sort of a behavior change in the createImage/createBackup/shelve APIs and (b) to make sure the glance team is aware and acknowledges this is an OK thing to do, i.e. are there any kind of unforeseen side effects of automatically granting image membership like this (I would think not since the owner of the instance has access to the root disk of the server anyway - it is their data).
This is an OK thing to do. The Glance image sharing philosophy is that the image owner can share with anyone and a shared image is immediately usable by any of its members. It's not accessible to any non-owner non-members, and as you point out, it won't "spam" an image member by showing up in the member's default image-list unless the member (not the sharer) decides otherwise. The only thing I can think of that might impact your propoal is the 'image_member_quota' configuration option in glance-api.conf, which limits how many projects a single image may be shared with. (It's not really a quota, it's a hard limit across all users.) The default is 128. There's no limit to how many images a single project can be a member of, so you don't need to worry about that.
Also note that some really crusty legacy code in most of the in-tree virt drivers had to be removed (some virt drivers would change the image visibility back to private during the actual data upload to glance) which could mean out of tree drivers have the same issue.
Yes, this is an important point to note. Since Ocata, when the 'shared' and 'community' visibility values were added, an image with visibility of 'private' is not accessible to any members of the image. (The visibility transition does not affect the member-list. The member-list persists even when the visibility has changed to a value that renders it inert.)
[1] https://bugs.launchpad.net/nova/+bug/1675791 [2] https://review.openstack.org/#/c/616843/ [3] https://review.openstack.org/#/c/616037/3/specs/stein/approved/cross-cell-re...
On 1/16/2019 8:34 AM, Brian Rosmaita wrote:
The only thing I can think of that might impact your propoal is the 'image_member_quota' configuration option in glance-api.conf, which limits how many projects a single image may be shared with. (It's not really a quota, it's a hard limit across all users.) The default is 128.
This shouldn't be a problem in this case. The image is still essentially private (protected?) since it's only shared between the admin tenant and the user tenant. -- Thanks, Matt
On 1/14/2019 6:04 PM, Matt Riedemann wrote:
I have a fix proposed for a pretty old bug (1675791 [1]). This originally came up because of a scenario where an admin shelves a server and then the owner of the shelved server cannot unshelve it since they do not have access to the shelve snapshot image.
The same is true for normal snapshot and backup operations though, see this proposed spec for Stein [2].
It also came up during the cross-cell resize spec review [3] since that solution depends on snapshot to get the root disk from one cell to another.
In a nutshell, when creating a snapshot now, the compute API will check if the project creating the snapshot is the same as the project owner of the server. If not, the image is created with visibility=shared and the project owner of the instance is granted member access to the image, which allows them to GET the image directly via the ID, but not list it by default (the tenant user has to accept the pending membership for that). I have tested this out in devstack today and everything seems to work well.
I am posting this to (a) raise awareness of the bug and proposed fix since it is sort of a behavior change in the createImage/createBackup/shelve APIs and (b) to make sure the glance team is aware and acknowledges this is an OK thing to do, i.e. are there any kind of unforeseen side effects of automatically granting image membership like this (I would think not since the owner of the instance has access to the root disk of the server anyway - it is their data).
Also note that some really crusty legacy code in most of the in-tree virt drivers had to be removed (some virt drivers would change the image visibility back to private during the actual data upload to glance) which could mean out of tree drivers have the same issue.
[1] https://bugs.launchpad.net/nova/+bug/1675791 [2] https://review.openstack.org/#/c/616843/ [3] https://review.openstack.org/#/c/616037/3/specs/stein/approved/cross-cell-re...
One more note about this - I have not yet tested doing a snapshot of a volume-backed server where the admin creates the snapshot and the tenant user tries to create a new server from that snapshot. In that case the tenant user should have access to the snapshot image, but they might not have access to the volume snapshot so that could still fail. For that to work, we'd likely need to force an ownership transfer of the volume to the tenant user (owner of the server) I guess. -- Thanks, Matt
Matt Riedemann <mriedemos@gmail.com> wrote on 01/16/2019 09:34:17 AM:
One more note about this - I have not yet tested doing a snapshot of a volume-backed server where the admin creates the snapshot and the tenant user tries to create a new server from that snapshot. In that case the tenant user should have access to the snapshot image, but they might not have access to the volume snapshot so that could still fail. For that to work, we'd likely need to force an ownership transfer of the volume to the tenant user (owner of the server) I guess.
Or depend on a new cinder capability for sharing volumes across projects, similar to how images can be shared across projects. This would be useful for other things as well. I do agree that in this case, forcing ownership transfer would probably be better than sharing. In fact, I wish we could do that for images here.
On 1/16/2019 9:22 AM, William M Edmonds wrote:
I do agree that in this case, forcing ownership transfer would probably be better than sharing. In fact, I wish we could do that for images here.
I'm pretty sure we can simply force the glance snapshot image to be owned privately by the tenant user that owns the instance by simply specifying: image['owner'] = instance.project_id The reason I didn't just do that was because it's a more drastic change in behavior than what we have today with the image being owned by the tenant that created the image. I also thought about making that configurable, but that is (1) kind of gross since it's config-driven API behavior which also makes it (2) not really interoperable, although that behavior could probably be discoverable by end users. If you go back to the proposed spec from Brin Zhang [1] the proposal there is to change the compute API to allow passing in the owner project_id so nova doesn't have to fumble with this. I could see that being reasonable for snapshots and backups, but I think the original bug about shelve is really just that - a bug, and easily fixed with what I've proposed (and is not a problem for volume-backed servers because shelve doesn't create a snapshot image for those). [1] https://review.openstack.org/#/c/616843/ -- Thanks, Matt
participants (4)
-
Brian Rosmaita
-
Eric Fried
-
Matt Riedemann
-
William M Edmonds