[dev][nova][glance] Interesting bug about deleting shelved server snapshot
I came across this bug during triage today: https://bugs.launchpad.net/nova/+bug/1807110 They are advocating that nova/glance somehow keep a shelved server snapshot image from being inadvertently deleted by the user since it could result in data loss as they can't unshelve the server later (there is metadata in nova that links the shelved server to the snapshot image in glance which is used during unshelve). I don't see a base description field on images but I suppose nova could write a description property that explains what the snapshot is and warn against deleting it. Going a step further, nova could potentially set the protected flag to true so the image cannot be deleted, but I have two concerns about that: 1. I don't see any way to force delete a protected image in glance - does that exist or has it been discussed before? 2. Would the user be able to PATCH the image to change the protected value to false and then delete the image if they really wanted to? The other problem with nova marking the image as protected is that if the user deletes the server, the compute API tries to delete the snapshot image [1] which would fail if it's still protected, and then we could see snapshot images getting orphaned in glance. Arguably nova could detect this situation, update the protected field to false, and then delete the image. Other thoughts? Has this come up before? [1] https://github.com/openstack/nova/blob/c9dca64fa64005e5bea327f06a7a3f4821ab7... -- Thanks, Matt
On Thu, Dec 6, 2018 at 12:52 PM Matt Riedemann <mriedemos@gmail.com> wrote:
I came across this bug during triage today:
https://bugs.launchpad.net/nova/+bug/1807110
They are advocating that nova/glance somehow keep a shelved server snapshot image from being inadvertently deleted by the user since it could result in data loss as they can't unshelve the server later (there is metadata in nova that links the shelved server to the snapshot image in glance which is used during unshelve).
I don't see a base description field on images but I suppose nova could write a description property that explains what the snapshot is and warn against deleting it.
Going a step further, nova could potentially set the protected flag to true so the image cannot be deleted, but I have two concerns about that:
1. I don't see any way to force delete a protected image in glance - does that exist or has it been discussed before?
2. Would the user be able to PATCH the image to change the protected value to false and then delete the image if they really wanted to?
would they need too? if they wanted to delete the snapshot could thye not just delete the shelved instnace. if the snapshot is goin i assume we will not be able to unshvel it anyway by falling back to the base image or something like that so is there a usecase where deleteing the snap shot leave the shelved instance in a valid unshelvable state? if not i think setting the protected flag is ok to do.
The other problem with nova marking the image as protected is that if the user deletes the server, the compute API tries to delete the snapshot image [1] which would fail if it's still protected, and then we could see snapshot images getting orphaned in glance. Arguably nova could detect this situation, update the protected field to false, and then delete the image.
that seams sane to me. if nova set teh protected field when shelving the instance it shold be able to unprotect the snapshot when unshelving.
Other thoughts? Has this come up before?
[1] https://github.com/openstack/nova/blob/c9dca64fa64005e5bea327f06a7a3f4821ab7...
--
Thanks,
Matt
On 12/6/2018 8:14 AM, Sean Mooney wrote:
2. Would the user be able to PATCH the image to change the protected value to false and then delete the image if they really wanted to? would they need too? if they wanted to delete the snapshot could thye not just delete the shelved instnace. if the snapshot is goin i assume we will not be able to unshvel it anyway by falling back to the base image or something like that so is there a usecase where deleteing the snap shot leave the shelved instance in a valid unshelvable state? if not i think setting the protected flag is ok to do.
I'm having a hard time understanding what you're saying. Are you saying, the user should delete the protected snapshot via deleting the shelved server? I don't think that's very clear. But yes you can't unshelve the instance if the image is deleted (or if the user does not have access to it, which is a separate bug [1675791]). I think you're just saying, the user shouldn't need to delete the protected shelve snapshot image and if they do, the server should be deleted as well.
The other problem with nova marking the image as protected is that if the user deletes the server, the compute API tries to delete the snapshot image [1] which would fail if it's still protected, and then we could see snapshot images getting orphaned in glance. Arguably nova could detect this situation, update the protected field to false, and then delete the image. that seams sane to me. if nova set teh protected field when shelving the instance it shold be able to unprotect the snapshot when unshelving.
It's not unshelve, it's delete. -- Thanks, Matt
On Thu, Dec 6, 2018 at 2:50 PM Matt Riedemann <mriedemos@gmail.com> wrote:
On 12/6/2018 8:14 AM, Sean Mooney wrote:
2. Would the user be able to PATCH the image to change the protected value to false and then delete the image if they really wanted to? would they need too? if they wanted to delete the snapshot could thye not just delete the shelved instnace. if the snapshot is goin i assume we will not be able to unshvel it anyway by falling back to the base image or something like that so is there a usecase where deleteing the snap shot leave the shelved instance in a valid unshelvable state? if not i think setting the protected flag is ok to do.
I'm having a hard time understanding what you're saying. Are you saying, the user should delete the protected snapshot via deleting the shelved server? I don't think that's very clear. But yes you can't unshelve the instance if the image is deleted (or if the user does not have access to it, which is a separate bug [1675791]). I think you're just saying, the user shouldn't need to delete the protected shelve snapshot image and if they do, the server should be deleted as well.
yes sorry i did not say that clearly. basically i wanted to say that since the user would break the unshelving of an instance by deleting the snapshot nova created we should prevent them from doing that by setting the protected flag. if they really wanted to still delete the snappshot they should therefor delete the shelved instance which should cause nova to delete the snapshot.
The other problem with nova marking the image as protected is that if the user deletes the server, the compute API tries to delete the snapshot image [1] which would fail if it's still protected, and then we could see snapshot images getting orphaned in glance. Arguably nova could detect this situation, update the protected field to false, and then delete the image. that seams sane to me. if nova set teh protected field when shelving the instance it shold be able to unprotect the snapshot when unshelving.
It's not unshelve, it's delete.
sorry you are correct on deleting the instance i think nova should be able to unprotect the snapshot if the instnace is still shelved. that said there could be issues with this if someone manually booted another instance form the snapshot but im not sure if that would have other issues.
--
Thanks,
Matt
(Just addressing the specific Glance questions, not taking a position on the proposal.) On 12/6/18 7:45 AM, Matt Riedemann wrote:
I came across this bug during triage today:
https://bugs.launchpad.net/nova/+bug/1807110
They are advocating that nova/glance somehow keep a shelved server snapshot image from being inadvertently deleted by the user since it could result in data loss as they can't unshelve the server later (there is metadata in nova that links the shelved server to the snapshot image in glance which is used during unshelve).
I don't see a base description field on images but I suppose nova could write a description property that explains what the snapshot is and warn against deleting it.
Yes, any user can add a 'description' property (unless prohibited by property protections).
Going a step further, nova could potentially set the protected flag to true so the image cannot be deleted, but I have two concerns about that:
1. I don't see any way to force delete a protected image in glance - does that exist or has it been discussed before?
You cannot force delete a protected image in glance, but an admin can PATCH the image to update 'protected' to false, and then delete the image, which is functionally the same thing.
2. Would the user be able to PATCH the image to change the protected value to false and then delete the image if they really wanted to?
Yes, replacing the value of the 'protected' property on an image can be done by the image owner. (There is no specific policy for this other than the generic "modify_image" policy. I guess I should mention that there's also a "delete_image" policy. The default value for both policies is unrestricted ("").)
The other problem with nova marking the image as protected is that if the user deletes the server, the compute API tries to delete the snapshot image [1] which would fail if it's still protected, and then we could see snapshot images getting orphaned in glance. Arguably nova could detect this situation, update the protected field to false, and then delete the image.
Other thoughts? Has this come up before?
[1] https://github.com/openstack/nova/blob/c9dca64fa64005e5bea327f06a7a3f4821ab7...
participants (3)
-
Brian Rosmaita
-
Matt Riedemann
-
Sean Mooney