openstack volume snapshots and deleting
If I create a volume, snapshot it, and create a new volume from that snapshot, I cannot seem to delete the snapshot until the volume is deleted. Is this intentional, or am I doing something wrong? Is the new volume based on the snapshot, and that's why I can't delete it? We're using CEPH as a storage backend if that makes a difference. openstack volume snapshot create -volume base-volume base-ss openstack volume create -snapshot base-ss new-volume openstack volume snapshot delete base-ss The last command is the one that fails. It gives no error message. It just doesn't actually delete the snapshot. But If I do this: openstack volume delete new-volume openstack volume snapshot delete base-ss Then it works. I also tried creating a snapshot, creating a volume from the snapshot, then creating a volume from that volume, but I still cannot delete the snapshot until both volumes are deleted. I do not want everlasting snapshots, so I hope I'm doing something wrong that someone can point out. Thanks. --John
On Thu, Dec 05, 2019 at 03:24:35PM +0000, Ratliff, John wrote:
If I create a volume, snapshot it, and create a new volume from that snapshot, I cannot seem to delete the snapshot until the volume is deleted. Is this intentional, or am I doing something wrong? Is the new volume based on the snapshot, and that's why I can't delete it? We're using CEPH as a storage backend if that makes a difference.
That is the expected behavior as the snapshot is in-use by the volume created from it. So it can't be deleted until there isn't anything using it. Cloning would be the only way to get around this. There's not a quick and easy call to do this from a snapshot base, so unfortunately to get around that and get the final effect that it appears you are looking for, you would need to create a snapshot, create a volume from the snapshot, then *clone* yet another volume from that new one. Then delete the temporary volume and snapshot once the clone is done. As long as the backend driver is honoring the clone semantics, that should result in a fully independent volume that has its inital data set from the snapshot. So the process would look something like: openstack volume snapshot create --volume base-volume base-ss openstack volume create --snapshot base-ss temp-volume openstack volume create --volume temp-volume new-volume openstack volume delete temp-volume openstack volume snapshot delete base-ss Otherwise, it should be fairly safe to just keep the snapshot around. A lot of storage devices handle snapshots as internal pointers, so the only additional space consumption comes from anything new you end up writing to the new volume. Sean
Yes, I tried this second thing, but it doesn't work either. I created a new instance called test1.example.net. I created a volume snapshot (test1.example.net_ss1) I created a volume from that snapshot (vol_from_ss) I created a volume from the volume that was from the snapshot (vol_from_vol_from_ss) I deleted the vol_from_ss (this works) I deleted the snapshot (this does not work) If I then delete the vol_from_vol_from_ss volume, I can then delete the snapshot. P.S. My openstack cli says the option is --source, not --volume. Here is my command output. $ openstack volume list | grep test1 | cc698f20-c56f-47c7-b767-3c554c8cb517 | test1.example.net | in-use | 20 | Attached to test1.example.net on /dev/sda | $ openstack volume snapshot list | grep test1 $ openstack volume snapshot create --volume test1.example.net test1.example.net_ss1 Invalid volume: Volume cc698f20-c56f-47c7-b767-3c554c8cb517 status must be available, but current status is: in-use. (HTTP 400) (Request-ID: req-1ab60e86-3809-4a3b-9ac3-98e7d09ae7e9) $ openstack volume snapshot create --force --volume test1.example.net test1.example.net_ss1 $ openstack volume create --snapshot test1.example.net_ss1 vol_from_ss $ openstack volume create --volume vol_from_ss vol_from_vol_from_ss usage: openstack volume create [-h] [-f {json,shell,table,value,yaml}] [-c COLUMN] [--max-width <integer>] [--fit-width] [--print-empty] [--noindent] [--prefix PREFIX] [--size <size>] [--type <volume-type>] [--image <image> | --snapshot <snapshot> | --source <volume>] [--description <description>] [--availability-zone <availability-zone>] [--consistency-group consistency-group>] [--property <key=value>] [--hint <key=value>] [--multi-attach] [--bootable | --non-bootable] [--read-only | --read-write] <name> openstack volume create: error: unrecognized arguments: --volume vol_from_vol_from_ss $ openstack volume create --source vol_from_ss vol_from_vol_from_ss $ openstack volume delete vol_from_ss $ openstack volume snapshot delete test1.example.net_ss1 $ openstack volume snapshot list | grep test1 | 386186e9-6121-4222-b487-3dc026acf70d | test1.example.net_ss1 | None | available | 20 | $ openstack volume list | grep 'vol_\|test1' | ccbfdabe-d2ff-4bbd-a0ef-3b08935fe29a | vol_from_vol_from_ss | available | 20 | | | cc698f20-c56f-47c7-b767-3c554c8cb517 | test1.example.net | in-use | 20 | Attached to test1.example.net on /dev/sda | ---------------------------------------------------------------------------- ---- $ openstack volume delete vol_from_vol_from_ss $ openstack volume snapshot delete test1.example.net_ss1 $ openstack volume snapshot list | grep 'test1' $ openstack volume list | grep 'vol_\|test1' | cc698f20-c56f-47c7-b767-3c554c8cb517 | test1.example.net | in-use | 20 | Attached to test1.example.net on /dev/sda | -----Original Message----- From: Sean McGinnis <sean.mcginnis@gmx.com> Sent: Thursday, December 5, 2019 12:58 PM To: Ratliff, John <jdratlif@iu.edu> Cc: openstack-discuss@lists.openstack.org Subject: [External] Re: openstack volume snapshots and deleting This message was sent from a non-IU address. Please exercise caution when clicking links or opening attachments from external sources. ------- On Thu, Dec 05, 2019 at 03:24:35PM +0000, Ratliff, John wrote:
If I create a volume, snapshot it, and create a new volume from that snapshot, I cannot seem to delete the snapshot until the volume is deleted. Is this intentional, or am I doing something wrong? Is the new volume based on the snapshot, and that's why I can't delete it? We're using CEPH as a storage backend if that makes a difference.
That is the expected behavior as the snapshot is in-use by the volume created from it. So it can't be deleted until there isn't anything using it. Cloning would be the only way to get around this. There's not a quick and easy call to do this from a snapshot base, so unfortunately to get around that and get the final effect that it appears you are looking for, you would need to create a snapshot, create a volume from the snapshot, then *clone* yet another volume from that new one. Then delete the temporary volume and snapshot once the clone is done. As long as the backend driver is honoring the clone semantics, that should result in a fully independent volume that has its inital data set from the snapshot. So the process would look something like: openstack volume snapshot create --volume base-volume base-ss openstack volume create --snapshot base-ss temp-volume openstack volume create --volume temp-volume new-volume openstack volume delete temp-volume openstack volume snapshot delete base-ss Otherwise, it should be fairly safe to just keep the snapshot around. A lot of storage devices handle snapshots as internal pointers, so the only additional space consumption comes from anything new you end up writing to the new volume. Sean
Yes, I tried this second thing, but it doesn't work either. I created a new instance called test1.example.net. I created a volume snapshot (test1.example.net_ss1) I created a volume from that snapshot (vol_from_ss) I created a volume from the volume that was from the snapshot (vol_from_vol_from_ss) I deleted the vol_from_ss (this works) I deleted the snapshot (this does not work) If I then delete the vol_from_vol_from_ss volume, I can then delete the snapshot. P.S. My openstack cli says the option is --source, not --volume. Here is my command output. $ openstack volume list | grep test1 | cc698f20-c56f-47c7-b767-3c554c8cb517 | test1.example.net | in-use | 20 | Attached to test1.example.net on /dev/sda | $ openstack volume snapshot list | grep test1 $ openstack volume snapshot create --volume test1.example.net test1.example.net_ss1 Invalid volume: Volume cc698f20-c56f-47c7-b767-3c554c8cb517 status must be available, but current status is: in-use. (HTTP 400) (Request-ID: req-1ab60e86-3809-4a3b-9ac3-98e7d09ae7e9) $ openstack volume snapshot create --force --volume test1.example.net test1.example.net_ss1 $ openstack volume create --snapshot test1.example.net_ss1 vol_from_ss $ openstack volume create --volume vol_from_ss vol_from_vol_from_ss usage: openstack volume create [-h] [-f {json,shell,table,value,yaml}] [-c COLUMN] [--max-width <integer>] [--fit-width] [--print-empty] [--noindent] [--prefix PREFIX] [--size <size>] [--type <volume-type>] [--image <image> | --snapshot <snapshot> | --source <volume>] [--description <description>] [--availability-zone <availability-zone>] [--consistency-group consistency-group>] [--property <key=value>] [--hint <key=value>] [--multi-attach] [--bootable | --non-bootable] [--read-only | --read-write] <name> openstack volume create: error: unrecognized arguments: --volume vol_from_vol_from_ss $ openstack volume create --source vol_from_ss vol_from_vol_from_ss $ openstack volume delete vol_from_ss $ openstack volume snapshot delete test1.example.net_ss1 $ openstack volume snapshot list | grep test1 | 386186e9-6121-4222-b487-3dc026acf70d | test1.example.net_ss1 | None | available | 20 | $ openstack volume list | grep 'vol_\|test1' | ccbfdabe-d2ff-4bbd-a0ef-3b08935fe29a | vol_from_vol_from_ss | available | 20 | | | cc698f20-c56f-47c7-b767-3c554c8cb517 | test1.example.net | in-use | 20 | Attached to test1.example.net on /dev/sda | ---------------------------------------------------------------------------- ---- $ openstack volume delete vol_from_vol_from_ss $ openstack volume snapshot delete test1.example.net_ss1 $ openstack volume snapshot list | grep 'test1' $ openstack volume list | grep 'vol_\|test1' | cc698f20-c56f-47c7-b767-3c554c8cb517 | test1.example.net | in-use | 20 | Attached to test1.example.net on /dev/sda | --John -----Original Message----- From: Sean McGinnis <sean.mcginnis@gmx.com> Sent: Thursday, December 5, 2019 12:58 PM To: Ratliff, John <jdratlif@iu.edu> Cc: openstack-discuss@lists.openstack.org Subject: [External] Re: openstack volume snapshots and deleting This message was sent from a non-IU address. Please exercise caution when clicking links or opening attachments from external sources. ------- On Thu, Dec 05, 2019 at 03:24:35PM +0000, Ratliff, John wrote:
If I create a volume, snapshot it, and create a new volume from that snapshot, I cannot seem to delete the snapshot until the volume is deleted. Is this intentional, or am I doing something wrong? Is the new volume based on the snapshot, and that's why I can't delete it? We're using CEPH as a storage backend if that makes a difference.
That is the expected behavior as the snapshot is in-use by the volume created from it. So it can't be deleted until there isn't anything using it. Cloning would be the only way to get around this. There's not a quick and easy call to do this from a snapshot base, so unfortunately to get around that and get the final effect that it appears you are looking for, you would need to create a snapshot, create a volume from the snapshot, then *clone* yet another volume from that new one. Then delete the temporary volume and snapshot once the clone is done. As long as the backend driver is honoring the clone semantics, that should result in a fully independent volume that has its inital data set from the snapshot. So the process would look something like: openstack volume snapshot create --volume base-volume base-ss openstack volume create --snapshot base-ss temp-volume openstack volume create --volume temp-volume new-volume openstack volume delete temp-volume openstack volume snapshot delete base-ss Otherwise, it should be fairly safe to just keep the snapshot around. A lot of storage devices handle snapshots as internal pointers, so the only additional space consumption comes from anything new you end up writing to the new volume. Sean
On Thu, Dec 05, 2019 at 06:25:28PM +0000, Ratliff, John wrote:
Yes, I tried this second thing, but it doesn't work either.
I created a new instance called test1.example.net. I created a volume snapshot (test1.example.net_ss1) I created a volume from that snapshot (vol_from_ss) I created a volume from the volume that was from the snapshot (vol_from_vol_from_ss) I deleted the vol_from_ss (this works) I deleted the snapshot (this does not work)
Looking at the RBD driver, it looks like it does not actually follow the expected clone semantics. Unless rbd_max_clone is set to 0, or the number of clones is less than whatever that is set to, it will actually do a COW volume on top of the snapshot. This is a performance optimization (it doesn't have to copy all of the data into the new volume), but is actually a bug, IMO. You can try changing that config option and restarting services to pick that up, then trying again. Otherwise, it looks like that drivers clone handling will create its own snapshot of a volume. So unless you need to keep that snapshot around for some other reason, it looks like you should be able to just: openstack volume create --source orig-volume new-volume And if I follow it right, it will create a snapshot under the covers, then use that to create the new volume. Hopefully one of those helps. Sean
Creating a volume from a volume with ceph, at least in our configuration (which I don't have any authority to change) will not create full volume, but rather a COW volume. However, I can use rbd flatten on the image, and then delete the snapshot. It takes time to copy the image if they're large, which makes sense as to why it doesn't do this by default. But that is a reasonable workaround. Thanks. --John -----Original Message----- From: Sean McGinnis <sean.mcginnis@gmx.com> Sent: Thursday, December 5, 2019 1:59 PM To: Ratliff, John <jdratlif@iu.edu> Cc: openstack-discuss@lists.openstack.org Subject: Re: [External] Re: openstack volume snapshots and deleting On Thu, Dec 05, 2019 at 06:25:28PM +0000, Ratliff, John wrote:
Yes, I tried this second thing, but it doesn't work either.
I created a new instance called test1.example.net. I created a volume snapshot (test1.example.net_ss1) I created a volume from that snapshot (vol_from_ss) I created a volume from the volume that was from the snapshot (vol_from_vol_from_ss) I deleted the vol_from_ss (this works) I deleted the snapshot (this does not work)
Looking at the RBD driver, it looks like it does not actually follow the expected clone semantics. Unless rbd_max_clone is set to 0, or the number of clones is less than whatever that is set to, it will actually do a COW volume on top of the snapshot. This is a performance optimization (it doesn't have to copy all of the data into the new volume), but is actually a bug, IMO. You can try changing that config option and restarting services to pick that up, then trying again. Otherwise, it looks like that drivers clone handling will create its own snapshot of a volume. So unless you need to keep that snapshot around for some other reason, it looks like you should be able to just: openstack volume create --source orig-volume new-volume And if I follow it right, it will create a snapshot under the covers, then use that to create the new volume. Hopefully one of those helps. Sean
To avoid endless COW chains you could also set: rbd_flatten_volume_from_snapshot = True in your cinder.conf. Of course you have to be fine with all volumes created from snapshots to be flattened: depending on the size the volume creation process can require a sizable amount of time. Best, Mattia On 12/6/19 4:22 PM, Ratliff, John wrote:
Creating a volume from a volume with ceph, at least in our configuration (which I don't have any authority to change) will not create full volume, but rather a COW volume.
However, I can use rbd flatten on the image, and then delete the snapshot. It takes time to copy the image if they're large, which makes sense as to why it doesn't do this by default. But that is a reasonable workaround.
Thanks.
--John
-----Original Message----- From: Sean McGinnis <sean.mcginnis@gmx.com> Sent: Thursday, December 5, 2019 1:59 PM To: Ratliff, John <jdratlif@iu.edu> Cc: openstack-discuss@lists.openstack.org Subject: Re: [External] Re: openstack volume snapshots and deleting
On Thu, Dec 05, 2019 at 06:25:28PM +0000, Ratliff, John wrote:
Yes, I tried this second thing, but it doesn't work either.
I created a new instance called test1.example.net. I created a volume snapshot (test1.example.net_ss1) I created a volume from that snapshot (vol_from_ss) I created a volume from the volume that was from the snapshot (vol_from_vol_from_ss) I deleted the vol_from_ss (this works) I deleted the snapshot (this does not work)
Looking at the RBD driver, it looks like it does not actually follow the expected clone semantics. Unless rbd_max_clone is set to 0, or the number of clones is less than whatever that is set to, it will actually do a COW volume on top of the snapshot.
This is a performance optimization (it doesn't have to copy all of the data into the new volume), but is actually a bug, IMO.
You can try changing that config option and restarting services to pick that up, then trying again.
Otherwise, it looks like that drivers clone handling will create its own snapshot of a volume. So unless you need to keep that snapshot around for some other reason, it looks like you should be able to just:
openstack volume create --source orig-volume new-volume
And if I follow it right, it will create a snapshot under the covers, then use that to create the new volume.
Hopefully one of those helps.
Sean
-- Mattia Belluco S3IT Services and Support for Science IT Office Y11 F 52 University of Zürich Winterthurerstrasse 190, CH-8057 Zürich (Switzerland) Tel: +41 44 635 42 22
participants (3)
-
Mattia Belluco
-
Ratliff, John
-
Sean McGinnis