[cinder][nova] fsfreeze hooks issues with cinder snapshot/backup
Hello, We are trying to trigger an fsfreeze via a cinder backup or snapshot. We confirmed that the fsfreeze hooks are actually called with a nova snapshot with `/var/log/qga-fsfreeze-hook.log` in the VM, but we can't achieve the same thing with a cinder backup/snapshot attached to the same instance. We are using Wallaby, libvirt, RBD for cinder volumes and RBD as well for cinder-backup. According to this (old) spec [0], cinder should call the `quiesce()` method in nova during backup/snapshot. We looked in the cinder code and couldn't find any clear evidence that this method is actually called by cinder (but we may have missed something). We added some debug messages on quiesce/can_quiesce/require_quiesce/... in `nova/virt/libvirt/driver.py` and they are never called with a cinder backup/snapshot in our setup while they are (and succeed) if we do a nova snapshot. We are starting to suspect that something is missing in cinder, but it could very well be a problem with our setup as well... Does someone use this feature or know if it should be working/implemented? [0]: https://wiki.openstack.org/wiki/Cinder/QuiescedSnapshotWithQemuGuestAgent#Ci... Cheers, -- Arthur Outhenin-Chalandre
Hello, As far as I can see cinder hasn't implemented this. However, I'm not sure about the status of this feature because the last update was on 2014[1] I think it's important to mention that this would only affect the live snapshots (handled by Nova) but for any other scenario every cinder driver optimized the snapshot/backup creation in a different way. This sounds like a good PTG discussion topic. You can add it to the planning etherpad here: https://etherpad.opendev.org/p/yoga-ptg-cinder-planning There's also info about the dates and times we'll be meeting on that etherpad. Cheers, Sofia [1] https://blueprints.launchpad.net/cinder/+spec/quiesced-snapshots-with-qemu-g... On Tue, Aug 31, 2021 at 11:52 AM Arthur Outhenin-Chalandre < arthur.outhenin-chalandre@cern.ch> wrote:
Hello,
We are trying to trigger an fsfreeze via a cinder backup or snapshot. We confirmed that the fsfreeze hooks are actually called with a nova snapshot with `/var/log/qga-fsfreeze-hook.log` in the VM, but we can't achieve the same thing with a cinder backup/snapshot attached to the same instance. We are using Wallaby, libvirt, RBD for cinder volumes and RBD as well for cinder-backup.
According to this (old) spec [0], cinder should call the `quiesce()` method in nova during backup/snapshot. We looked in the cinder code and couldn't find any clear evidence that this method is actually called by cinder (but we may have missed something). We added some debug messages on quiesce/can_quiesce/require_quiesce/... in `nova/virt/libvirt/driver.py` and they are never called with a cinder backup/snapshot in our setup while they are (and succeed) if we do a nova snapshot.
We are starting to suspect that something is missing in cinder, but it could very well be a problem with our setup as well... Does someone use this feature or know if it should be working/implemented?
[0]:
https://wiki.openstack.org/wiki/Cinder/QuiescedSnapshotWithQemuGuestAgent#Ci...
Cheers,
-- Arthur Outhenin-Chalandre
-- L. Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>
Hello, On 8/31/21 5:19 PM, Sofia Enriquez wrote:
As far as I can see cinder hasn't implemented this. However, I'm not sure about the status of this feature because the last update was on 2014[1]
Oh ok, I was hoping it was a setup problem on our side. Thanks for letting me know, it has saved me from further testing/research!
This sounds like a good PTG discussion topic. You can add it to the planning etherpad here: https://etherpad.opendev.org/p/yoga-ptg-cinder-planning <https://etherpad.opendev.org/p/yoga-ptg-cinder-planning>
There's also info about the dates and times we'll be meeting on that etherpad.
I added the discussion topic and I will be around to discuss it. I should be able to participate in the dev if needed as well. Cheers, -- Arthur Outhenin-Chalandre
On 31-08-21 12:19:54, Sofia Enriquez wrote:
Hello,
As far as I can see cinder hasn't implemented this. However, I'm not sure about the status of this feature because the last update was on 2014[1]
Oddly enough imageCreate [1] in Nova does orchestrate this when creating image based snapshots of volume backed instances [2] when called directly. Nova doesn't provide an external API to quiesce an instance so outside of the imageCreate and os-assisted-volume-snapshots [3][4] (only used by the NFS c-vol backend) flows it isn't going to happen. [1] https://docs.openstack.org/api-ref/compute/?expanded=create-image-createimag... [2] https://github.com/openstack/nova/blob/e81211318a29c5f07ee2b9c753dbab041ee01... [3] https://docs.openstack.org/api-ref/compute/#assisted-volume-snapshots-os-ass... [4] https://github.com/openstack/nova/blob/e81211318a29c5f07ee2b9c753dbab041ee01...
I think it's important to mention that this would only affect the live snapshots (handled by Nova) but for any other scenario every cinder driver optimized the snapshot/backup creation in a different way.
Well to be clear this specific problem is with Cinder orchestrated live attached volume snapshots of non-NFS backends.
This sounds like a good PTG discussion topic. You can add it to the planning etherpad here: https://etherpad.opendev.org/p/yoga-ptg-cinder-planning
I'll add the above to the pad and will try to attend as this will require work on the Nova side to provide an external API or preferably an os-server-external-events event to quiesce the instance. Cheers, Lee
On Tue, Aug 31, 2021 at 11:52 AM Arthur Outhenin-Chalandre < arthur.outhenin-chalandre@cern.ch> wrote:
Hello,
We are trying to trigger an fsfreeze via a cinder backup or snapshot. We confirmed that the fsfreeze hooks are actually called with a nova snapshot with `/var/log/qga-fsfreeze-hook.log` in the VM, but we can't achieve the same thing with a cinder backup/snapshot attached to the same instance. We are using Wallaby, libvirt, RBD for cinder volumes and RBD as well for cinder-backup.
According to this (old) spec [0], cinder should call the `quiesce()` method in nova during backup/snapshot. We looked in the cinder code and couldn't find any clear evidence that this method is actually called by cinder (but we may have missed something). We added some debug messages on quiesce/can_quiesce/require_quiesce/... in `nova/virt/libvirt/driver.py` and they are never called with a cinder backup/snapshot in our setup while they are (and succeed) if we do a nova snapshot.
We are starting to suspect that something is missing in cinder, but it could very well be a problem with our setup as well... Does someone use this feature or know if it should be working/implemented?
[0]:
https://wiki.openstack.org/wiki/Cinder/QuiescedSnapshotWithQemuGuestAgent#Ci...
Cheers,
-- Arthur Outhenin-Chalandre
--
L. Sofía Enriquez
she/her
Software Engineer
Red Hat PnT <https://www.redhat.com>
IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>
-- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
On 8/31/2021 10:19 AM, Sofia Enriquez wrote:
Hello,
As far as I can see cinder hasn't implemented this. However, I'm not sure about the status of this feature because the last update was on 2014[1]
I remember being a part of these discussions and the fact that there was interest in getting it working. I am wondering if this was just a case where no one followed through on implementation. It might be that we encountered an unexpected challenge that I don't remember. Either way, I do agree that this would be a good topic for the PTG.
I think it's important to mention that this would only affect the live snapshots (handled by Nova) but for any other scenario every cinder driver optimized the snapshot/backup creation in a different way.
This sounds like a good PTG discussion topic. You can add it to the planning etherpad here: https://etherpad.opendev.org/p/yoga-ptg-cinder-planning <https://etherpad.opendev.org/p/yoga-ptg-cinder-planning>
There's also info about the dates and times we'll be meeting on that etherpad.
Cheers, Sofia
[1] https://blueprints.launchpad.net/cinder/+spec/quiesced-snapshots-with-qemu-g... <https://blueprints.launchpad.net/cinder/+spec/quiesced-snapshots-with-qemu-guest-agent>
On Tue, Aug 31, 2021 at 11:52 AM Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch <mailto:arthur.outhenin-chalandre@cern.ch>> wrote:
Hello,
We are trying to trigger an fsfreeze via a cinder backup or snapshot. We confirmed that the fsfreeze hooks are actually called with a nova snapshot with `/var/log/qga-fsfreeze-hook.log` in the VM, but we can't achieve the same thing with a cinder backup/snapshot attached to the same instance. We are using Wallaby, libvirt, RBD for cinder volumes and RBD as well for cinder-backup.
According to this (old) spec [0], cinder should call the `quiesce()` method in nova during backup/snapshot. We looked in the cinder code and couldn't find any clear evidence that this method is actually called by cinder (but we may have missed something). We added some debug messages on quiesce/can_quiesce/require_quiesce/... in `nova/virt/libvirt/driver.py` and they are never called with a cinder backup/snapshot in our setup while they are (and succeed) if we do a nova snapshot.
We are starting to suspect that something is missing in cinder, but it could very well be a problem with our setup as well... Does someone use this feature or know if it should be working/implemented?
[0]: https://wiki.openstack.org/wiki/Cinder/QuiescedSnapshotWithQemuGuestAgent#Ci... <https://wiki.openstack.org/wiki/Cinder/QuiescedSnapshotWithQemuGuestAgent#Cinder>
Cheers,
-- Arthur Outhenin-Chalandre
--
L. Sofía Enriquez
she/her
Software Engineer
Red Hat PnT <https://www.redhat.com>
IRC: @enriquetaso
@RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>
participants (4)
-
Arthur Outhenin-Chalandre
-
Jay Bryant
-
Lee Yarwood
-
Sofia Enriquez