[Openstack] nova backup - instances unreachable
Eugen Block
eblock at nde.ag
Mon Jan 23 08:16:41 UTC 2017
Have you enabled live snapshots in nova.conf?
The default for this option is "true", so you should check that:
disable_libvirt_livesnapshot = false
Is it really a live snaphot? What's in the nova-compute.log? It should
say something like
[instance: XXX] Beginning live snapshot process
Regards,
Eugen
Zitat von John Petrini <jpetrini at coredial.com>:
> Hi All,
>
> Following up after making this change. Adding write permissions to the
> images pool in Ceph did the trick and RBD snapshots now work. However the
> instance is still paused for the duration of the snapshot. Is it possible
> to do a live snapshot without pausing the instance?
>
> Thanks,
>
> John
>
> On Fri, Jan 13, 2017 at 5:49 AM, Eugen Block <eblock at nde.ag> wrote:
>
>> Thanks,
>>
>> for anyone interested in this issue, I filed a bug report:
>> https://bugs.launchpad.net/nova/+bug/1656242
>>
>>
>> Regards,
>> Eugen
>>
>>
>> Zitat von Mohammed Naser <mnaser at vexxhost.com>:
>>
>> It is likely because this has been tested with QEMU only. I think you
>>> might want to bring this up with the Nova team.
>>>
>>> Sent from my iPhone
>>>
>>> On Jan 12, 2017, at 11:28 AM, Eugen Block <eblock at nde.ag> wrote:
>>>>
>>>> I'm not sure if this is the right spot, but I added some log statements
>>>> into driver.py.
>>>> First, there's this if-block:
>>>>
>>>> if (self._host.has_min_version(MIN_LIBVIRT_LIVESNAPSHOT_VERSION,
>>>> MIN_QEMU_LIVESNAPSHOT_VERSION,
>>>> host.HV_DRIVER_QEMU)
>>>> and source_type not in ('lvm')
>>>> and not CONF.ephemeral_storage_encryption.enabled
>>>> and not CONF.workarounds.disable_libvirt_livesnapshot):
>>>> live_snapshot = True
>>>> [...]
>>>> else:
>>>> live_snapshot = False
>>>>
>>>> And I know that it lands in the else-statement. Turns out that
>>>> _host.has_min_version is "false", because of host.HV_DRIVER_QEMU. We are
>>>> running on Xen hypervisors. So I tried it with host.HV_DRIVER_XEN and now
>>>> nova-compute says:
>>>>
>>>> [instance: 14b75237-7619-481f-9636-792b64d1be17] instance snapshotting
>>>> [instance: 14b75237-7619-481f-9636-792b64d1be17] Beginning live
>>>> snapshot process
>>>>
>>>> Now I'm waiting for the result, but at least the VM is still running, so
>>>> it looks quite promising...
>>>>
>>>> And there it is:
>>>>
>>>> [instance: 14b75237-7619-481f-9636-792b64d1be17] Snapshot image upload
>>>> complete
>>>>
>>>> I'm testing the image now, and it works!
>>>>
>>>> Now the question is, why is it defaulting to HV_DRIVER_QEMU and is it
>>>> really necessary to change this directly in the code? Is there any other
>>>> way?
>>>>
>>>> Regards,
>>>> Eugen
>>>>
>>>> Zitat von Eugen Block <eblock at nde.ag>:
>>>>
>>>> Yes, I truncated the file and uploaded it:
>>>>>
>>>>> http://dropcanvas.com/ta7nu
>>>>> (First time I used this service, please give me feedback if this
>>>>> doesn't work for you)
>>>>>
>>>>> I see the "Beginning cold snapshot process" message, but I don't know
>>>>> why. Any help is appreciated!
>>>>>
>>>>> Regards,
>>>>> Eugen
>>>>>
>>>>>
>>>>> Zitat von Mohammed Naser <mnaser at vexxhost.com>:
>>>>>
>>>>> Would you be able to share the logs of a full snapshot run with the
>>>>>> compute node in debug?
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>> On Jan 12, 2017, at 7:47 AM, Eugen Block <eblock at nde.ag> wrote:
>>>>>>>
>>>>>>> That's strange, I also searched for this message, but nothing there.
>>>>>>> I have debug logs enabled on compute node but I don't see anything
>>>>>>> regarding ceph. No matter, what I do, my instance is always
>>>>>>> shutdown before
>>>>>>> a snapshot is taken. What else can I try?
>>>>>>>
>>>>>>>
>>>>>>> Zitat von John Petrini <jpetrini at coredial.com>:
>>>>>>>
>>>>>>> Mohammed,
>>>>>>>>
>>>>>>>> It looks like you may be right. Just found the permissions issue in
>>>>>>>> the
>>>>>>>> nova log on the compute node.
>>>>>>>>
>>>>>>>> 4-e8f52e4fbcfb 691caf1c10354efab3e3c8ed61b7d89a
>>>>>>>> 49bc5e5bf2684bd0948d9f94c7875027 - - -] Performing standard snapshot
>>>>>>>> because direct snapshot failed: no write permission on storage pool
>>>>>>>> images
>>>>>>>>
>>>>>>>> I'm going to test the change and will send an update you all with the
>>>>>>>> results.
>>>>>>>>
>>>>>>>> Thank You,
>>>>>>>>
>>>>>>>> ___
>>>>>>>>
>>>>>>>> John Petrini
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>> Yes, we are also running Mitaka and I also read Sebastien Han's
>>>>>>>>> blogs ;-)
>>>>>>>>>
>>>>>>>>> our snapshots are not happening at the RBD level,
>>>>>>>>>
>>>>>>>>>> they are being copied and uploaded to glance which takes up a lot
>>>>>>>>>> of space
>>>>>>>>>> and is very slow.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Unfortunately, that's what we are experiencing, too. I don't know if
>>>>>>>>> there's something I missed in the nova configs or somewhere else,
>>>>>>>>> but I'm
>>>>>>>>> relieved that I'm not the only one :-)
>>>>>>>>>
>>>>>>>>> While writing this email I searched again and found something:
>>>>>>>>>
>>>>>>>>> https://specs.openstack.org/openstack/nova-specs/specs/mitak
>>>>>>>>> a/implemented/rbd-instance-snapshots.html
>>>>>>>>>
>>>>>>>>> https://review.openstack.org/#/c/205282/
>>>>>>>>>
>>>>>>>>> It seems to be implemented already, I'm looking for the config
>>>>>>>>> options to
>>>>>>>>> set. If you manage to get nova to make rbd snapshots, please let me
>>>>>>>>> know ;-)
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Eugen
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Zitat von John Petrini <jpetrini at coredial.com>:
>>>>>>>>>
>>>>>>>>> Hi Eugen,
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks for the response! That makes a lost of sense and is what I
>>>>>>>>>> figured
>>>>>>>>>> was going on but I missed it in the documentation. We use Ceph as
>>>>>>>>>> well and
>>>>>>>>>> I had considered doing the snapshots at the RBD level but I was
>>>>>>>>>> hoping
>>>>>>>>>> there was someway to accomplish this via nova. I came across this
>>>>>>>>>> Sebastien
>>>>>>>>>> Han write-up that claims this functionality was added to Mitaka:
>>>>>>>>>> http://www.sebastien-han.fr/blog/2015/10/05/openstack-nova-
>>>>>>>>>> snapshots-on-ceph-rbd/
>>>>>>>>>>
>>>>>>>>>> We are running Mitaka but our snapshots are not happening at the
>>>>>>>>>> RBD
>>>>>>>>>> level,
>>>>>>>>>> they are being copied and uploaded to glance which takes up a lot
>>>>>>>>>> of space
>>>>>>>>>> and is very slow.
>>>>>>>>>>
>>>>>>>>>> Have you or anyone else implemented this in Mitaka? Other than
>>>>>>>>>> Sebastian's
>>>>>>>>>> blog I haven't found any documentation on this.
>>>>>>>>>>
>>>>>>>>>> Thank You,
>>>>>>>>>>
>>>>>>>>>> ___
>>>>>>>>>>
>>>>>>>>>> John Petrini
>>>>>>>>>>
>>>>>>>>>> On Wed, Jan 11, 2017 at 3:32 AM, Eugen Block <eblock at nde.ag>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> this seems to be exptected, the docs say:
>>>>>>>>>>>
>>>>>>>>>>> "Shut down the source VM before you take the snapshot to ensure
>>>>>>>>>>> that all
>>>>>>>>>>> data is flushed to disk."
>>>>>>>>>>>
>>>>>>>>>>> So if the VM is not shut down, it's freezed to prevent data loss
>>>>>>>>>>> (I
>>>>>>>>>>> guess). Depending on your storage backend, there are other ways to
>>>>>>>>>>> perform
>>>>>>>>>>> backups of your VMs.
>>>>>>>>>>> We use Ceph as backend for nova, glance and cinder. Ceph stores
>>>>>>>>>>> the
>>>>>>>>>>> disks,
>>>>>>>>>>> images and volumes as Rados block device objects. We have a
>>>>>>>>>>> backup script
>>>>>>>>>>> that creates snapshots of these RBDs, which are exported to our
>>>>>>>>>>> backup
>>>>>>>>>>> drive. This way the running VM is not stopped or freezed, the user
>>>>>>>>>>> doesn't
>>>>>>>>>>> notice any issues. Unlike a nova snapshot, the rbd snapshot is
>>>>>>>>>>> created
>>>>>>>>>>> immediately within a few seconds. After a successful backup the
>>>>>>>>>>> snapshots
>>>>>>>>>>> are removed.
>>>>>>>>>>>
>>>>>>>>>>> Hope this helps! If you are interested in Ceph, visit [1].
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Eugen
>>>>>>>>>>>
>>>>>>>>>>> [1] http://docs.ceph.com/docs/giant/start/intro/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Zitat von John Petrini <jpetrini at coredial.com>:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I've just started experimenting with nova backup and discovered
>>>>>>>>>>>> that
>>>>>>>>>>>> there
>>>>>>>>>>>> is a period of time during the snapshot where the instance
>>>>>>>>>>>> becomes
>>>>>>>>>>>> unreachable. Is this behavior expected during a live snapshot?
>>>>>>>>>>>> Is there
>>>>>>>>>>>> any
>>>>>>>>>>>> way to prevent this?
>>>>>>>>>>>>
>>>>>>>>>>>> ___
>>>>>>>>>>>>
>>>>>>>>>>>> John Petrini
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Eugen Block voice : +49-40-559 51
>>>>>>>>>>> 75
>>>>>>>>>>> NDE Netzdesign und -entwicklung AG fax : +49-40-559 51
>>>>>>>>>>> 77
>>>>>>>>>>> Postfach 61 03 15
>>>>>>>>>>> D-22423 Hamburg e-mail : eblock at nde.ag
>>>>>>>>>>>
>>>>>>>>>>> Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>>>>>>>>>> Sitz und Registergericht: Hamburg, HRB 90934
>>>>>>>>>>> Vorstand: Jens-U. Mozdzen
>>>>>>>>>>> USt-IdNr. DE 814 013 983
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Mailing list: http://lists.openstack.org/cgi
>>>>>>>>>>> -bin/mailman/listinfo/openstac
>>>>>>>>>>> k
>>>>>>>>>>> Post to : openstack at lists.openstack.org
>>>>>>>>>>> Unsubscribe : http://lists.openstack.org/cgi
>>>>>>>>>>> -bin/mailman/listinfo/openstac
>>>>>>>>>>> k
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Eugen Block voice : +49-40-559 51 75
>>>>>>>>> NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77
>>>>>>>>> Postfach 61 03 15
>>>>>>>>> D-22423 Hamburg e-mail : eblock at nde.ag
>>>>>>>>>
>>>>>>>>> Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>>>>>>>> Sitz und Registergericht: Hamburg, HRB 90934
>>>>>>>>> Vorstand: Jens-U. Mozdzen
>>>>>>>>> USt-IdNr. DE 814 013 983
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Eugen Block voice : +49-40-559 51 75
>>>>>>> NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77
>>>>>>> Postfach 61 03 15
>>>>>>> D-22423 Hamburg e-mail : eblock at nde.ag
>>>>>>>
>>>>>>> Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>>>>>> Sitz und Registergericht: Hamburg, HRB 90934
>>>>>>> Vorstand: Jens-U. Mozdzen
>>>>>>> USt-IdNr. DE 814 013 983
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Mailing list: http://lists.openstack.org/cgi
>>>>>>> -bin/mailman/listinfo/openstack
>>>>>>> Post to : openstack at lists.openstack.org
>>>>>>> Unsubscribe : http://lists.openstack.org/cgi
>>>>>>> -bin/mailman/listinfo/openstack
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Eugen Block voice : +49-40-559 51 75
>>>>> NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77
>>>>> Postfach 61 03 15
>>>>> D-22423 Hamburg e-mail : eblock at nde.ag
>>>>>
>>>>> Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>>>> Sitz und Registergericht: Hamburg, HRB 90934
>>>>> Vorstand: Jens-U. Mozdzen
>>>>> USt-IdNr. DE 814 013 983
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Eugen Block voice : +49-40-559 51 75
>>>> NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77
>>>> Postfach 61 03 15
>>>> D-22423 Hamburg e-mail : eblock at nde.ag
>>>>
>>>> Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>>> Sitz und Registergericht: Hamburg, HRB 90934
>>>> Vorstand: Jens-U. Mozdzen
>>>> USt-IdNr. DE 814 013 983
>>>>
>>>>
>>
>>
>> --
>> Eugen Block voice : +49-40-559 51 75
>> NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77
>> Postfach 61 03 15
>> D-22423 Hamburg e-mail : eblock at nde.ag
>>
>> Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>> Sitz und Registergericht: Hamburg, HRB 90934
>> Vorstand: Jens-U. Mozdzen
>> USt-IdNr. DE 814 013 983
>>
>>
--
Eugen Block voice : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail : eblock at nde.ag
Vorsitzende des Aufsichtsrates: Angelika Mozdzen
Sitz und Registergericht: Hamburg, HRB 90934
Vorstand: Jens-U. Mozdzen
USt-IdNr. DE 814 013 983
More information about the Openstack
mailing list