[Openstack] nova backup - instances unreachable
Eugen Block
eblock at nde.ag
Fri Jan 13 10:49:44 UTC 2017
Thanks,
for anyone interested in this issue, I filed a bug report:
https://bugs.launchpad.net/nova/+bug/1656242
Regards,
Eugen
Zitat von Mohammed Naser <mnaser at vexxhost.com>:
> It is likely because this has been tested with QEMU only. I think
> you might want to bring this up with the Nova team.
>
> Sent from my iPhone
>
>> On Jan 12, 2017, at 11:28 AM, Eugen Block <eblock at nde.ag> wrote:
>>
>> I'm not sure if this is the right spot, but I added some log
>> statements into driver.py.
>> First, there's this if-block:
>>
>> if (self._host.has_min_version(MIN_LIBVIRT_LIVESNAPSHOT_VERSION,
>> MIN_QEMU_LIVESNAPSHOT_VERSION,
>> host.HV_DRIVER_QEMU)
>> and source_type not in ('lvm')
>> and not CONF.ephemeral_storage_encryption.enabled
>> and not CONF.workarounds.disable_libvirt_livesnapshot):
>> live_snapshot = True
>> [...]
>> else:
>> live_snapshot = False
>>
>> And I know that it lands in the else-statement. Turns out that
>> _host.has_min_version is "false", because of host.HV_DRIVER_QEMU.
>> We are running on Xen hypervisors. So I tried it with
>> host.HV_DRIVER_XEN and now nova-compute says:
>>
>> [instance: 14b75237-7619-481f-9636-792b64d1be17] instance snapshotting
>> [instance: 14b75237-7619-481f-9636-792b64d1be17] Beginning live
>> snapshot process
>>
>> Now I'm waiting for the result, but at least the VM is still
>> running, so it looks quite promising...
>>
>> And there it is:
>>
>> [instance: 14b75237-7619-481f-9636-792b64d1be17] Snapshot image
>> upload complete
>>
>> I'm testing the image now, and it works!
>>
>> Now the question is, why is it defaulting to HV_DRIVER_QEMU and is
>> it really necessary to change this directly in the code? Is there
>> any other way?
>>
>> Regards,
>> Eugen
>>
>> Zitat von Eugen Block <eblock at nde.ag>:
>>
>>> Yes, I truncated the file and uploaded it:
>>>
>>> http://dropcanvas.com/ta7nu
>>> (First time I used this service, please give me feedback if this
>>> doesn't work for you)
>>>
>>> I see the "Beginning cold snapshot process" message, but I don't
>>> know why. Any help is appreciated!
>>>
>>> Regards,
>>> Eugen
>>>
>>>
>>> Zitat von Mohammed Naser <mnaser at vexxhost.com>:
>>>
>>>> Would you be able to share the logs of a full snapshot run with
>>>> the compute node in debug?
>>>>
>>>> Sent from my iPhone
>>>>
>>>>> On Jan 12, 2017, at 7:47 AM, Eugen Block <eblock at nde.ag> wrote:
>>>>>
>>>>> That's strange, I also searched for this message, but nothing
>>>>> there. I have debug logs enabled on compute node but I don't see
>>>>> anything regarding ceph. No matter, what I do, my instance is
>>>>> always shutdown before a snapshot is taken. What else can I try?
>>>>>
>>>>>
>>>>> Zitat von John Petrini <jpetrini at coredial.com>:
>>>>>
>>>>>> Mohammed,
>>>>>>
>>>>>> It looks like you may be right. Just found the permissions issue in the
>>>>>> nova log on the compute node.
>>>>>>
>>>>>> 4-e8f52e4fbcfb 691caf1c10354efab3e3c8ed61b7d89a
>>>>>> 49bc5e5bf2684bd0948d9f94c7875027 - - -] Performing standard snapshot
>>>>>> because direct snapshot failed: no write permission on storage
>>>>>> pool images
>>>>>>
>>>>>> I'm going to test the change and will send an update you all with the
>>>>>> results.
>>>>>>
>>>>>> Thank You,
>>>>>>
>>>>>> ___
>>>>>>
>>>>>> John Petrini
>>>>>>
>>>>>>
>>>>>>
>>>>>>>>
>>>>>>> Yes, we are also running Mitaka and I also read Sebastien
>>>>>>> Han's blogs ;-)
>>>>>>>
>>>>>>> our snapshots are not happening at the RBD level,
>>>>>>>> they are being copied and uploaded to glance which takes up a
>>>>>>>> lot of space
>>>>>>>> and is very slow.
>>>>>>>>
>>>>>>>
>>>>>>> Unfortunately, that's what we are experiencing, too. I don't know if
>>>>>>> there's something I missed in the nova configs or somewhere
>>>>>>> else, but I'm
>>>>>>> relieved that I'm not the only one :-)
>>>>>>>
>>>>>>> While writing this email I searched again and found something:
>>>>>>>
>>>>>>> https://specs.openstack.org/openstack/nova-specs/specs/mitak
>>>>>>> a/implemented/rbd-instance-snapshots.html
>>>>>>>
>>>>>>> https://review.openstack.org/#/c/205282/
>>>>>>>
>>>>>>> It seems to be implemented already, I'm looking for the config
>>>>>>> options to
>>>>>>> set. If you manage to get nova to make rbd snapshots, please
>>>>>>> let me know ;-)
>>>>>>>
>>>>>>> Regards,
>>>>>>> Eugen
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Zitat von John Petrini <jpetrini at coredial.com>:
>>>>>>>
>>>>>>> Hi Eugen,
>>>>>>>>
>>>>>>>> Thanks for the response! That makes a lost of sense and is
>>>>>>>> what I figured
>>>>>>>> was going on but I missed it in the documentation. We use
>>>>>>>> Ceph as well and
>>>>>>>> I had considered doing the snapshots at the RBD level but I was hoping
>>>>>>>> there was someway to accomplish this via nova. I came across this
>>>>>>>> Sebastien
>>>>>>>> Han write-up that claims this functionality was added to Mitaka:
>>>>>>>> http://www.sebastien-han.fr/blog/2015/10/05/openstack-nova-
>>>>>>>> snapshots-on-ceph-rbd/
>>>>>>>>
>>>>>>>> We are running Mitaka but our snapshots are not happening at the RBD
>>>>>>>> level,
>>>>>>>> they are being copied and uploaded to glance which takes up a
>>>>>>>> lot of space
>>>>>>>> and is very slow.
>>>>>>>>
>>>>>>>> Have you or anyone else implemented this in Mitaka? Other
>>>>>>>> than Sebastian's
>>>>>>>> blog I haven't found any documentation on this.
>>>>>>>>
>>>>>>>> Thank You,
>>>>>>>>
>>>>>>>> ___
>>>>>>>>
>>>>>>>> John Petrini
>>>>>>>>
>>>>>>>> On Wed, Jan 11, 2017 at 3:32 AM, Eugen Block <eblock at nde.ag> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> this seems to be exptected, the docs say:
>>>>>>>>>
>>>>>>>>> "Shut down the source VM before you take the snapshot to
>>>>>>>>> ensure that all
>>>>>>>>> data is flushed to disk."
>>>>>>>>>
>>>>>>>>> So if the VM is not shut down, it's freezed to prevent data loss (I
>>>>>>>>> guess). Depending on your storage backend, there are other ways to
>>>>>>>>> perform
>>>>>>>>> backups of your VMs.
>>>>>>>>> We use Ceph as backend for nova, glance and cinder. Ceph stores the
>>>>>>>>> disks,
>>>>>>>>> images and volumes as Rados block device objects. We have a
>>>>>>>>> backup script
>>>>>>>>> that creates snapshots of these RBDs, which are exported to
>>>>>>>>> our backup
>>>>>>>>> drive. This way the running VM is not stopped or freezed, the user
>>>>>>>>> doesn't
>>>>>>>>> notice any issues. Unlike a nova snapshot, the rbd snapshot
>>>>>>>>> is created
>>>>>>>>> immediately within a few seconds. After a successful backup
>>>>>>>>> the snapshots
>>>>>>>>> are removed.
>>>>>>>>>
>>>>>>>>> Hope this helps! If you are interested in Ceph, visit [1].
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Eugen
>>>>>>>>>
>>>>>>>>> [1] http://docs.ceph.com/docs/giant/start/intro/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Zitat von John Petrini <jpetrini at coredial.com>:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I've just started experimenting with nova backup and discovered that
>>>>>>>>>> there
>>>>>>>>>> is a period of time during the snapshot where the instance becomes
>>>>>>>>>> unreachable. Is this behavior expected during a live
>>>>>>>>>> snapshot? Is there
>>>>>>>>>> any
>>>>>>>>>> way to prevent this?
>>>>>>>>>>
>>>>>>>>>> ___
>>>>>>>>>>
>>>>>>>>>> John Petrini
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Eugen Block voice : +49-40-559 51 75
>>>>>>>>> NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77
>>>>>>>>> Postfach 61 03 15
>>>>>>>>> D-22423 Hamburg e-mail : eblock at nde.ag
>>>>>>>>>
>>>>>>>>> Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>>>>>>>> Sitz und Registergericht: Hamburg, HRB 90934
>>>>>>>>> Vorstand: Jens-U. Mozdzen
>>>>>>>>> USt-IdNr. DE 814 013 983
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Mailing list: http://lists.openstack.org/cgi
>>>>>>>>> -bin/mailman/listinfo/openstac
>>>>>>>>> k
>>>>>>>>> Post to : openstack at lists.openstack.org
>>>>>>>>> Unsubscribe : http://lists.openstack.org/cgi
>>>>>>>>> -bin/mailman/listinfo/openstac
>>>>>>>>> k
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Eugen Block voice : +49-40-559 51 75
>>>>>>> NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77
>>>>>>> Postfach 61 03 15
>>>>>>> D-22423 Hamburg e-mail : eblock at nde.ag
>>>>>>>
>>>>>>> Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>>>>>> Sitz und Registergericht: Hamburg, HRB 90934
>>>>>>> Vorstand: Jens-U. Mozdzen
>>>>>>> USt-IdNr. DE 814 013 983
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Eugen Block voice : +49-40-559 51 75
>>>>> NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77
>>>>> Postfach 61 03 15
>>>>> D-22423 Hamburg e-mail : eblock at nde.ag
>>>>>
>>>>> Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>>>> Sitz und Registergericht: Hamburg, HRB 90934
>>>>> Vorstand: Jens-U. Mozdzen
>>>>> USt-IdNr. DE 814 013 983
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list:
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>> Post to : openstack at lists.openstack.org
>>>>> Unsubscribe :
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>
>>>
>>>
>>> --
>>> Eugen Block voice : +49-40-559 51 75
>>> NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77
>>> Postfach 61 03 15
>>> D-22423 Hamburg e-mail : eblock at nde.ag
>>>
>>> Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>> Sitz und Registergericht: Hamburg, HRB 90934
>>> Vorstand: Jens-U. Mozdzen
>>> USt-IdNr. DE 814 013 983
>>
>>
>>
>> --
>> Eugen Block voice : +49-40-559 51 75
>> NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77
>> Postfach 61 03 15
>> D-22423 Hamburg e-mail : eblock at nde.ag
>>
>> Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>> Sitz und Registergericht: Hamburg, HRB 90934
>> Vorstand: Jens-U. Mozdzen
>> USt-IdNr. DE 814 013 983
>>
--
Eugen Block voice : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail : eblock at nde.ag
Vorsitzende des Aufsichtsrates: Angelika Mozdzen
Sitz und Registergericht: Hamburg, HRB 90934
Vorstand: Jens-U. Mozdzen
USt-IdNr. DE 814 013 983
More information about the Openstack
mailing list