Open Stack

It is likely because this has been tested with QEMU only. I think you might want to bring this up with the Nova team. 

Sent from my iPhone

> On Jan 12, 2017, at 11:28 AM, Eugen Block <eblock at nde.ag> wrote:
> 
> I'm not sure if this is the right spot, but I added some log statements into driver.py.
> First, there's this if-block:
> 
>        if (self._host.has_min_version(MIN_LIBVIRT_LIVESNAPSHOT_VERSION,
>                                       MIN_QEMU_LIVESNAPSHOT_VERSION,
>                                       host.HV_DRIVER_QEMU)
>             and source_type not in ('lvm')
>             and not CONF.ephemeral_storage_encryption.enabled
>             and not CONF.workarounds.disable_libvirt_livesnapshot):
>            live_snapshot = True
>       [...]
>        else:
>            live_snapshot = False
> 
> And I know that it lands in the else-statement. Turns out that _host.has_min_version is "false", because of host.HV_DRIVER_QEMU. We are running on Xen hypervisors. So I tried it with host.HV_DRIVER_XEN and now nova-compute says:
> 
> [instance: 14b75237-7619-481f-9636-792b64d1be17] instance snapshotting
> [instance: 14b75237-7619-481f-9636-792b64d1be17] Beginning live snapshot process
> 
> Now I'm waiting for the result, but at least the VM is still running, so it looks quite promising...
> 
> And there it is:
> 
> [instance: 14b75237-7619-481f-9636-792b64d1be17] Snapshot image upload complete
> 
> I'm testing the image now, and it works!
> 
> Now the question is, why is it defaulting to HV_DRIVER_QEMU and is it really necessary to change this directly in the code? Is there any other way?
> 
> Regards,
> Eugen
> 
> Zitat von Eugen Block <eblock at nde.ag>:
> 
>> Yes, I truncated the file and uploaded it:
>> 
>> http://dropcanvas.com/ta7nu
>> (First time I used this service, please give me feedback if this doesn't work for you)
>> 
>> I see the "Beginning cold snapshot process" message, but I don't know why. Any help is appreciated!
>> 
>> Regards,
>> Eugen
>> 
>> 
>> Zitat von Mohammed Naser <mnaser at vexxhost.com>:
>> 
>>> Would you be able to share the logs of a full snapshot run with the compute node in debug?
>>> 
>>> Sent from my iPhone
>>> 
>>>> On Jan 12, 2017, at 7:47 AM, Eugen Block <eblock at nde.ag> wrote:
>>>> 
>>>> That's strange, I also searched for this message, but nothing there. I have debug logs enabled on compute node but I don't see anything regarding ceph. No matter, what I do, my instance is always shutdown before a snapshot is taken. What else can I try?
>>>> 
>>>> 
>>>> Zitat von John Petrini <jpetrini at coredial.com>:
>>>> 
>>>>> Mohammed,
>>>>> 
>>>>> It looks like you may be right. Just found the permissions issue in the
>>>>> nova log on the compute node.
>>>>> 
>>>>> 4-e8f52e4fbcfb 691caf1c10354efab3e3c8ed61b7d89a
>>>>> 49bc5e5bf2684bd0948d9f94c7875027 - - -] Performing standard snapshot
>>>>> because direct snapshot failed: no write permission on storage pool images
>>>>> 
>>>>> I'm going to test the change and will send an update you all with the
>>>>> results.
>>>>> 
>>>>> Thank You,
>>>>> 
>>>>> ___
>>>>> 
>>>>> John Petrini
>>>>> 
>>>>> 
>>>>> 
>>>>>>> 
>>>>>> Yes, we are also running Mitaka and I also read Sebastien Han's blogs ;-)
>>>>>> 
>>>>>> our snapshots are not happening at the RBD level,
>>>>>>> they are being copied and uploaded to glance which takes up a lot of space
>>>>>>> and is very slow.
>>>>>>> 
>>>>>> 
>>>>>> Unfortunately, that's what we are experiencing, too. I don't know if
>>>>>> there's something I missed in the nova configs or somewhere else, but I'm
>>>>>> relieved that I'm not the only one :-)
>>>>>> 
>>>>>> While writing this email I searched again and found something:
>>>>>> 
>>>>>> https://specs.openstack.org/openstack/nova-specs/specs/mitak
>>>>>> a/implemented/rbd-instance-snapshots.html
>>>>>> 
>>>>>> https://review.openstack.org/#/c/205282/
>>>>>> 
>>>>>> It seems to be implemented already, I'm looking for the config options to
>>>>>> set. If you manage to get nova to make rbd snapshots, please let me know ;-)
>>>>>> 
>>>>>> Regards,
>>>>>> Eugen
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Zitat von John Petrini <jpetrini at coredial.com>:
>>>>>> 
>>>>>> Hi Eugen,
>>>>>>> 
>>>>>>> Thanks for the response! That makes a lost of sense and is what I figured
>>>>>>> was going on but I missed it in the documentation. We use Ceph as well and
>>>>>>> I had considered doing the snapshots at the RBD level but I was hoping
>>>>>>> there was someway to accomplish this via nova. I came across this
>>>>>>> Sebastien
>>>>>>> Han write-up that claims this functionality was added to Mitaka:
>>>>>>> http://www.sebastien-han.fr/blog/2015/10/05/openstack-nova-
>>>>>>> snapshots-on-ceph-rbd/
>>>>>>> 
>>>>>>> We are running Mitaka but our snapshots are not happening at the RBD
>>>>>>> level,
>>>>>>> they are being copied and uploaded to glance which takes up a lot of space
>>>>>>> and is very slow.
>>>>>>> 
>>>>>>> Have you or anyone else implemented this in Mitaka? Other than Sebastian's
>>>>>>> blog I haven't found any documentation on this.
>>>>>>> 
>>>>>>> Thank You,
>>>>>>> 
>>>>>>> ___
>>>>>>> 
>>>>>>> John Petrini
>>>>>>> 
>>>>>>> On Wed, Jan 11, 2017 at 3:32 AM, Eugen Block <eblock at nde.ag> wrote:
>>>>>>> 
>>>>>>> Hi,
>>>>>>>> 
>>>>>>>> this seems to be exptected, the docs say:
>>>>>>>> 
>>>>>>>> "Shut down the source VM before you take the snapshot to ensure that all
>>>>>>>> data is flushed to disk."
>>>>>>>> 
>>>>>>>> So if the VM is not shut down, it's freezed to prevent data loss (I
>>>>>>>> guess). Depending on your storage backend, there are other ways to
>>>>>>>> perform
>>>>>>>> backups of your VMs.
>>>>>>>> We use Ceph as backend for nova, glance and cinder. Ceph stores the
>>>>>>>> disks,
>>>>>>>> images and volumes as Rados block device objects. We have a backup script
>>>>>>>> that creates snapshots of these RBDs, which are exported to our backup
>>>>>>>> drive. This way the running VM is not stopped or freezed, the user
>>>>>>>> doesn't
>>>>>>>> notice any issues. Unlike a nova snapshot, the rbd snapshot is created
>>>>>>>> immediately within a few seconds. After a successful backup the snapshots
>>>>>>>> are removed.
>>>>>>>> 
>>>>>>>> Hope this helps! If you are interested in Ceph, visit [1].
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> Eugen
>>>>>>>> 
>>>>>>>> [1] http://docs.ceph.com/docs/giant/start/intro/
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Zitat von John Petrini <jpetrini at coredial.com>:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Hello,
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> I've just started experimenting with nova backup and discovered that
>>>>>>>>> there
>>>>>>>>> is a period of time during the snapshot where the instance becomes
>>>>>>>>> unreachable. Is this behavior expected during a live snapshot? Is there
>>>>>>>>> any
>>>>>>>>> way to prevent this?
>>>>>>>>> 
>>>>>>>>> ___
>>>>>>>>> 
>>>>>>>>> John Petrini
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> Eugen Block                             voice   : +49-40-559 51 75
>>>>>>>> NDE Netzdesign und -entwicklung AG      fax     : +49-40-559 51 77
>>>>>>>> Postfach 61 03 15
>>>>>>>> D-22423 Hamburg                         e-mail  : eblock at nde.ag
>>>>>>>> 
>>>>>>>>      Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>>>>>>>        Sitz und Registergericht: Hamburg, HRB 90934
>>>>>>>>                Vorstand: Jens-U. Mozdzen
>>>>>>>>                 USt-IdNr. DE 814 013 983
>>>>>>>> 
>>>>>>>> 
>>>>>>>> _______________________________________________
>>>>>>>> Mailing list: http://lists.openstack.org/cgi
>>>>>>>> -bin/mailman/listinfo/openstac
>>>>>>>> k
>>>>>>>> Post to     : openstack at lists.openstack.org
>>>>>>>> Unsubscribe : http://lists.openstack.org/cgi
>>>>>>>> -bin/mailman/listinfo/openstac
>>>>>>>> k
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Eugen Block                             voice   : +49-40-559 51 75
>>>>>> NDE Netzdesign und -entwicklung AG      fax     : +49-40-559 51 77
>>>>>> Postfach 61 03 15
>>>>>> D-22423 Hamburg                         e-mail  : eblock at nde.ag
>>>>>> 
>>>>>>      Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>>>>>        Sitz und Registergericht: Hamburg, HRB 90934
>>>>>>                Vorstand: Jens-U. Mozdzen
>>>>>>                 USt-IdNr. DE 814 013 983
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Eugen Block                             voice   : +49-40-559 51 75
>>>> NDE Netzdesign und -entwicklung AG      fax     : +49-40-559 51 77
>>>> Postfach 61 03 15
>>>> D-22423 Hamburg                         e-mail  : eblock at nde.ag
>>>> 
>>>>      Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>>>        Sitz und Registergericht: Hamburg, HRB 90934
>>>>                Vorstand: Jens-U. Mozdzen
>>>>                 USt-IdNr. DE 814 013 983
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>> Post to     : openstack at lists.openstack.org
>>>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> 
>> 
>> 
>> -- 
>> Eugen Block                             voice   : +49-40-559 51 75
>> NDE Netzdesign und -entwicklung AG      fax     : +49-40-559 51 77
>> Postfach 61 03 15
>> D-22423 Hamburg                         e-mail  : eblock at nde.ag
>> 
>>        Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>          Sitz und Registergericht: Hamburg, HRB 90934
>>                  Vorstand: Jens-U. Mozdzen
>>                   USt-IdNr. DE 814 013 983
> 
> 
> 
> -- 
> Eugen Block                             voice   : +49-40-559 51 75
> NDE Netzdesign und -entwicklung AG      fax     : +49-40-559 51 77
> Postfach 61 03 15
> D-22423 Hamburg                         e-mail  : eblock at nde.ag
> 
>        Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>          Sitz und Registergericht: Hamburg, HRB 90934
>                  Vorstand: Jens-U. Mozdzen
>                   USt-IdNr. DE 814 013 983
>