Open Stack

I'm not sure if this is the right spot, but I added some log  
statements into driver.py.
First, there's this if-block:

         if (self._host.has_min_version(MIN_LIBVIRT_LIVESNAPSHOT_VERSION,
                                        MIN_QEMU_LIVESNAPSHOT_VERSION,
                                        host.HV_DRIVER_QEMU)
              and source_type not in ('lvm')
              and not CONF.ephemeral_storage_encryption.enabled
              and not CONF.workarounds.disable_libvirt_livesnapshot):
             live_snapshot = True
        [...]
         else:
             live_snapshot = False

And I know that it lands in the else-statement. Turns out that  
_host.has_min_version is "false", because of host.HV_DRIVER_QEMU. We  
are running on Xen hypervisors. So I tried it with host.HV_DRIVER_XEN  
and now nova-compute says:

[instance: 14b75237-7619-481f-9636-792b64d1be17] instance snapshotting
[instance: 14b75237-7619-481f-9636-792b64d1be17] Beginning live  
snapshot process

Now I'm waiting for the result, but at least the VM is still running,  
so it looks quite promising...

And there it is:

[instance: 14b75237-7619-481f-9636-792b64d1be17] Snapshot image upload  
complete

I'm testing the image now, and it works!

Now the question is, why is it defaulting to HV_DRIVER_QEMU and is it  
really necessary to change this directly in the code? Is there any  
other way?

Regards,
Eugen

Zitat von Eugen Block <eblock at nde.ag>:

> Yes, I truncated the file and uploaded it:
>
> http://dropcanvas.com/ta7nu
> (First time I used this service, please give me feedback if this  
> doesn't work for you)
>
> I see the "Beginning cold snapshot process" message, but I don't  
> know why. Any help is appreciated!
>
> Regards,
> Eugen
>
>
> Zitat von Mohammed Naser <mnaser at vexxhost.com>:
>
>> Would you be able to share the logs of a full snapshot run with the  
>> compute node in debug?
>>
>> Sent from my iPhone
>>
>>> On Jan 12, 2017, at 7:47 AM, Eugen Block <eblock at nde.ag> wrote:
>>>
>>> That's strange, I also searched for this message, but nothing  
>>> there. I have debug logs enabled on compute node but I don't see  
>>> anything regarding ceph. No matter, what I do, my instance is  
>>> always shutdown before a snapshot is taken. What else can I try?
>>>
>>>
>>> Zitat von John Petrini <jpetrini at coredial.com>:
>>>
>>>> Mohammed,
>>>>
>>>> It looks like you may be right. Just found the permissions issue in the
>>>> nova log on the compute node.
>>>>
>>>> 4-e8f52e4fbcfb 691caf1c10354efab3e3c8ed61b7d89a
>>>> 49bc5e5bf2684bd0948d9f94c7875027 - - -] Performing standard snapshot
>>>> because direct snapshot failed: no write permission on storage pool images
>>>>
>>>> I'm going to test the change and will send an update you all with the
>>>> results.
>>>>
>>>> Thank You,
>>>>
>>>> ___
>>>>
>>>> John Petrini
>>>>
>>>>
>>>>
>>>>>>
>>>>> Yes, we are also running Mitaka and I also read Sebastien Han's blogs ;-)
>>>>>
>>>>> our snapshots are not happening at the RBD level,
>>>>>> they are being copied and uploaded to glance which takes up a  
>>>>>> lot of space
>>>>>> and is very slow.
>>>>>>
>>>>>
>>>>> Unfortunately, that's what we are experiencing, too. I don't know if
>>>>> there's something I missed in the nova configs or somewhere else, but I'm
>>>>> relieved that I'm not the only one :-)
>>>>>
>>>>> While writing this email I searched again and found something:
>>>>>
>>>>> https://specs.openstack.org/openstack/nova-specs/specs/mitak
>>>>> a/implemented/rbd-instance-snapshots.html
>>>>>
>>>>> https://review.openstack.org/#/c/205282/
>>>>>
>>>>> It seems to be implemented already, I'm looking for the config options to
>>>>> set. If you manage to get nova to make rbd snapshots, please let  
>>>>> me know ;-)
>>>>>
>>>>> Regards,
>>>>> Eugen
>>>>>
>>>>>
>>>>>
>>>>> Zitat von John Petrini <jpetrini at coredial.com>:
>>>>>
>>>>> Hi Eugen,
>>>>>>
>>>>>> Thanks for the response! That makes a lost of sense and is what  
>>>>>> I figured
>>>>>> was going on but I missed it in the documentation. We use Ceph  
>>>>>> as well and
>>>>>> I had considered doing the snapshots at the RBD level but I was hoping
>>>>>> there was someway to accomplish this via nova. I came across this
>>>>>> Sebastien
>>>>>> Han write-up that claims this functionality was added to Mitaka:
>>>>>> http://www.sebastien-han.fr/blog/2015/10/05/openstack-nova-
>>>>>> snapshots-on-ceph-rbd/
>>>>>>
>>>>>> We are running Mitaka but our snapshots are not happening at the RBD
>>>>>> level,
>>>>>> they are being copied and uploaded to glance which takes up a  
>>>>>> lot of space
>>>>>> and is very slow.
>>>>>>
>>>>>> Have you or anyone else implemented this in Mitaka? Other than  
>>>>>> Sebastian's
>>>>>> blog I haven't found any documentation on this.
>>>>>>
>>>>>> Thank You,
>>>>>>
>>>>>> ___
>>>>>>
>>>>>> John Petrini
>>>>>>
>>>>>> On Wed, Jan 11, 2017 at 3:32 AM, Eugen Block <eblock at nde.ag> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>>
>>>>>>> this seems to be exptected, the docs say:
>>>>>>>
>>>>>>> "Shut down the source VM before you take the snapshot to  
>>>>>>> ensure that all
>>>>>>> data is flushed to disk."
>>>>>>>
>>>>>>> So if the VM is not shut down, it's freezed to prevent data loss (I
>>>>>>> guess). Depending on your storage backend, there are other ways to
>>>>>>> perform
>>>>>>> backups of your VMs.
>>>>>>> We use Ceph as backend for nova, glance and cinder. Ceph stores the
>>>>>>> disks,
>>>>>>> images and volumes as Rados block device objects. We have a  
>>>>>>> backup script
>>>>>>> that creates snapshots of these RBDs, which are exported to our backup
>>>>>>> drive. This way the running VM is not stopped or freezed, the user
>>>>>>> doesn't
>>>>>>> notice any issues. Unlike a nova snapshot, the rbd snapshot is created
>>>>>>> immediately within a few seconds. After a successful backup  
>>>>>>> the snapshots
>>>>>>> are removed.
>>>>>>>
>>>>>>> Hope this helps! If you are interested in Ceph, visit [1].
>>>>>>>
>>>>>>> Regards,
>>>>>>> Eugen
>>>>>>>
>>>>>>> [1] http://docs.ceph.com/docs/giant/start/intro/
>>>>>>>
>>>>>>>
>>>>>>> Zitat von John Petrini <jpetrini at coredial.com>:
>>>>>>>
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>>>
>>>>>>>> I've just started experimenting with nova backup and discovered that
>>>>>>>> there
>>>>>>>> is a period of time during the snapshot where the instance becomes
>>>>>>>> unreachable. Is this behavior expected during a live  
>>>>>>>> snapshot? Is there
>>>>>>>> any
>>>>>>>> way to prevent this?
>>>>>>>>
>>>>>>>> ___
>>>>>>>>
>>>>>>>> John Petrini
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Eugen Block                             voice   : +49-40-559 51 75
>>>>>>> NDE Netzdesign und -entwicklung AG      fax     : +49-40-559 51 77
>>>>>>> Postfach 61 03 15
>>>>>>> D-22423 Hamburg                         e-mail  : eblock at nde.ag
>>>>>>>
>>>>>>>       Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>>>>>>         Sitz und Registergericht: Hamburg, HRB 90934
>>>>>>>                 Vorstand: Jens-U. Mozdzen
>>>>>>>                  USt-IdNr. DE 814 013 983
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Mailing list: http://lists.openstack.org/cgi
>>>>>>> -bin/mailman/listinfo/openstac
>>>>>>> k
>>>>>>> Post to     : openstack at lists.openstack.org
>>>>>>> Unsubscribe : http://lists.openstack.org/cgi
>>>>>>> -bin/mailman/listinfo/openstac
>>>>>>> k
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Eugen Block                             voice   : +49-40-559 51 75
>>>>> NDE Netzdesign und -entwicklung AG      fax     : +49-40-559 51 77
>>>>> Postfach 61 03 15
>>>>> D-22423 Hamburg                         e-mail  : eblock at nde.ag
>>>>>
>>>>>       Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>>>>         Sitz und Registergericht: Hamburg, HRB 90934
>>>>>                 Vorstand: Jens-U. Mozdzen
>>>>>                  USt-IdNr. DE 814 013 983
>>>>>
>>>>>
>>>
>>>
>>>
>>> --
>>> Eugen Block                             voice   : +49-40-559 51 75
>>> NDE Netzdesign und -entwicklung AG      fax     : +49-40-559 51 77
>>> Postfach 61 03 15
>>> D-22423 Hamburg                         e-mail  : eblock at nde.ag
>>>
>>>       Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>>         Sitz und Registergericht: Hamburg, HRB 90934
>>>                 Vorstand: Jens-U. Mozdzen
>>>                  USt-IdNr. DE 814 013 983
>>>
>>>
>>> _______________________________________________
>>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>> Post to     : openstack at lists.openstack.org
>>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
>
> -- 
> Eugen Block                             voice   : +49-40-559 51 75
> NDE Netzdesign und -entwicklung AG      fax     : +49-40-559 51 77
> Postfach 61 03 15
> D-22423 Hamburg                         e-mail  : eblock at nde.ag
>
>         Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>           Sitz und Registergericht: Hamburg, HRB 90934
>                   Vorstand: Jens-U. Mozdzen
>                    USt-IdNr. DE 814 013 983



-- 
Eugen Block                             voice   : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG      fax     : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg                         e-mail  : eblock at nde.ag

         Vorsitzende des Aufsichtsrates: Angelika Mozdzen
           Sitz und Registergericht: Hamburg, HRB 90934
                   Vorstand: Jens-U. Mozdzen
                    USt-IdNr. DE 814 013 983