Open Stack

Would you be able to share the logs of a full snapshot run with the compute node in debug?

Sent from my iPhone

> On Jan 12, 2017, at 7:47 AM, Eugen Block <eblock at nde.ag> wrote:
> 
> That's strange, I also searched for this message, but nothing there. I have debug logs enabled on compute node but I don't see anything regarding ceph. No matter, what I do, my instance is always shutdown before a snapshot is taken. What else can I try?
> 
> 
> Zitat von John Petrini <jpetrini at coredial.com>:
> 
>> Mohammed,
>> 
>> It looks like you may be right. Just found the permissions issue in the
>> nova log on the compute node.
>> 
>> 4-e8f52e4fbcfb 691caf1c10354efab3e3c8ed61b7d89a
>> 49bc5e5bf2684bd0948d9f94c7875027 - - -] Performing standard snapshot
>> because direct snapshot failed: no write permission on storage pool images
>> 
>> I'm going to test the change and will send an update you all with the
>> results.
>> 
>> Thank You,
>> 
>> ___
>> 
>> John Petrini
>> 
>> 
>> 
>>>> 
>>> Yes, we are also running Mitaka and I also read Sebastien Han's blogs ;-)
>>> 
>>> our snapshots are not happening at the RBD level,
>>>> they are being copied and uploaded to glance which takes up a lot of space
>>>> and is very slow.
>>>> 
>>> 
>>> Unfortunately, that's what we are experiencing, too. I don't know if
>>> there's something I missed in the nova configs or somewhere else, but I'm
>>> relieved that I'm not the only one :-)
>>> 
>>> While writing this email I searched again and found something:
>>> 
>>> https://specs.openstack.org/openstack/nova-specs/specs/mitak
>>> a/implemented/rbd-instance-snapshots.html
>>> 
>>> https://review.openstack.org/#/c/205282/
>>> 
>>> It seems to be implemented already, I'm looking for the config options to
>>> set. If you manage to get nova to make rbd snapshots, please let me know ;-)
>>> 
>>> Regards,
>>> Eugen
>>> 
>>> 
>>> 
>>> Zitat von John Petrini <jpetrini at coredial.com>:
>>> 
>>> Hi Eugen,
>>>> 
>>>> Thanks for the response! That makes a lost of sense and is what I figured
>>>> was going on but I missed it in the documentation. We use Ceph as well and
>>>> I had considered doing the snapshots at the RBD level but I was hoping
>>>> there was someway to accomplish this via nova. I came across this
>>>> Sebastien
>>>> Han write-up that claims this functionality was added to Mitaka:
>>>> http://www.sebastien-han.fr/blog/2015/10/05/openstack-nova-
>>>> snapshots-on-ceph-rbd/
>>>> 
>>>> We are running Mitaka but our snapshots are not happening at the RBD
>>>> level,
>>>> they are being copied and uploaded to glance which takes up a lot of space
>>>> and is very slow.
>>>> 
>>>> Have you or anyone else implemented this in Mitaka? Other than Sebastian's
>>>> blog I haven't found any documentation on this.
>>>> 
>>>> Thank You,
>>>> 
>>>> ___
>>>> 
>>>> John Petrini
>>>> 
>>>> On Wed, Jan 11, 2017 at 3:32 AM, Eugen Block <eblock at nde.ag> wrote:
>>>> 
>>>> Hi,
>>>>> 
>>>>> this seems to be exptected, the docs say:
>>>>> 
>>>>> "Shut down the source VM before you take the snapshot to ensure that all
>>>>> data is flushed to disk."
>>>>> 
>>>>> So if the VM is not shut down, it's freezed to prevent data loss (I
>>>>> guess). Depending on your storage backend, there are other ways to
>>>>> perform
>>>>> backups of your VMs.
>>>>> We use Ceph as backend for nova, glance and cinder. Ceph stores the
>>>>> disks,
>>>>> images and volumes as Rados block device objects. We have a backup script
>>>>> that creates snapshots of these RBDs, which are exported to our backup
>>>>> drive. This way the running VM is not stopped or freezed, the user
>>>>> doesn't
>>>>> notice any issues. Unlike a nova snapshot, the rbd snapshot is created
>>>>> immediately within a few seconds. After a successful backup the snapshots
>>>>> are removed.
>>>>> 
>>>>> Hope this helps! If you are interested in Ceph, visit [1].
>>>>> 
>>>>> Regards,
>>>>> Eugen
>>>>> 
>>>>> [1] http://docs.ceph.com/docs/giant/start/intro/
>>>>> 
>>>>> 
>>>>> Zitat von John Petrini <jpetrini at coredial.com>:
>>>>> 
>>>>> 
>>>>> Hello,
>>>>> 
>>>>>> 
>>>>>> I've just started experimenting with nova backup and discovered that
>>>>>> there
>>>>>> is a period of time during the snapshot where the instance becomes
>>>>>> unreachable. Is this behavior expected during a live snapshot? Is there
>>>>>> any
>>>>>> way to prevent this?
>>>>>> 
>>>>>> ___
>>>>>> 
>>>>>> John Petrini
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Eugen Block                             voice   : +49-40-559 51 75
>>>>> NDE Netzdesign und -entwicklung AG      fax     : +49-40-559 51 77
>>>>> Postfach 61 03 15
>>>>> D-22423 Hamburg                         e-mail  : eblock at nde.ag
>>>>> 
>>>>>        Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>>>>          Sitz und Registergericht: Hamburg, HRB 90934
>>>>>                  Vorstand: Jens-U. Mozdzen
>>>>>                   USt-IdNr. DE 814 013 983
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Mailing list: http://lists.openstack.org/cgi
>>>>> -bin/mailman/listinfo/openstac
>>>>> k
>>>>> Post to     : openstack at lists.openstack.org
>>>>> Unsubscribe : http://lists.openstack.org/cgi
>>>>> -bin/mailman/listinfo/openstac
>>>>> k
>>>>> 
>>>>> 
>>> 
>>> 
>>> --
>>> Eugen Block                             voice   : +49-40-559 51 75
>>> NDE Netzdesign und -entwicklung AG      fax     : +49-40-559 51 77
>>> Postfach 61 03 15
>>> D-22423 Hamburg                         e-mail  : eblock at nde.ag
>>> 
>>>        Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>>          Sitz und Registergericht: Hamburg, HRB 90934
>>>                  Vorstand: Jens-U. Mozdzen
>>>                   USt-IdNr. DE 814 013 983
>>> 
>>> 
> 
> 
> 
> -- 
> Eugen Block                             voice   : +49-40-559 51 75
> NDE Netzdesign und -entwicklung AG      fax     : +49-40-559 51 77
> Postfach 61 03 15
> D-22423 Hamburg                         e-mail  : eblock at nde.ag
> 
>        Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>          Sitz und Registergericht: Hamburg, HRB 90934
>                  Vorstand: Jens-U. Mozdzen
>                   USt-IdNr. DE 814 013 983
> 
> 
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack