[Openstack] nova backup - instances unreachable

Eugen Block eblock at nde.ag
Thu Jan 12 15:43:20 UTC 2017


Yes, I truncated the file and uploaded it:

http://dropcanvas.com/ta7nu
(First time I used this service, please give me feedback if this  
doesn't work for you)

I see the "Beginning cold snapshot process" message, but I don't know  
why. Any help is appreciated!

Regards,
Eugen


Zitat von Mohammed Naser <mnaser at vexxhost.com>:

> Would you be able to share the logs of a full snapshot run with the  
> compute node in debug?
>
> Sent from my iPhone
>
>> On Jan 12, 2017, at 7:47 AM, Eugen Block <eblock at nde.ag> wrote:
>>
>> That's strange, I also searched for this message, but nothing  
>> there. I have debug logs enabled on compute node but I don't see  
>> anything regarding ceph. No matter, what I do, my instance is  
>> always shutdown before a snapshot is taken. What else can I try?
>>
>>
>> Zitat von John Petrini <jpetrini at coredial.com>:
>>
>>> Mohammed,
>>>
>>> It looks like you may be right. Just found the permissions issue in the
>>> nova log on the compute node.
>>>
>>> 4-e8f52e4fbcfb 691caf1c10354efab3e3c8ed61b7d89a
>>> 49bc5e5bf2684bd0948d9f94c7875027 - - -] Performing standard snapshot
>>> because direct snapshot failed: no write permission on storage pool images
>>>
>>> I'm going to test the change and will send an update you all with the
>>> results.
>>>
>>> Thank You,
>>>
>>> ___
>>>
>>> John Petrini
>>>
>>>
>>>
>>>>>
>>>> Yes, we are also running Mitaka and I also read Sebastien Han's blogs ;-)
>>>>
>>>> our snapshots are not happening at the RBD level,
>>>>> they are being copied and uploaded to glance which takes up a  
>>>>> lot of space
>>>>> and is very slow.
>>>>>
>>>>
>>>> Unfortunately, that's what we are experiencing, too. I don't know if
>>>> there's something I missed in the nova configs or somewhere else, but I'm
>>>> relieved that I'm not the only one :-)
>>>>
>>>> While writing this email I searched again and found something:
>>>>
>>>> https://specs.openstack.org/openstack/nova-specs/specs/mitak
>>>> a/implemented/rbd-instance-snapshots.html
>>>>
>>>> https://review.openstack.org/#/c/205282/
>>>>
>>>> It seems to be implemented already, I'm looking for the config options to
>>>> set. If you manage to get nova to make rbd snapshots, please let  
>>>> me know ;-)
>>>>
>>>> Regards,
>>>> Eugen
>>>>
>>>>
>>>>
>>>> Zitat von John Petrini <jpetrini at coredial.com>:
>>>>
>>>> Hi Eugen,
>>>>>
>>>>> Thanks for the response! That makes a lost of sense and is what I figured
>>>>> was going on but I missed it in the documentation. We use Ceph  
>>>>> as well and
>>>>> I had considered doing the snapshots at the RBD level but I was hoping
>>>>> there was someway to accomplish this via nova. I came across this
>>>>> Sebastien
>>>>> Han write-up that claims this functionality was added to Mitaka:
>>>>> http://www.sebastien-han.fr/blog/2015/10/05/openstack-nova-
>>>>> snapshots-on-ceph-rbd/
>>>>>
>>>>> We are running Mitaka but our snapshots are not happening at the RBD
>>>>> level,
>>>>> they are being copied and uploaded to glance which takes up a  
>>>>> lot of space
>>>>> and is very slow.
>>>>>
>>>>> Have you or anyone else implemented this in Mitaka? Other than  
>>>>> Sebastian's
>>>>> blog I haven't found any documentation on this.
>>>>>
>>>>> Thank You,
>>>>>
>>>>> ___
>>>>>
>>>>> John Petrini
>>>>>
>>>>> On Wed, Jan 11, 2017 at 3:32 AM, Eugen Block <eblock at nde.ag> wrote:
>>>>>
>>>>> Hi,
>>>>>>
>>>>>> this seems to be exptected, the docs say:
>>>>>>
>>>>>> "Shut down the source VM before you take the snapshot to ensure that all
>>>>>> data is flushed to disk."
>>>>>>
>>>>>> So if the VM is not shut down, it's freezed to prevent data loss (I
>>>>>> guess). Depending on your storage backend, there are other ways to
>>>>>> perform
>>>>>> backups of your VMs.
>>>>>> We use Ceph as backend for nova, glance and cinder. Ceph stores the
>>>>>> disks,
>>>>>> images and volumes as Rados block device objects. We have a  
>>>>>> backup script
>>>>>> that creates snapshots of these RBDs, which are exported to our backup
>>>>>> drive. This way the running VM is not stopped or freezed, the user
>>>>>> doesn't
>>>>>> notice any issues. Unlike a nova snapshot, the rbd snapshot is created
>>>>>> immediately within a few seconds. After a successful backup the  
>>>>>> snapshots
>>>>>> are removed.
>>>>>>
>>>>>> Hope this helps! If you are interested in Ceph, visit [1].
>>>>>>
>>>>>> Regards,
>>>>>> Eugen
>>>>>>
>>>>>> [1] http://docs.ceph.com/docs/giant/start/intro/
>>>>>>
>>>>>>
>>>>>> Zitat von John Petrini <jpetrini at coredial.com>:
>>>>>>
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>>>
>>>>>>> I've just started experimenting with nova backup and discovered that
>>>>>>> there
>>>>>>> is a period of time during the snapshot where the instance becomes
>>>>>>> unreachable. Is this behavior expected during a live snapshot? Is there
>>>>>>> any
>>>>>>> way to prevent this?
>>>>>>>
>>>>>>> ___
>>>>>>>
>>>>>>> John Petrini
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Eugen Block                             voice   : +49-40-559 51 75
>>>>>> NDE Netzdesign und -entwicklung AG      fax     : +49-40-559 51 77
>>>>>> Postfach 61 03 15
>>>>>> D-22423 Hamburg                         e-mail  : eblock at nde.ag
>>>>>>
>>>>>>        Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>>>>>          Sitz und Registergericht: Hamburg, HRB 90934
>>>>>>                  Vorstand: Jens-U. Mozdzen
>>>>>>                   USt-IdNr. DE 814 013 983
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Mailing list: http://lists.openstack.org/cgi
>>>>>> -bin/mailman/listinfo/openstac
>>>>>> k
>>>>>> Post to     : openstack at lists.openstack.org
>>>>>> Unsubscribe : http://lists.openstack.org/cgi
>>>>>> -bin/mailman/listinfo/openstac
>>>>>> k
>>>>>>
>>>>>>
>>>>
>>>>
>>>> --
>>>> Eugen Block                             voice   : +49-40-559 51 75
>>>> NDE Netzdesign und -entwicklung AG      fax     : +49-40-559 51 77
>>>> Postfach 61 03 15
>>>> D-22423 Hamburg                         e-mail  : eblock at nde.ag
>>>>
>>>>        Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>>>          Sitz und Registergericht: Hamburg, HRB 90934
>>>>                  Vorstand: Jens-U. Mozdzen
>>>>                   USt-IdNr. DE 814 013 983
>>>>
>>>>
>>
>>
>>
>> --
>> Eugen Block                             voice   : +49-40-559 51 75
>> NDE Netzdesign und -entwicklung AG      fax     : +49-40-559 51 77
>> Postfach 61 03 15
>> D-22423 Hamburg                         e-mail  : eblock at nde.ag
>>
>>        Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>          Sitz und Registergericht: Hamburg, HRB 90934
>>                  Vorstand: Jens-U. Mozdzen
>>                   USt-IdNr. DE 814 013 983
>>
>>
>> _______________________________________________
>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to     : openstack at lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



-- 
Eugen Block                             voice   : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG      fax     : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg                         e-mail  : eblock at nde.ag

         Vorsitzende des Aufsichtsrates: Angelika Mozdzen
           Sitz und Registergericht: Hamburg, HRB 90934
                   Vorstand: Jens-U. Mozdzen
                    USt-IdNr. DE 814 013 983





More information about the Openstack mailing list