<div dir="ltr"><div><span style="font-size:12.8px">Hi Eugen,</span></div><div><span style="font-size:12.8px"><br></span></div><span style="font-size:12.8px">disable_libvirt_livesnapshot is not present in the nova.conf. During the snapshot the nova logs says "Beginning cold snapshot".</span><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">I read about this option in the nova documentation but did not realize it was the default. In fact I assumed it wasn't since it's in the workarounds section. I'll try setting it to false.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Thank You,<br></span><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px"><br></span><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="margin-top:8px;margin-bottom:8px;font-family:helvetica,verdana,arial,sans-serif;font-size:0.85em;color:rgb(51,51,51);line-height:1.4"><span style="font-size:0.85em;line-height:1.4">___</span></p><p style="padding-top:15px;padding-bottom:15px;max-width:600px;font-family:helvetica,verdana,arial,sans-serif;font-size:0.6em;color:rgb(51,51,51);line-height:1.4">The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.</p></div></div></div></div><a href="http:///" target="_blank"></a></div></div></div></div>
<br><div class="gmail_quote">On Mon, Jan 23, 2017 at 3:16 AM, Eugen Block <span dir="ltr"><<a href="mailto:eblock@nde.ag" target="_blank">eblock@nde.ag</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Have you enabled live snapshots in nova.conf?<br>
<br>
The default for this option is "true", so you should check that:<br>
<br>
disable_libvirt_livesnapshot = false<br>
<br>
Is it really a live snaphot? What's in the nova-compute.log? It should say something like<br>
<br>
[instance: XXX] Beginning live snapshot process<div><div class="gmail-h5"><br>
<br>
<br>
Regards,<br>
Eugen<br>
<br>
<br>
<br>
Zitat von John Petrini <<a href="mailto:jpetrini@coredial.com" target="_blank">jpetrini@coredial.com</a>>:<br>
<br>
</div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-h5">
Hi All,<br>
<br>
Following up after making this change. Adding write permissions to the<br>
images pool in Ceph did the trick and RBD snapshots now work. However the<br>
instance is still paused for the duration of the snapshot. Is it possible<br>
to do a live snapshot without pausing the instance?<br>
<br>
Thanks,<br>
<br>
John<br>
<br>
On Fri, Jan 13, 2017 at 5:49 AM, Eugen Block <<a href="mailto:eblock@nde.ag" target="_blank">eblock@nde.ag</a>> wrote:<br>
<br>
</div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-h5">
Thanks,<br>
<br>
for anyone interested in this issue, I filed a bug report:<br>
<a href="https://bugs.launchpad.net/nova/+bug/1656242" rel="noreferrer" target="_blank">https://bugs.launchpad.net/nov<wbr>a/+bug/1656242</a><br>
<br>
<br>
Regards,<br>
Eugen<br>
<br>
<br>
Zitat von Mohammed Naser <<a href="mailto:mnaser@vexxhost.com" target="_blank">mnaser@vexxhost.com</a>>:<br>
<br>
It is likely because this has been tested with QEMU only. I think you<br>
</div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-h5">
might want to bring this up with the Nova team.<br>
<br>
Sent from my iPhone<br>
<br>
On Jan 12, 2017, at 11:28 AM, Eugen Block <<a href="mailto:eblock@nde.ag" target="_blank">eblock@nde.ag</a>> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-h5">
<br>
I'm not sure if this is the right spot, but I added some log statements<br>
into driver.py.<br>
First, there's this if-block:<br>
<br>
if (self._host.has_min_version(MI<wbr>N_LIBVIRT_LIVESNAPSHOT_VERSION<wbr>,<br>
MIN_QEMU_LIVESNAPSHOT_VERSION,<br>
host.HV_DRIVER_QEMU)<br>
and source_type not in ('lvm')<br>
and not CONF.ephemeral_storage_encrypt<wbr>ion.enabled<br>
and not CONF.workarounds.disable_libvi<wbr>rt_livesnapshot):<br>
live_snapshot = True<br>
[...]<br>
else:<br>
live_snapshot = False<br>
<br>
And I know that it lands in the else-statement. Turns out that<br>
_host.has_min_version is "false", because of host.HV_DRIVER_QEMU. We are<br>
running on Xen hypervisors. So I tried it with host.HV_DRIVER_XEN and now<br>
nova-compute says:<br>
<br>
[instance: 14b75237-7619-481f-9636-792b64<wbr>d1be17] instance snapshotting<br>
[instance: 14b75237-7619-481f-9636-792b64<wbr>d1be17] Beginning live<br>
snapshot process<br>
<br>
Now I'm waiting for the result, but at least the VM is still running, so<br>
it looks quite promising...<br>
<br>
And there it is:<br>
<br>
[instance: 14b75237-7619-481f-9636-792b64<wbr>d1be17] Snapshot image upload<br>
complete<br>
<br>
I'm testing the image now, and it works!<br>
<br>
Now the question is, why is it defaulting to HV_DRIVER_QEMU and is it<br>
really necessary to change this directly in the code? Is there any other<br>
way?<br>
<br>
Regards,<br>
Eugen<br>
<br>
Zitat von Eugen Block <<a href="mailto:eblock@nde.ag" target="_blank">eblock@nde.ag</a>>:<br>
<br>
Yes, I truncated the file and uploaded it:<br>
</div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-h5">
<br>
<a href="http://dropcanvas.com/ta7nu" rel="noreferrer" target="_blank">http://dropcanvas.com/ta7nu</a><br>
(First time I used this service, please give me feedback if this<br>
doesn't work for you)<br>
<br>
I see the "Beginning cold snapshot process" message, but I don't know<br>
why. Any help is appreciated!<br>
<br>
Regards,<br>
Eugen<br>
<br>
<br>
Zitat von Mohammed Naser <<a href="mailto:mnaser@vexxhost.com" target="_blank">mnaser@vexxhost.com</a>>:<br>
<br>
Would you be able to share the logs of a full snapshot run with the<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
compute node in debug?<br>
<br>
Sent from my iPhone<br>
<br>
On Jan 12, 2017, at 7:47 AM, Eugen Block <<a href="mailto:eblock@nde.ag" target="_blank">eblock@nde.ag</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
That's strange, I also searched for this message, but nothing there.<br>
I have debug logs enabled on compute node but I don't see anything<br>
regarding ceph. No matter, what I do, my instance is always shutdown before<br>
a snapshot is taken. What else can I try?<br>
<br>
<br>
Zitat von John Petrini <<a href="mailto:jpetrini@coredial.com" target="_blank">jpetrini@coredial.com</a>>:<br>
<br>
Mohammed,<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
It looks like you may be right. Just found the permissions issue in<br>
the<br>
nova log on the compute node.<br>
<br>
4-e8f52e4fbcfb 691caf1c10354efab3e3c8ed61b7d8<wbr>9a<br>
49bc5e5bf2684bd0948d9f94c78750<wbr>27 - - -] Performing standard snapshot<br>
because direct snapshot failed: no write permission on storage pool<br>
images<br>
<br>
I'm going to test the change and will send an update you all with the<br>
results.<br>
<br>
Thank You,<br>
<br>
___<br>
<br>
John Petrini<br>
<br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Yes, we are also running Mitaka and I also read Sebastien Han's<br>
</blockquote>
blogs ;-)<br>
<br>
our snapshots are not happening at the RBD level,<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
they are being copied and uploaded to glance which takes up a lot<br>
of space<br>
and is very slow.<br>
<br>
<br>
</blockquote>
Unfortunately, that's what we are experiencing, too. I don't know if<br>
there's something I missed in the nova configs or somewhere else,<br>
but I'm<br>
relieved that I'm not the only one :-)<br>
<br>
While writing this email I searched again and found something:<br>
<br>
<a href="https://specs.openstack.org/openstack/nova-specs/specs/mitak" rel="noreferrer" target="_blank">https://specs.openstack.org/op<wbr>enstack/nova-specs/specs/mitak</a><br>
a/implemented/rbd-instance-sna<wbr>pshots.html<br>
<br>
<a href="https://review.openstack.org/#/c/205282/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/205282/</a><br>
<br>
It seems to be implemented already, I'm looking for the config<br>
options to<br>
set. If you manage to get nova to make rbd snapshots, please let me<br>
know ;-)<br>
<br>
Regards,<br>
Eugen<br>
<br>
<br>
<br>
Zitat von John Petrini <<a href="mailto:jpetrini@coredial.com" target="_blank">jpetrini@coredial.com</a>>:<br>
<br>
Hi Eugen,<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Thanks for the response! That makes a lost of sense and is what I<br>
figured<br>
was going on but I missed it in the documentation. We use Ceph as<br>
well and<br>
I had considered doing the snapshots at the RBD level but I was<br>
hoping<br>
there was someway to accomplish this via nova. I came across this<br>
Sebastien<br>
Han write-up that claims this functionality was added to Mitaka:<br>
<a href="http://www.sebastien-han.fr/blog/2015/10/05/openstack-nova-" rel="noreferrer" target="_blank">http://www.sebastien-han.fr/bl<wbr>og/2015/10/05/openstack-nova-</a><br>
snapshots-on-ceph-rbd/<br>
<br>
We are running Mitaka but our snapshots are not happening at the<br>
RBD<br>
level,<br>
they are being copied and uploaded to glance which takes up a lot<br>
of space<br>
and is very slow.<br>
<br>
Have you or anyone else implemented this in Mitaka? Other than<br>
Sebastian's<br>
blog I haven't found any documentation on this.<br>
<br>
Thank You,<br>
<br>
___<br>
<br>
John Petrini<br>
<br>
On Wed, Jan 11, 2017 at 3:32 AM, Eugen Block <<a href="mailto:eblock@nde.ag" target="_blank">eblock@nde.ag</a>><br>
wrote:<br>
<br>
Hi,<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
this seems to be exptected, the docs say:<br>
<br>
"Shut down the source VM before you take the snapshot to ensure<br>
that all<br>
data is flushed to disk."<br>
<br>
So if the VM is not shut down, it's freezed to prevent data loss<br>
(I<br>
guess). Depending on your storage backend, there are other ways to<br>
perform<br>
backups of your VMs.<br>
We use Ceph as backend for nova, glance and cinder. Ceph stores<br>
the<br>
disks,<br>
images and volumes as Rados block device objects. We have a<br>
backup script<br>
that creates snapshots of these RBDs, which are exported to our<br>
backup<br>
drive. This way the running VM is not stopped or freezed, the user<br>
doesn't<br>
notice any issues. Unlike a nova snapshot, the rbd snapshot is<br>
created<br>
immediately within a few seconds. After a successful backup the<br>
snapshots<br>
are removed.<br>
<br>
Hope this helps! If you are interested in Ceph, visit [1].<br>
<br>
Regards,<br>
Eugen<br>
<br>
[1] <a href="http://docs.ceph.com/docs/giant/start/intro/" rel="noreferrer" target="_blank">http://docs.ceph.com/docs/gian<wbr>t/start/intro/</a><br>
<br>
<br>
Zitat von John Petrini <<a href="mailto:jpetrini@coredial.com" target="_blank">jpetrini@coredial.com</a>>:<br>
<br>
<br>
Hello,<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I've just started experimenting with nova backup and discovered<br>
that<br>
there<br>
is a period of time during the snapshot where the instance<br>
becomes<br>
unreachable. Is this behavior expected during a live snapshot?<br>
Is there<br>
any<br>
way to prevent this?<br>
<br>
___<br>
<br>
John Petrini<br>
<br>
<br>
<br>
</blockquote>
<br>
--<br>
Eugen Block voice : <a href="tel:%2B49-40-559%2051" value="+494055951" target="_blank">+49-40-559 51</a><br>
75<br>
NDE Netzdesign und -entwicklung AG fax : <a href="tel:%2B49-40-559%2051" value="+494055951" target="_blank">+49-40-559 51</a><br>
77<br>
Postfach 61 03 15<br>
D-22423 Hamburg e-mail : <a href="mailto:eblock@nde.ag" target="_blank">eblock@nde.ag</a><br>
<br>
Vorsitzende des Aufsichtsrates: Angelika Mozdzen<br>
Sitz und Registergericht: Hamburg, HRB 90934<br>
Vorstand: Jens-U. Mozdzen<br>
USt-IdNr. DE 814 013 983<br>
<br>
<br>
______________________________<wbr>_________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi</a><br>
-bin/mailman/listinfo/openstac<br>
k<br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi</a><br>
-bin/mailman/listinfo/openstac<br>
k<br>
<br>
<br>
<br>
</blockquote></blockquote>
<br>
--<br>
Eugen Block voice : <a href="tel:%2B49-40-559%2051%2075" value="+49405595175" target="_blank">+49-40-559 51 75</a><br>
NDE Netzdesign und -entwicklung AG fax : <a href="tel:%2B49-40-559%2051%2077" value="+49405595177" target="_blank">+49-40-559 51 77</a><br>
Postfach 61 03 15<br>
D-22423 Hamburg e-mail : <a href="mailto:eblock@nde.ag" target="_blank">eblock@nde.ag</a><br>
<br>
Vorsitzende des Aufsichtsrates: Angelika Mozdzen<br>
Sitz und Registergericht: Hamburg, HRB 90934<br>
Vorstand: Jens-U. Mozdzen<br>
USt-IdNr. DE 814 013 983<br>
<br>
<br>
<br>
</blockquote></blockquote>
<br>
<br>
--<br>
Eugen Block voice : <a href="tel:%2B49-40-559%2051%2075" value="+49405595175" target="_blank">+49-40-559 51 75</a><br>
NDE Netzdesign und -entwicklung AG fax : <a href="tel:%2B49-40-559%2051%2077" value="+49405595177" target="_blank">+49-40-559 51 77</a><br>
Postfach 61 03 15<br>
D-22423 Hamburg e-mail : <a href="mailto:eblock@nde.ag" target="_blank">eblock@nde.ag</a><br>
<br>
Vorsitzende des Aufsichtsrates: Angelika Mozdzen<br>
Sitz und Registergericht: Hamburg, HRB 90934<br>
Vorstand: Jens-U. Mozdzen<br>
USt-IdNr. DE 814 013 983<br>
<br>
<br>
______________________________<wbr>_________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi</a><br>
-bin/mailman/listinfo/openstac<wbr>k<br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi</a><br>
-bin/mailman/listinfo/openstac<wbr>k<br>
<br>
</blockquote>
<br>
</blockquote>
<br>
<br></div></div><div><div class="gmail-h5">
--<br>
Eugen Block voice : <a href="tel:%2B49-40-559%2051%2075" value="+49405595175" target="_blank">+49-40-559 51 75</a><br>
NDE Netzdesign und -entwicklung AG fax : <a href="tel:%2B49-40-559%2051%2077" value="+49405595177" target="_blank">+49-40-559 51 77</a><br>
Postfach 61 03 15<br>
D-22423 Hamburg e-mail : <a href="mailto:eblock@nde.ag" target="_blank">eblock@nde.ag</a><br>
<br>
Vorsitzende des Aufsichtsrates: Angelika Mozdzen<br>
Sitz und Registergericht: Hamburg, HRB 90934<br>
Vorstand: Jens-U. Mozdzen<br>
USt-IdNr. DE 814 013 983<br>
<br>
</div></div></blockquote><div><div class="gmail-h5">
<br>
<br>
<br>
--<br>
Eugen Block voice : <a href="tel:%2B49-40-559%2051%2075" value="+49405595175" target="_blank">+49-40-559 51 75</a><br>
NDE Netzdesign und -entwicklung AG fax : <a href="tel:%2B49-40-559%2051%2077" value="+49405595177" target="_blank">+49-40-559 51 77</a><br>
Postfach 61 03 15<br>
D-22423 Hamburg e-mail : <a href="mailto:eblock@nde.ag" target="_blank">eblock@nde.ag</a><br>
<br>
Vorsitzende des Aufsichtsrates: Angelika Mozdzen<br>
Sitz und Registergericht: Hamburg, HRB 90934<br>
Vorstand: Jens-U. Mozdzen<br>
USt-IdNr. DE 814 013 983<br>
<br>
<br>
</div></div></blockquote></blockquote><div><div class="gmail-h5">
<br>
<br>
--<br>
Eugen Block voice : <a href="tel:%2B49-40-559%2051%2075" value="+49405595175" target="_blank">+49-40-559 51 75</a><br>
NDE Netzdesign und -entwicklung AG fax : <a href="tel:%2B49-40-559%2051%2077" value="+49405595177" target="_blank">+49-40-559 51 77</a><br>
Postfach 61 03 15<br>
D-22423 Hamburg e-mail : <a href="mailto:eblock@nde.ag" target="_blank">eblock@nde.ag</a><br>
<br>
Vorsitzende des Aufsichtsrates: Angelika Mozdzen<br>
Sitz und Registergericht: Hamburg, HRB 90934<br>
Vorstand: Jens-U. Mozdzen<br>
USt-IdNr. DE 814 013 983<br>
<br>
<br>
</div></div></blockquote></blockquote>
<br>
<br><div class="gmail-HOEnZb"><div class="gmail-h5">
<br>
-- <br>
Eugen Block voice : <a href="tel:%2B49-40-559%2051%2075" value="+49405595175" target="_blank">+49-40-559 51 75</a><br>
NDE Netzdesign und -entwicklung AG fax : <a href="tel:%2B49-40-559%2051%2077" value="+49405595177" target="_blank">+49-40-559 51 77</a><br>
Postfach 61 03 15<br>
D-22423 Hamburg e-mail : <a href="mailto:eblock@nde.ag" target="_blank">eblock@nde.ag</a><br>
<br>
Vorsitzende des Aufsichtsrates: Angelika Mozdzen<br>
Sitz und Registergericht: Hamburg, HRB 90934<br>
Vorstand: Jens-U. Mozdzen<br>
USt-IdNr. DE 814 013 983<br>
<br>
</div></div></blockquote></div><br></div></div></div></div>