<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Wed, Jan 27, 2016 at 8:19 PM, Fausto Marzi <span dir="ltr"><<a href="mailto:fausto.marzi@gmail.com" target="_blank">fausto.marzi@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span><p><span lang="EN-US"><span>Hi Sam,</span></span></p><p><span lang="EN-US"><span>After our conversation, I have few questions and consideration about Ekko, mainly on how it works et similar. Also to make available to the community our discussions:<br></span></span></p><p><span lang="EN-US"><span>-<span style="font:7pt "Times New Roman"">
</span></span></span><span lang="EN-US">In
understand you are placing a backup-agent on the compute node and execute
actions interacting directly with the hypervisor. I’m thinking that while Ekko
execute this actions, the Nova service have no visibility whatsoever of this. I
do not think is a good idea to execute actions directly on the hypervisor without
interacting with the Nova API.</span><span lang="EN-US"> </span></p></span></div></blockquote><div>This is not an ideal situation, no. Nova should be aware of what we are doing and when we are doing it. We are aware of this and plan on purposing ways to be better integrated with Nova (and Cinder for that matter).</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span><p>
</p></span><span><p><span lang="EN-US"><span>-<span style="font:7pt "Times New Roman"">
</span></span></span><span lang="EN-US">On
your assumptions, you said that Nova snapshots creation generate a VM downtime.
I don’t think the assumption is correct, at least in Kilo, Liberty and Mitaka.
The only downtime you may have related to the snapshot, is when you merge back
the snapshot to the original root image, and this is not our case here.</span></p></span></div></blockquote><div>For Kilo and up Nova does leverage the live snapshot allowing for a snapshot without placing the instance into a Paused state. That is correct. Some of the same underlying functions are used for the IncrementalBackup feature of QEMU as well. So you are right its not fair to say that snapshots always cause downtime as that hasn't been the case since Kilo. </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span>
</span><p><span lang="EN-US"><span>-<span style="font:7pt "Times New Roman"">
</span></span></span><span lang="EN-US">How
the restore would work? If you do a restore of the VM and the record of
that VM instance is not available in the Nova DB (i.e. restoring a VM on
a newly
installed Openstack cloud</span>, or in another region, or after a vm has beed destroyed)<span lang="EN-US">what would happen? How do you manage the consistency
of the data between Nova DB and VM status</span></p></div></blockquote><div>Restore has two pieces. The one we are definitely implementing is restoring an backup image to a glance image. At that point anyone could start an instance off of it. Additionally, it _could_ be restored directly back to the instance in question by powering off the instance and restoring the data directly back then starting the instance again.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">
<p><span lang="EN-US"><span>-<span style="font:7pt "Times New Roman"">
</span></span></span><span lang="EN-US">If
you execute a backup of the VM image file without executing a backup of the related VM metadata information (in the shortest time frame as possible) there are chances
the backup can be inconsistent. <br></span></p></div></blockquote><div>I don't see how VM metadata information has anything to do with a proper backup of the data in this case.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><p><span lang="EN-US"></span></p><p>- How the restore would happen if on that moment Keystone or Swift is not available?<br></p></div></blockquote><div>How does anything happen if Keystone isn't available? User can't auth so nothing happens. </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><p></p><p><span lang="EN-US"></span></p><span><p><span lang="EN-US"><span>-<span style="font:7pt "Times New Roman"">
</span></span></span><span lang="EN-US">Does
the backup that Ekko execute, generates bootable image? If not, the image is
not usable and the restore process will take longer to execute the steps to
make the image bootable.</span><span lang="EN-US"> </span></p></span></div></blockquote><div>The backup Ekko would take would be a bit-for-bit copy of what is on the underlying disk. If that is bootable, the it is bootable. </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span><p>
</p></span><span><p><span lang="EN-US"><span>-<span style="font:7pt "Times New Roman"">
</span></span></span><span lang="EN-US"><span> </span>I do not see any advantage in Ekko over using Nova
API to snapshot -> Generate an image -> upload to Glance -> upload to
Swift.</span></p></span></div></blockquote><div>Snapshots are not backups. This is a very important point. Additionally the process you describe is extremely expensive in terms of time, bandwidth, and IO. For sake of example, if you have 1TB of data on an instance and you snapshot it you must upload 1TB to Glance/Swift. The following day you do another snapshot and you must upload another 1TB, likely the majority of the data is exactly the same. With Ekko (or any true backup) you should only be uploading what has changed since the last backup.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span>
<p><span lang="EN-US"><span>-<span style="font:7pt "Times New Roman"">
</span></span></span><span lang="EN-US">The
Ekko approach is limited to Nova, KVM QEMU, having a qemu-agent running on the
VM. I think the scope is probably a bit limited. This is more a feature than a
tool itself, but the problem is being solved I think more efficiently already.</span></p></span></div></blockquote><div>It is not limited to Nova, Libvirt nor QEMU. It also does not _require_ the qemu-agent. It can be used independent of OpenStack (though that is not the endgoal) and even with VMWare or Hyper-V since they both support CBT which is the main component we leverage.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span>
</span><p><span lang="EN-US"><span>-<span style="font:7pt "Times New Roman"">
</span></span></span><span lang="EN-US">By
executing all the actions related to backup (i.e. compression, incremental computation,
upload, I/O and segmented upload to Swift) Ekko is adding a significant load to
the Compute Nodes.</span> All the work is done on the hypervisor and not taken into account by ceilometer (or similar), so for example not billable.<span lang="EN-US"><b> </b>I do not think this is a good idea as distributing the load
over multiple components helps OpenStack to scale and by leveraging the existing API you integrated better with existing tools.<br></span></p></div></blockquote><div>The backup-agent that is purposed to exist on the compute node does not necessarily perform the upload from the compute node since the data may exist on a backend like Ceph or NFS. But in the case of a local storage there is no way to get around this. Further, current nova-snapshots would do all the same things (minus the compression) and have the same overhead. I am not sure what you are speaking about when talking about Ceilometer. Ekko has full plans to have a plugin for this information as well since Ekko is in control of all of this information. The hypervisor is doing very, very little of "the work" and doing nothing that is intensive at all.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><p><span lang="EN-US"></span></p>
<p><span lang="EN-US"><span>-<span style="font:7pt "Times New Roman"">
</span></span></span><span lang="EN-US">There’s
no documentation whatsoever provided with Ekko. I </span>had to read the source code, have conversations directly with you and invest significant time on it<span lang="EN-US"><font color="#0000ff">.</font> I think
provide some documentation is helpful, as the doc link in the openstack/ekko
repo return 404 Not Found.</span></p></div></blockquote><div>This is true. We are a repo of a few weeks. Give it time :). Ekko has an informal mid-cycle planned since all the Core contributors as of now will be at the Kolla midcycle in Feb. We plan on documenting and presenting a roadmap at this time. <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><p><span lang="EN-US"></span></p><div class="gmail_extra">Please let me know what your thoughts are on this.<br><br></div><div class="gmail_extra">Thanks,<br></div><div class="gmail_extra">Fausto<br></div></div></blockquote><div><br></div><div>Hopefully these have answered your questions.</div><div><br></div><div>Sam Yaple </div></div></div></div>