<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Aug 16, 2013 at 3:05 AM, Daniel P. Berrange <span dir="ltr"><<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On Wed, Aug 14, 2013 at 04:53:01PM -0700, Vishvananda Ishaya wrote:<br>
> Hi Everyone,<br>
><br>
> I have been trying for some time to get the code for the live-snapshot blueprint[1]<br>
> in. Going through the review process for the rpc and interface code[2] was easy. I<br>
> suspect the api-extension code[3] will also be relatively trivial to get in. The<br>
> main concern is with the libvirt driver implementation[4]. I'd like to discuss the<br>
> concerns and see if we can make some progress.<br>
><br>
> Short Summary (tl;dr)<br>
> =====================<br>
><br>
> I propose we merge live-cloning as an experimental feature for havanna and have the<br>
> api extension disabled by default.<br>
><br>
> Overview<br>
> ========<br>
><br>
> First of all, let me express the value of live snapshoting. The slowest part of the<br>
> vm provisioning process is generally booting of the OS. The advantage of live-<br>
> snapshotting is that it allows the possibility of bringing up application servers<br>
> while skipping the overhead of vm (and application startup).<br>
<br>
</div>For Linux at least I think bootup time is a problem that is being solved by the<br>
distros. It is possible to boot up many modern Linux distros in a couple of seconds<br>
even in physical hardware - VMs can be even faster since they don't have such stupid<br>
BIOS to worry about & have a restricted set of possible hardware. This is on a par<br>
with, or better than, the overheads imposed by Nova itself in the boot up process.<br>
<br>
Windows may be a different story, but I've not used it in years so don't know what<br>
its boot performance is like.<br>
<div class="im"><br>
> I recognize that this capability comes with some security concerns, so I don't expect<br>
> this feature to go in and be ready to for use in production right away. Similarly,<br>
> containers have a lot of the same benefit, but have had their own security issues<br>
> which are gradually being resolved. My hope is that getting this feature in would<br>
> allow people to start experimenting with live-booting so that we could uncover some<br>
> of these security issues.<br>
><br>
> There are two specific concerns that have been raised regarding my patch. The first<br>
> concern is related to my use of libvirt. The second concern is related to the security<br>
> issues above. Let me address them separately.<br>
><br>
> 1. Libvirt Issues<br>
> =================<br>
><br>
> The only feature I require from the hypervisor is to load memory/processor state for<br>
> a vm from a file. Qemu supports this directly. The only way that libvirt exposes this<br>
> functionality is via its restore command which is specifically for restoring the<br>
> previous state of an existing vm. "Cloning", or restoring the memory state of a<br>
> cloned vm is considered unsafe (which I will address in the second point, below).<br>
><br>
> The result of the limited api is that I must include some hacks to make the restore<br>
> command actually allow me to restore the state of the new vm. I recognize that this<br>
> is using an undocumented libvirt api and isn't the ideal solution, but it seemed<br>
> "better" then avoiding libvirt and talking directly to qemu.<br>
><br>
> This is obviously not ideal. It is my hope that this 0.1 version of the feature will<br>
> allow us to iteratively improve the live-snapshot/clone proccess and get the security<br>
> to a point where the libvirt maintainers would be willing to accept a patch to directly<br>
> expose an api to load memory from a file.<br>
<br>
</div>To characterize this as a libvirt issue is somewhat misleading. The reason why libvirt<br>
does not explicitly allow this, is that from discussions with the upstream QEMU/KVM<br>
developers, the recommendation/advise that this is not a safe operation and should not<br>
be exposed to application developers.<br>
<br>
The expectation is that the functionality in QEMU is only targetted for taking point in<br>
time snapshots & allowing rollback of a VM to those snapshots, not creating clones of<br>
active VMs.<br></blockquote><div><br></div><div>Thanks for the clarification here. I wasn't aware that this requirement came from qemu</div><div>upstream.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><br>
> 2. Security Concerns<br>
> ====================<br>
><br>
> There are a number of security issues with loading state from another vm. Here is a<br>
> short list of things that need to be done just to make a cloned vm usable:<br>
><br>
> a) mac address needs to be recreated<br>
> b) entropy pool needs to be reset<br>
> c) host name must be reset<br>
> d) host keys bust be regenerated<br>
><br>
> There are others, and trying to clone a running application as well may expose other<br>
> sensitive data, especially if users are snaphsoting vms and making them public.<br>
><br>
> The only issue that I address on the driver side is the mac addresses. This is the<br>
> minimum that needs to be done just to be able to access the vm over the network. This<br>
> is implemented by unplugging all network devices before the snapshot and plugging new<br>
> network devices in on clone. This isn't the most friendly thing to guest applications,<br>
> but it seems like the safest option for the first version of this feature.<br>
<br>
</div>This is not really as safe as you portray. When restoring from the snapshot the VM<br>
will initially be running with virtual NIC with a different MAC address from the one<br>
associated with the in memory OS kernel state. Even if you hotunplug the device and<br>
plug in a new one, you still have a period where the virtual hardware exposed to the<br>
guest does not match the memory state of the guest OS. Perhaps you will be lucky and<br>
not hit problems with some OS, but equally you can be unlucky and do bad things to<br>
the OS kernel or application state. Relying on luck in this way does not lead to a<br>
supportable solution IMHO.<br></blockquote><div><br></div><div>Just a clarification, the most recent version of the patch unplugs network devices before</div><div>the snapshot, so there is no longer a window here.</div><div>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
There are other unique identifiers exposed in the virtual hardware that will/should<br>
change when you clone VMs too, the host UUID, the storage serial keys and you cannot<br>
easily fix those by just unplugging hardware & replugging it.<br></blockquote><div><br></div><div>Clearly, hence my comment above about needing a guest agent to do the important work.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><br>
> So cloning vms must be done with care. Sensitive data must be removed from the vm<br>
> pre-clone and new data needs to be generated post-clone. Ultimately this should all<br>
> be done via guest-agent of some sort. I have found some volunteers to make the guest<br>
> agent a reality, but it will take a bit of time to get something workable, and it<br>
> will be much more difficult if there isn't a way to test the feature.<br>
<br>
</div>Note that even if you think you have removed security data from a VM's filesystem,<br>
it is quite likely that the data will still in fact exist in the unallocated sectors<br>
of VM's block devices and can be fairly easily recovered from them.<br>
<br>
The libguestfs project provide tools to perform offline cloning of VM disk images.<br>
Its virt-sysprep knows how to delete alot (but by no means all possible) sensitive<br>
file data for common Linux & Windows OS. It still has to be combined with use of<br>
the virt-sparsify tool though, to ensure the deleted data is actually purged from<br>
the VM disk image as well as the filesystem, by releasing all unused VM disk sectors<br>
back to the host storage (and not all storage supports that).<br>
<div class="im"><br>
> Conclusion<br>
> ==========<br>
><br>
> There are obviously problems to be solved with the whole idea of live cloning, but<br>
> I think it enables some important new ways of deploying applications. Imagine for<br>
> example a PaaS built on fast-cloning vms instead of containers. This is clearly a<br>
> long term project but if we block it now it may never get the support it needs to<br>
> become a real option.<br>
><br>
> Proposal<br>
> ========<br>
><br>
> I propose we allow the patch in and we leave the live-snapshot extension disabled<br>
> by default. Deployers can turn on the extension to experiment with the feature.<br>
> This will allow other hypervisors do do an implementation, and the community in<br>
> general to improve the security and usefulness of live-cloned virtual machines.<br>
><br>
> I'm very interested in your thoughts and feedback. Thank you to everyone who made<br>
> it this far.<br>
<br>
</div>I don't think it is a good idea to add a feature which is considered to<br>
be unsupportable by the developers of the virt platform.<br></blockquote><div><br></div><div>You make excellent points. I'm not totally convinced that this feature is the right</div><div>long-term direction, but I still think it is interesting. To be fair, I'm not convinced that</div>
<div>virtual machines as a whole are the right long-term direction. I'm still looking for a way</div><div>for people experiment with this and see what use-cases that come out of it.</div><div><br></div><div>Over the past three years OpenStack has been a place where we can iterate quickly and<br>
</div><div>try new things. Multihost nova-network was an experiment of mine that turned into the</div><div>most common deployment strategy for a long time.</div><div><br></div><div>Maybe we've grown up to the point where we have to be more careful and not introduce</div>
<div>these kind of features and the maintenance cost of introducing experimental features is</div><div>too great. If that is the community consensus, then I'm happy keep the live snapshot stuff</div><div>in a branch on github for people to experiment with.</div>
<div><br></div><div>Vish</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Regards,<br>
Daniel<br>
<span class="HOEnZb"><font color="#888888">--<br>
|: <a href="http://berrange.com" target="_blank">http://berrange.com</a> -o- <a href="http://www.flickr.com/photos/dberrange/" target="_blank">http://www.flickr.com/photos/dberrange/</a> :|<br>
|: <a href="http://libvirt.org" target="_blank">http://libvirt.org</a> -o- <a href="http://virt-manager.org" target="_blank">http://virt-manager.org</a> :|<br>
|: <a href="http://autobuild.org" target="_blank">http://autobuild.org</a> -o- <a href="http://search.cpan.org/~danberr/" target="_blank">http://search.cpan.org/~danberr/</a> :|<br>
|: <a href="http://entangle-photo.org" target="_blank">http://entangle-photo.org</a> -o- <a href="http://live.gnome.org/gtk-vnc" target="_blank">http://live.gnome.org/gtk-vnc</a> :|<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</font></span></blockquote></div><br></div></div>