Open Stack

Currently the libvirt driver (mostly) considers the nova db canonical. That
is, we can throw away libvirt's domain XML at any time and recreate it from
Nova. Anywhere that doesn't assume this is a bug, because whatever
direction we choose we don't need 2 different sources of truth. The
thinking behind this is that we should always know what we told libvirt,
and if we lose that information then that's a bug.

This is true to a degree, and it's the reason I proposed the persistent
instance storage metadata spec: we lose track of how we configured an
instance's storage. I realised recently that this isn't the whole story,
though. Libvirt also automatically creates a bunch of state for us which we
didn't specify explicitly. We lose this every time we drop it and recreate.
For example, consider device addressing and ordering:

$ nova boot ...

We tell libvirt to give us a root disk, config disk, and a memballoon
device (amongst other things).

Libvirt assigns pci addresses to all of these things.

$ nova volume-attach ...

We tell libvirt to create a new disk attached to the given volume.

Libvirt assigns it a pci address.

$ nova reboot

We throw away libvirt's domain xml and create a new one from scratch.

Libvirt assigns new addresses for all of these devices.

Before reboot, the device order was: root disk, config disk, memballoon,
volume. After reboot the device order is: root disk, volume, config disk,
memballoon. Not only have all our devices changed address, which makes
Windows sad and paranoid about its licensing, and causes it to offline
volumes under certain circumstances, but our disks have been reordered.

This isn't all we've thrown away, though. Libvirt also gave us a default
machine type. When we create a new domain we'll get a new default machine
type. If libvirt has been upgraded, eg during host maintenance, this isn't
necessarily what it was before. Again, this can make our guests sad. Same
goes for CPU model, default devices, and probably many more things I
haven't thought of.

Also... we lost the storage configuration of the guest: the information I
propose to persist in persistent instance storage metadata.

We could store all of this information in Nova, but with the possible
exception of storage metadata it really isn't at the level of 'management':
it's the minutia of the hypervisor. In order to persist all of these things
in Nova we'd have to implement them explicitly, and when libvirt/kvm grows
more stuff we'll have to do that too. We'll need to mirror the
functionality of libvirt in Nova, feature for feature. This is a red flag
for me, and I think it means we should switch to libvirt being canonical.

I think we should be able to create a domain, but once created we should
never redefine a domain. We can do adding and removing devices dynamically
using libvirt's apis, secure in the knowledge that libvirt will persist
this for us. When we upgrade the host, libvirt can ensure we don't break
guests which are on it. Evacuate should be pretty much the only reason to
start again.

This would potentially obsolete my persistent instance metadata spec, and
the libvirt stable rescue spec, as well as this one:
https://review.openstack.org/#/c/347161/ .

I raised this in the live migration sub-team meeting, and the immediate
response was understandably conservative. I think this solves more problems
than it creates, though, and it would result in Nova's libvirt driver
getting a bit smaller and a bit simpler. That's a big win in my book.

Matt
-- 
Matthew Booth
Red Hat Engineering, Virtualisation Team

Phone: +442070094448 (UK)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160927/7a6439a5/attachment.html>