Open Stack

On 11/07/14 12:36, Daniel P. Berrange wrote:
> On Fri, Jul 11, 2014 at 12:30:19PM +0100, John Garbutt wrote:
>> On 10 July 2014 16:52, Matthew Booth <mbooth at redhat.com> wrote:
>>> Currently we create a rescue instance by creating a new VM with the
>>> original instance's image, then adding the original instance's first
>>> disk to it, and booting. This means we have 2 VMs, which we need to be
>>> careful of when cleaning up. Also when suspending, and probably other
>>> edge cases. We also don't support:
>>>
>>> * Rescue images other than the instance's creation image
>>> * Rescue of an instance which wasn't created from an image
>>> * Access to cinder volumes from a rescue instance
>>>
>>> I've created a dirty hack which, instead of creating a new VM, attaches
>>> the given rescue image to the VM and boots from it:
>>>
>>> https://review.openstack.org/#/c/106078/
>>
>> I do worry about different drivers having such radically different
>> implementation approaches.
>>
>> Currently rescue only attaches the root disk to the rescue image.
>> Having a separate VM does side step having to work out where to
>> reattach all the disks when you boot up the original VM, as you
>> haven't modified that. But there are plans to change that here:
>> http://git.openstack.org/cgit/openstack/nova-specs/tree/specs/juno/rescue-attach-all-disks.rst
>>
>> You can now specify and image when you go into rescue mode:
>> http://git.openstack.org/cgit/openstack/nova-specs/tree/specs/juno/allow-image-to-be-specified-during-rescue.rst
>>
>> I guess the rescue image could technically change how the VM boots, or
>> what hardware it has attached, so you might end up making so many
>> "tweaks" to the original VM that you might want to just create a new
>> VM, then through way those changes when you restore the original VM.
>>
>> It feels a lot like we need to better understand the use cases for
>> this feature, and work out what we need in the long term.
>>
>>> Does this seem a reasonable way to go?
>>
>> Maybe, but I am not totally keen on making it such a different
>> implementation to all the other drivers. Mostly for the sake of people
>> who might run two hypervisors in their cloud, or people who support
>> customers running various hypervisors.
> 
> My view is that rescue mode should have as few differences from
> normal mode as possible. Ideally the exact same VM configuration
> would be used, with the exception that you add in one extra disk
> and set the BIOS to boot of that new disk.  The spec you mention
> above gets us closer to that in libvirt, but it still has the
> problem that it re-shuffles the disk order. To fix this I think
> we need to change the rescue image disk so that isntead of being
> a virtio-blk or IDE disk, it is a hotplugged USB disk and make
> the BIOS boot from this USB disk. That way none of the existing
> disk attachments will change in any way. This would also feel
> more like the way a physical machine would be rescued where you
> would typically insert a bootable CDROM or a rescue USB stick
> 
> So in that sense I think that Matt suggests for VMWare is good
> because it gets the vmware driver moving in the right direction.
> I'd encourage them to also follow that libvirt blueprint and
> ensure all disks are attached.

That's interesting. I didn't realise that other drivers had the same
limitations. Does anybody understand the original thinking which lead to
this design? The single VM approach seems intuitively correct to me, so
presumably at some point there was a good reason not to choose it.

I'm not sufficiently familiar with the libvirt driver to know what state
persists in the hypervisor, but in the vmware driver an attached volume
remains attached until detached. By re-using the original VM for rescue,
we wouldn't have to concern ourselves with volumes at all, because they
are all already attached.

As for following the linked BP, I *think* we would achieve that 'for
free' with a single VM approach. Is there any semantic subtlety here
relating to actually attaching the volumes? i.e. Does it matter that we
wouldn't attach them, because they are already attached?

I like the USB idea, and I'm fairly sure it should be achievable in the
VMware driver. Is it worth codifying it in a BP? Perhaps the same BP.

Incidentally, I hit an obvious problem when testing this with the Cirros
image, which mounts filesystems by label. If you have an image which
mounts by label, or has LVM volumes, and you use the same image as a
rescue disk, you are adding a second disk containing the same filesystem
labels and/or LVM volumes. Under these circumstances, the behaviour of
mount during the boot sequence is not well defined afaik. Consequently,
re-using the original image as a rescue image doesn't sound to me like a
good idea in general.

Matt
-- 
Matthew Booth
Red Hat Engineering, Virtualisation Team

Phone: +442070094448 (UK)
GPG ID:  D33C3490
GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490