Open Stack

Nice timing.  I was working on a blog post on this topic.

On 10/13/2014 05:40 PM, Fei Long Wang wrote:
> I think Adam is talking about this bp:
> https://blueprints.launchpad.net/nova/+spec/evacuate-instance-automatically
> 
> For now, we're using Nagios probe/event to trigger the Nova evacuate 
> command, but I think it's possible to do that in Nova if we can find a
> good way to define the trigger policy.

I actually think that's the right way to do it.  There are a couple of
other things to consider:

1) An ideal solution also includes fencing.  When you evacuate, you want
to make sure you've fenced the original compute node.  You need to make
absolutely sure that the same VM can't be running more than once,
especially when the disks are backed by shared storage.

Because of the fencing requirement, another option would be to use
Pacemaker to orchestrate this whole thing.  Historically Pacemaker
hasn't been suitable to scale to the number of compute nodes an
OpenStack deployment might have, but Pacemaker has a new feature called
pacemaker_remote [1] that may be suitable.

2) Looking forward, there is a lot of demand for doing this on a per
instance basis.  We should decide on a best practice for allowing end
users to indicate whether they would like their VMs automatically
rescued by the infrastructure, or just left down in the case of a
failure.  It could be as simple as a special tag set on an instance [2].

[1]
http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Remote/
[2] https://review.openstack.org/#/c/127281/

-- 
Russell Bryant