Open Stack

On Wed, Oct 15, 2014 at 9:58 PM, Jay Pipes <jaypipes at gmail.com> wrote:
> On 10/15/2014 03:16 PM, Florian Haas wrote:
>>
>> On Wed, Oct 15, 2014 at 7:20 PM, Russell Bryant <rbryant at redhat.com>
>> wrote:
>>>
>>> On 10/13/2014 05:59 PM, Russell Bryant wrote:
>>>>
>>>> Nice timing.  I was working on a blog post on this topic.
>>>
>>>
>>> which is now here:
>>>
>>> http://blog.russellbryant.net/2014/10/15/openstack-instance-ha-proposal/
>>
>>
>> I am absolutely loving the fact that we are finally having a
>> discussion in earnest about this. i think this deserves a Design
>> Summit session.
>>
>> If I may weigh in here, let me share what I've seen users do and what
>> can currently be done, and what may be supported in the future.
>>
>> Problem: automatically ensure that a Nova guest continues to run, even
>> if its host fails.
>>
>> (That's the general problem description and I don't need to go into
>> further details explaining the problem, because Russell has done that
>> beautifully in his blog post.)
>>
>> Now, what are the options?
>>
>> (1) Punt and leave it to the hypervisor.
>>
>> This essentially means that you must use a hypervisor that already has
>> HA built in, such as VMware with the VCenter driver. In that scenario,
>> Nova itself neither deals with HA, nor exposes any HA switches to the
>> user. Obvious downside: not generic, doesn't work with all
>> hypervisors, most importantly doesn't work with the most popular one
>> (libvirt/KVM).
>>
>> (2) Deploy Nova nodes in pairs/groups, and pretend that they are one node.
>>
>> You can already do that by overriding "host" in nova-compute.conf,
>> setting resume_guests_state_on_host_boot, and using VIPs with
>> Corosync/Pacemaker. You can then group these hosts in host aggregates,
>> and the user's scheduler hint to point a newly scheduled guest to such
>> a host aggregate becomes, effectively, the "keep this guest running at
>> all times" flag. Upside: no changes to Nova at all, monitoring,
>> fencing and recovery for free from Corosync/Pacemaker. Downsides:
>> requires vendors to automate Pacemaker configuration in deployment
>> tools (because you really don't want to do those things manually).
>> Additional downside: you either have some idle hardware, or you might
>> be overcommitting resources in case of failover.
>>
>> (3) Automatic host evacuation.
>>
>> Not supported in Nova right now, as Adam pointed out at the top of the
>> thread, and repeatedly shot down. If someone were to implement this,
>> it would *still* require that Corosync/Pacemaker be used for
>> monitoring and fencing of nodes, because re-implementing this from
>> scratch would be the reinvention of a wheel while painting a bikeshed.
>>
>> (4) Per-guest HA.
>>
>> This is the idea of just doing "nova boot --keep-this running", i.e.
>> setting a per-guest flag that still means the machine is to be kept up
>> at all times. Again, not supported in Nova right now, and probably
>> even more complex to implement generically than (3), at the same or
>> greater cost.
>>
>> I have a suggestion to tackle this that I *think* is reasonably
>> user-friendly while still bearable in terms of Nova development
>> effort:
>>
>> (a) Define a well-known metadata key for a host aggregate, say "ha".
>> Define that any host aggregate that represents a highly available
>> group of compute nodes should have this metadata key set.
>>
>> (b) Then define a flavor that sets extra_specs "ha=true".
>>
>> Granted, this places an additional burden on distro vendors to
>> integrate highly-available compute nodes into their deployment
>> infrastructure. But since practically all of them already include
>> Pacemaker, the additional scaffolding required is actually rather
>> limited.
>
>
> Or:
>
> (5) Let monitoring and orchestration services deal with these use cases and
> have Nova simply provide the primitive API calls that it already does (i.e.
> host evacuate).

That would arguably lead to an incredible amount of wheel reinvention
for node failure detection, service failure detection, etc. etc.

Florian