Open Stack

On Thu, Oct 16, 2014 at 5:04 AM, Russell Bryant <rbryant at redhat.com> wrote:
> On 10/15/2014 05:07 PM, Florian Haas wrote:
>> On Wed, Oct 15, 2014 at 10:03 PM, Russell Bryant <rbryant at redhat.com> wrote:
>>>> Am I making sense?
>>>
>>> Yep, the downside is just that you need to provide a new set of flavors
>>> for "ha" vs "non-ha".  A benefit though is that it's a way to support it
>>> today without *any* changes to OpenStack.
>>
>> Users are already very used to defining new flavors. Nova itself
>> wouldn't even need to define those; if the vendor's deployment tools
>> defined them it would be just fine.
>
> Yes, I know Nova wouldn't need to define it.  I was saying I didn't like
> that it was required at all.

Fair enough, but do consider that, for example, Trove already
routinely defines flavors of its own.

So I don't think that's quite as painful (to users) as you think.

>>> This seems like the kind of thing we should also figure out how to offer
>>> on a per-guest basis without needing a new set of flavors.  That's why I
>>> also listed the server tagging functionality as another possible solution.
>>
>> This still doesn't do away with the requirement to reliably detect
>> node failure, and to fence misbehaving nodes. Detecting that a node
>> has failed, and fencing it if unsure, is a prerequisite for any
>> recovery action. So you need Corosync/Pacemaker anyway.
>
> Obviously, yes.  My post covered all of that directly ... the tagging
> bit was just additional input into the recovery operation.

This is essentially why I am saying using the Pacemaker stack is the
smarter approach than hacking something into Ceilometer and Heat. You
already need Pacemaker for service availability (and all major vendors
have adopted it for that purpose), so a highly available cloud that
does *not* use Pacemaker at all won't be a vendor supported option for
some time. So people will already be running Pacemaker — then why not
use it for what it's good at?

(Yes, I am aware of things like etcd and fleet. I think that's headed
in the right direction, but hasn't nearly achieved the degree of
maturity that Pacemaker has. All of HA is about performing correctly
in weird corner cases, and you're only able to do that if you've run
into them and got your nose bloody.)

And just so my position is clear, what Pacemaker is good at is node
and service monitoring, recovery, and fencing. It's *not* particularly
good at usability. Which is why it makes perfect sense to not have
your Pacemaker configurations managed directly by a human, but have an
automated deployment facility do it. Which the vendors are already
doing.

>> Note also that when using an approach where you have physically
>> clustered nodes, but you are also running non-HA VMs on those, then
>> the user must understand that the following applies:
>>
>> (1) If your guest is marked HA, then it will automatically recover on
>> node failure, but
>> (2) if your guest is *not* marked HA, then it will go down with the
>> node not only if it fails, but also if it is fenced.
>>
>> So a non-HA guest on an HA node group actually has a slightly
>> *greater* chance of going down than a non-HA guest on a non-HA host.
>> (And let's not get into "don't use fencing then"; we all know why
>> that's a bad idea.)
>>
>> Which is why I think it makes sense to just distinguish between
>> HA-capable and non-HA-capable hosts, and have the user decide whether
>> they want HA or non-HA guests simply by assigning them to the
>> appropriate host aggregates.
>
> Very good point.  I hadn't considered that.

Yay, I've contributed something useful to this discussion then. :)

Cheers,
Florian