Open Stack

Be forewarned; here's my two cents before I've had my morning coffee.

It would seem to me that if we were seeking some level of resiliency
against host failures (if a host fails, evacuate the instances that were
hosted on it to a host that isn't broken), it would seem that host HA is a
good approach. The ultimate goal of course is instance HA but the task of
monitoring individual instances and determining what constitutes "down"
seems like a much more complex task than detecting when a compute node is
down. I know that requiring the presence of agents should probably need
some more brain-cycles since we can't expect additional bytes consuming
memory on each individual VM.

Additionally, I'm not really hung up on the 'how' as we all realize there
several ways to skin that cat, so long as that 'how' is leveraged via tools
over which we have control and direct influence. Reason being, we may not
want to leverage features as important as this on tools that change outside
our control and subsequently shifts the foundation of the feature we
implemented that was based on how the product USED to work. Basically if
Pacemaker does what we need then cool but it seems that implementing a
feature should be built upon a bedrock of programs over which we have a
direct influence. This is why Nagios may be able to do it but it's a hack
at best. I'm not saying Nagios isn't good or ythe hack doesn't work but in
the context of an Openstack solution, we can't require a single external
tool for a feature like host or VM HA. Are we suggesting that we tell
people who want HA - "go use Nagios"? Call me a purist but if we're going
to implement a feature, it should be our community implementing it because
we have some of the best minds on staff. ; )



*Adam Lawson*

AQORN, Inc.
427 North Tatnall Street
Ste. 58461
Wilmington, Delaware 19801-2230
Toll-free: (844) 4-AQORN-NOW ext. 101
International: +1 302-387-4660
Direct: +1 916-246-2072


On Thu, Oct 16, 2014 at 7:53 AM, Russell Bryant <rbryant at redhat.com> wrote:

> On 10/16/2014 09:00 AM, Florian Haas wrote:
> > On Thu, Oct 16, 2014 at 1:59 PM, Russell Bryant <rbryant at redhat.com>
> wrote:
> >> On 10/16/2014 04:29 AM, Florian Haas wrote:
> >>>>>>> (5) Let monitoring and orchestration services deal with these use
> >>>>>>> cases and
> >>>>>>> have Nova simply provide the primitive API calls that it already
> does
> >>>>>>> (i.e.
> >>>>>>> host evacuate).
> >>>>>>
> >>>>>> That would arguably lead to an incredible amount of wheel
> reinvention
> >>>>>> for node failure detection, service failure detection, etc. etc.
> >>>>>
> >>>>> How so? (5) would use existing wheels for monitoring and
> orchestration
> >>>>> instead of writing all new code paths inside Nova to do the same
> thing.
> >>>>
> >>>> Right, there may be some confusion here ... I thought you were both
> >>>> agreeing that the use of an external toolset was a good approach for
> the
> >>>> problem, but Florian's last message makes that not so clear ...
> >>>
> >>> While one of us (Jay or me) speaking for the other and saying we agree
> >>> is a distributed consensus problem that dwarfs the complexity of
> >>> Paxos, *I* for my part do think that an "external" toolset (i.e. one
> >>> that lives outside the Nova codebase) is the better approach versus
> >>> duplicating the functionality of said toolset in Nova.
> >>>
> >>> I just believe that the toolset that should be used here is
> >>> Corosync/Pacemaker and not Ceilometer/Heat. And I believe the former
> >>> approach leads to *much* fewer necessary code changes *in* Nova than
> >>> the latter.
> >>
> >> Have you tried pacemaker_remote yet?  It seems like a better choice for
> >> this particular case, as opposed to using corosync, due to the potential
> >> number of compute nodes.
> >
> > I'll assume that you are *not* referring to running Corosync/Pacemaker
> > on the compute nodes plus pacemaker_remote in the VMs, because doing
> > so would blow up the separation between the cloud operator and tenant
> > space.
>
> Correct.
>
> > Running compute nodes as baremetal extensions of a different
> > Corosync/Pacemaker cluster (presumably the one that manages the other
> > Nova services)  would potentially be an option, although vendors would
> > need to buy into this. Ubuntu, for example, currently only ships
> > pacemaker-remote in universe.
>
> Yes, this is what I had in mind.
>
> --
> Russell Bryant
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20141016/fd05ebea/attachment.html>