Open Stack

Watchdogs with reporting are nice, even for horizontally scaled shared
nothing apps.

Knowing that an instance is dead immediately is better than what we do now
implementing complex logic with timeouts, trying to assess is the instance
dead.



On Wed, Dec 3, 2014 at 10:32 PM, Eric Saxe <eric.saxe at oracle.com> wrote:

>
> Coarse grained health checks/recovery at the host or instance level might
> be good enough in some cases, but I can imagine others where the
> host/instance/etc. is up but some aspect of the service has
> crashed/degraded/etc. What's hard about dealing with the latter (e.g.
> service specific monitoring) is that the implementation (and perhaps even
> the recovery) needs to be handled in a way that may be service specific as
> well. For those cases, I'd think it's likely that some HA solution (outside
> of OpenStack) is already being leveraged....and perhaps the question
> becomes more about ensuring in the context of OpenStack that the right
> automation & orchestration exists to properly deploy & host that workload
> with its HA solution, and that the underlying infrastructure provisioned
> can meet requirements around isolation, performance, etc.
>
> -Eric
>
>
> On 12/3/2014 11:04 AM, Maish Saidel-Keesing wrote:
>
> I would agree with you whole heartedly Tyler.
>
> It is perhaps blasphemy - but Enterprise wants the same they can get today
> with VMware - and the is standard HA.
>
> Maish
> On 03/12/2014 20:37, Britten, Tyler wrote:
>
>  It seems like the main ask from the ‘pets’ side of the enterprise is not
> instance monitoring/recovery, but hypervisor monitoring for instance
> recovery- KVM host fails, something is checking for a heartbeat, and once
> that host is marked as offline, it would check the db for the instances
> running on that host and schedule them to start on other remaining hosts.
> Ovbiously this would require shared ephemeral storage (NFS) or limit
> recovery to boot from volume instances.
>
>
>
> Am I offbase?
>
>
>
>
>
> *Tyler Britten*
>
> Global Cloud Solutions | EMC2
>
> 717.448.4057 | tyler.britten at emc.com | @VMTyler
> <https://twitter.com/vmtyler>
>
>
>
> *From:* Jason Venner [mailto:jvenner at mirantis.com <jvenner at mirantis.com>]
> *Sent:* Wednesday, December 03, 2014 13:31
> *To:* Daniel P. Berrange
> *Cc:* Enterprise-wg at lists.openstack.org; Stefano Maffulli
> *Subject:* Re: [Win The Enterprise-wg] libvirtWatchdog status
>
>
>
> Would having an event we push in the bus be sufficient?
>
> I can see about having this added to our nova contributor's work queues.
>
>
>
> On Wed, Dec 3, 2014 at 2:19 AM, Daniel P. Berrange <berrange at redhat.com>
> wrote:
>
> On Tue, Dec 02, 2014 at 08:48:52PM -0500, Steve Gordon wrote:
> > ----- Original Message -----
> > > From: "Stefano Maffulli" <stefano at openstack.org>
> > > To: "Daniel P. Berrange" <berrange at redhat.com>,
> Enterprise-wg at lists.openstack.org
> > >
> > > hi Daniel,
> > >
> > > during today's meeting for the Win The Enterprise working group we
> > > noticed libvirtWatchdog. The wiki page
> > > https://wiki.openstack.org/wiki/LibvirtWatchdog is authored by you
> > > originally so I'm reaching out to learn more about the status of this
> > > feature.
> > >
> > > In the WTE team, one of the priorities is to understand the status of
> > > features that allow non-ephemeral (persistent) workloads on OpenStack
> > > (aka the "pet" use case). libvirtWatchdog was mentioned during a
> session
> > > in Paris, saying that it currently supports KVM and Linux guests only.
> > >
> > > What are the plans for its future (can/should it be extended to other
> > > guests/hypervisors)? Who's maintaining it at the moment? Is there any
> > > other documentation besides the wiki page?
> >
> > I'll take a crack at it and then Dan can tell me how wrong I am since
> it's probably my fault it was in the etherpad ;). The watchdog feature in
> OpenStack is exposing capabilities in the underlying Libvirt [1] and Qemu
> [2][3] layers which allow you to attach an i6300esb watchdog device to the
> guest and assign a lifecycle action to take if it is triggered.
> Fundamentally there's nothing preventing other hypervisor projects from
> implementing this, I'm not sure which ones if any actually have however
> (and when I cover the second part of your question below it might become
> clear why).
> >
> > As to why it only works with Linux guests (or more accurately why it
> doesn't work for Windows - I wouldn't be surprised if the BSD family or
> other OSes do support it to some degree but I've never checked) I believe
> it was originally intended to but there were a few issues uncovered during
> the chase, in particular:
> >
> > 1) The default Window's driver for the device only displays the PCI
> information for it (it doesn't actually do anything with the device).
> >
> > 2) The Intel driver for this device on Windows only ever worked with
> 32-bit editions of Windows.
> >
> > 3) The Intel driver for this device on Windows always assumes it's in a
> specific PCI slot.
> >
> > 4) There's no framework within Windows for triggering a watchdog device
> and we weren't able to determine if there were any Windows applications
> capable of  triggering one either.
> >
> > Basically while you can attach the device to a Windows guest for it to
> actually be used it would require someone to write a proper driver for the
> device that works on Windows and there would need to be applications that
> know how to actually make use of it. In the Linux case I believe there is
> wider support for it and it can be triggered by common panics and lockups
> (Rich's blog [3] gives some more examples).
> >
> > For the gorier details see:
> https://bugzilla.redhat.com/show_bug.cgi?id=610063.
>
> Yep, that's pretty much it.
>
> Also note there's a missing feature in Nova in that we have no mechanism
> to notify the end user when a watchdog fires on their VMs. Libvirt has
> this notification ability but we've nowhere to send this info in OpenStack.
> We need some kind of formal alerting system to get a message back to the
> end user (or to an ochestration tool like Heat),  so they can take action
> when it fires.
>
> Regards,
> Daniel
> --
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/
> :|
> |: http://libvirt.org              -o-             http://virt-manager.org
> :|
> |: http://autobuild.org       -o-         http://search.cpan.org/~danberr/
> :|
> |: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc
> :|
>
>
> _______________________________________________
> Enterprise-wg mailing list
> Enterprise-wg at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/enterprise-wg
>
>
>
>
>
> --
>
> Jason Venner
>
> Vice President and Chief Architect
> Mirantis Inc
>
>
>
>
>
>
> _______________________________________________
> Enterprise-wg mailing listEnterprise-wg at lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/enterprise-wg
>
>
> --
> Maish Saidel-Keesing
>
>
>
> _______________________________________________
> Enterprise-wg mailing listEnterprise-wg at lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/enterprise-wg
>
>
>
> _______________________________________________
> Enterprise-wg mailing list
> Enterprise-wg at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/enterprise-wg
>
>


-- 
Jason Venner
Vice President and Chief Architect
Mirantis Inc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/enterprise-wg/attachments/20141203/33dedeb1/attachment.html>