Open Stack

The short of it is to get alerted if an instance fails, rather than waiting
for some heartbeat to timeout at the app level HA level.

One of the reasons I mentioned the diskio/console log, is I had a
production installation where something damaged the VM file system images
for a set of VM's [kicked the power cord on the rack] and the FSCK's didn't
recover them for some unknown reason.
The VM's would boot, the app would start and then the extfs would go into
read only mode.
All heart beats ran, the VM's were up, but the app was dead. The Disk
writes went to 0 and there were errors in the VM console logs

On Wed, Dec 3, 2014 at 10:37 AM, Britten, Tyler <Tyler.Britten at emc.com>
wrote:

>  It seems like the main ask from the ‘pets’ side of the enterprise is not
> instance monitoring/recovery, but hypervisor monitoring for instance
> recovery- KVM host fails, something is checking for a heartbeat, and once
> that host is marked as offline, it would check the db for the instances
> running on that host and schedule them to start on other remaining hosts.
> Ovbiously this would require shared ephemeral storage (NFS) or limit
> recovery to boot from volume instances.
>
>
>
> Am I offbase?
>
>
>
>
>
> *Tyler Britten*
>
> Global Cloud Solutions | EMC2
>
> 717.448.4057 | tyler.britten at emc.com | @VMTyler
> <https://twitter.com/vmtyler>
>
>
>
> *From:* Jason Venner [mailto:jvenner at mirantis.com]
> *Sent:* Wednesday, December 03, 2014 13:31
> *To:* Daniel P. Berrange
> *Cc:* Enterprise-wg at lists.openstack.org; Stefano Maffulli
> *Subject:* Re: [Win The Enterprise-wg] libvirtWatchdog status
>
>
>
> Would having an event we push in the bus be sufficient?
>
> I can see about having this added to our nova contributor's work queues.
>
>
>
> On Wed, Dec 3, 2014 at 2:19 AM, Daniel P. Berrange <berrange at redhat.com>
> wrote:
>
> On Tue, Dec 02, 2014 at 08:48:52PM -0500, Steve Gordon wrote:
> > ----- Original Message -----
> > > From: "Stefano Maffulli" <stefano at openstack.org>
> > > To: "Daniel P. Berrange" <berrange at redhat.com>,
> Enterprise-wg at lists.openstack.org
> > >
> > > hi Daniel,
> > >
> > > during today's meeting for the Win The Enterprise working group we
> > > noticed libvirtWatchdog. The wiki page
> > > https://wiki.openstack.org/wiki/LibvirtWatchdog is authored by you
> > > originally so I'm reaching out to learn more about the status of this
> > > feature.
> > >
> > > In the WTE team, one of the priorities is to understand the status of
> > > features that allow non-ephemeral (persistent) workloads on OpenStack
> > > (aka the "pet" use case). libvirtWatchdog was mentioned during a
> session
> > > in Paris, saying that it currently supports KVM and Linux guests only.
> > >
> > > What are the plans for its future (can/should it be extended to other
> > > guests/hypervisors)? Who's maintaining it at the moment? Is there any
> > > other documentation besides the wiki page?
> >
> > I'll take a crack at it and then Dan can tell me how wrong I am since
> it's probably my fault it was in the etherpad ;). The watchdog feature in
> OpenStack is exposing capabilities in the underlying Libvirt [1] and Qemu
> [2][3] layers which allow you to attach an i6300esb watchdog device to the
> guest and assign a lifecycle action to take if it is triggered.
> Fundamentally there's nothing preventing other hypervisor projects from
> implementing this, I'm not sure which ones if any actually have however
> (and when I cover the second part of your question below it might become
> clear why).
> >
> > As to why it only works with Linux guests (or more accurately why it
> doesn't work for Windows - I wouldn't be surprised if the BSD family or
> other OSes do support it to some degree but I've never checked) I believe
> it was originally intended to but there were a few issues uncovered during
> the chase, in particular:
> >
> > 1) The default Window's driver for the device only displays the PCI
> information for it (it doesn't actually do anything with the device).
> >
> > 2) The Intel driver for this device on Windows only ever worked with
> 32-bit editions of Windows.
> >
> > 3) The Intel driver for this device on Windows always assumes it's in a
> specific PCI slot.
> >
> > 4) There's no framework within Windows for triggering a watchdog device
> and we weren't able to determine if there were any Windows applications
> capable of  triggering one either.
> >
> > Basically while you can attach the device to a Windows guest for it to
> actually be used it would require someone to write a proper driver for the
> device that works on Windows and there would need to be applications that
> know how to actually make use of it. In the Linux case I believe there is
> wider support for it and it can be triggered by common panics and lockups
> (Rich's blog [3] gives some more examples).
> >
> > For the gorier details see:
> https://bugzilla.redhat.com/show_bug.cgi?id=610063.
>
> Yep, that's pretty much it.
>
> Also note there's a missing feature in Nova in that we have no mechanism
> to notify the end user when a watchdog fires on their VMs. Libvirt has
> this notification ability but we've nowhere to send this info in OpenStack.
> We need some kind of formal alerting system to get a message back to the
> end user (or to an ochestration tool like Heat),  so they can take action
> when it fires.
>
> Regards,
> Daniel
> --
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/
> :|
> |: http://libvirt.org              -o-             http://virt-manager.org
> :|
> |: http://autobuild.org       -o-         http://search.cpan.org/~danberr/
> :|
> |: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc
> :|
>
>
> _______________________________________________
> Enterprise-wg mailing list
> Enterprise-wg at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/enterprise-wg
>
>
>
>
>
> --
>
> Jason Venner
>
> Vice President and Chief Architect
> Mirantis Inc
>
>
>
>
>



-- 
Jason Venner
Vice President and Chief Architect
Mirantis Inc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/enterprise-wg/attachments/20141203/5cc843c4/attachment.html>