<div dir="ltr">The short of it is to get alerted if an instance fails, rather than waiting for some heartbeat to timeout at the app level HA level.<div><br></div><div>One of the reasons I mentioned the diskio/console log, is I had a production installation where something damaged the VM file system images for a set of VM's [kicked the power cord on the rack] and the FSCK's didn't recover them for some unknown reason.</div><div>The VM's would boot, the app would start and then the extfs would go into read only mode.</div><div>All heart beats ran, the VM's were up, but the app was dead. The Disk writes went to 0 and there were errors in the VM console logs</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Dec 3, 2014 at 10:37 AM, Britten, Tyler <span dir="ltr"><<a href="mailto:Tyler.Britten@emc.com" target="_blank">Tyler.Britten@emc.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">It seems like the main ask from the ‘pets’ side of the enterprise is not instance monitoring/recovery, but hypervisor monitoring for instance recovery- KVM
host fails, something is checking for a heartbeat, and once that host is marked as offline, it would check the db for the instances running on that host and schedule them to start on other remaining hosts. Ovbiously this would require shared ephemeral storage
(NFS) or limit recovery to boot from volume instances.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Am I offbase?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="FR" style="font-size:9.0pt;font-family:"Verdana","sans-serif";color:gray">Tyler Britten<u></u><u></u></span></b></p>
<p class="MsoNormal"><span lang="FR" style="font-size:9.0pt;font-family:"Verdana","sans-serif";color:gray">Global Cloud Solutions |
</span><span style="font-size:9.0pt;font-family:"Verdana","sans-serif";color:gray">EMC</span><sup><span style="font-size:9.0pt;font-family:"Verdana","sans-serif";color:gray">2</span></sup><span lang="FR" style="font-size:9.0pt;font-family:"Verdana","sans-serif";color:gray"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Verdana","sans-serif";color:gray"><a href="tel:717.448.4057" value="+17174484057" target="_blank">717.448.4057</a></span><span style="font-size:9.0pt;font-family:"Verdana","sans-serif";color:#1f497d">
</span><span style="font-size:9.0pt;font-family:"Verdana","sans-serif";color:gray">|
<a href="mailto:tyler.britten@emc.com" target="_blank">tyler.britten@emc.com</a> | <a href="https://twitter.com/vmtyler" target="_blank">
@VMTyler</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Jason Venner [mailto:<a href="mailto:jvenner@mirantis.com" target="_blank">jvenner@mirantis.com</a>]
<br>
<b>Sent:</b> Wednesday, December 03, 2014 13:31<br>
<b>To:</b> Daniel P. Berrange<br>
<b>Cc:</b> <a href="mailto:Enterprise-wg@lists.openstack.org" target="_blank">Enterprise-wg@lists.openstack.org</a>; Stefano Maffulli<span class=""><br>
<b>Subject:</b> Re: [Win The Enterprise-wg] libvirtWatchdog status<u></u><u></u></span></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Would having an event we push in the bus be sufficient?<u></u><u></u></p><div><div class="h5">
<div>
<p class="MsoNormal">I can see about having this added to our nova contributor's work queues.<u></u><u></u></p>
</div>
</div></div></div><div><div class="h5">
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Wed, Dec 3, 2014 at 2:19 AM, Daniel P. Berrange <<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">On Tue, Dec 02, 2014 at 08:48:52PM -0500, Steve Gordon wrote:<br>
> ----- Original Message -----<br>
> > From: "Stefano Maffulli" <<a href="mailto:stefano@openstack.org" target="_blank">stefano@openstack.org</a>><br>
> > To: "Daniel P. Berrange" <<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a>>,
<a href="mailto:Enterprise-wg@lists.openstack.org" target="_blank">Enterprise-wg@lists.openstack.org</a><br>
> ><br>
> > hi Daniel,<br>
> ><br>
> > during today's meeting for the Win The Enterprise working group we<br>
> > noticed libvirtWatchdog. The wiki page<br>
> > <a href="https://wiki.openstack.org/wiki/LibvirtWatchdog" target="_blank">https://wiki.openstack.org/wiki/LibvirtWatchdog</a> is authored by you<br>
> > originally so I'm reaching out to learn more about the status of this<br>
> > feature.<br>
> ><br>
> > In the WTE team, one of the priorities is to understand the status of<br>
> > features that allow non-ephemeral (persistent) workloads on OpenStack<br>
> > (aka the "pet" use case). libvirtWatchdog was mentioned during a session<br>
> > in Paris, saying that it currently supports KVM and Linux guests only.<br>
> ><br>
> > What are the plans for its future (can/should it be extended to other<br>
> > guests/hypervisors)? Who's maintaining it at the moment? Is there any<br>
> > other documentation besides the wiki page?<br>
><br>
> I'll take a crack at it and then Dan can tell me how wrong I am since it's probably my fault it was in the etherpad ;). The watchdog feature in OpenStack is exposing capabilities in the underlying Libvirt [1] and Qemu [2][3] layers which allow you to attach
an i6300esb watchdog device to the guest and assign a lifecycle action to take if it is triggered. Fundamentally there's nothing preventing other hypervisor projects from implementing this, I'm not sure which ones if any actually have however (and when I cover
the second part of your question below it might become clear why).<br>
><br>
> As to why it only works with Linux guests (or more accurately why it doesn't work for Windows - I wouldn't be surprised if the BSD family or other OSes do support it to some degree but I've never checked) I believe it was originally intended to but there
were a few issues uncovered during the chase, in particular:<br>
><br>
> 1) The default Window's driver for the device only displays the PCI information for it (it doesn't actually do anything with the device).<br>
><br>
> 2) The Intel driver for this device on Windows only ever worked with 32-bit editions of Windows.<br>
><br>
> 3) The Intel driver for this device on Windows always assumes it's in a specific PCI slot.<br>
><br>
> 4) There's no framework within Windows for triggering a watchdog device and we weren't able to determine if there were any Windows applications capable of triggering one either.<br>
><br>
> Basically while you can attach the device to a Windows guest for it to actually be used it would require someone to write a proper driver for the device that works on Windows and there would need to be applications that know how to actually make use of it.
In the Linux case I believe there is wider support for it and it can be triggered by common panics and lockups (Rich's blog [3] gives some more examples).<br>
><br>
> For the gorier details see: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=610063" target="_blank">
https://bugzilla.redhat.com/show_bug.cgi?id=610063</a>.<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal">Yep, that's pretty much it.<br>
<br>
Also note there's a missing feature in Nova in that we have no mechanism<br>
to notify the end user when a watchdog fires on their VMs. Libvirt has<br>
this notification ability but we've nowhere to send this info in OpenStack.<br>
We need some kind of formal alerting system to get a message back to the<br>
end user (or to an ochestration tool like Heat), so they can take action<br>
when it fires.<br>
<br>
Regards,<br>
Daniel<br>
<span><span style="color:#888888">--</span></span><span style="color:#888888"><br>
<span>|: <a href="http://berrange.com" target="_blank">http://berrange.com</a> -o-
<a href="http://www.flickr.com/photos/dberrange/" target="_blank">http://www.flickr.com/photos/dberrange/</a> :|</span><br>
<span>|: <a href="http://libvirt.org" target="_blank">http://libvirt.org</a> -o- <a href="http://virt-manager.org" target="_blank">http://virt-manager.org</a> :|</span><br>
<span>|: <a href="http://autobuild.org" target="_blank">http://autobuild.org</a> -o- <a href="http://search.cpan.org/~danberr/" target="_blank">http://search.cpan.org/~danberr/</a> :|</span><br>
<span>|: <a href="http://entangle-photo.org" target="_blank">http://entangle-photo.org</a> -o- <a href="http://live.gnome.org/gtk-vnc" target="_blank">http://live.gnome.org/gtk-vnc</a> :|</span></span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><br>
_______________________________________________<br>
Enterprise-wg mailing list<br>
<a href="mailto:Enterprise-wg@lists.openstack.org" target="_blank">Enterprise-wg@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/enterprise-wg" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/enterprise-wg</a><u></u><u></u></p>
</div>
</div>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">Jason Venner<u></u><u></u></p>
</div>
<p class="MsoNormal">Vice President and Chief Architect<br>
Mirantis Inc<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</div></div></div>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div>Jason Venner<br></div>Vice President and Chief Architect<br>Mirantis Inc<br><div><br></div><div><br></div></div></div>
</div>