Open Stack

Coarse grained health checks/recovery at the host or instance level 
might be good enough in some cases, but I can imagine others where the 
host/instance/etc. is up but some aspect of the service has 
crashed/degraded/etc. What's hard about dealing with the latter (e.g. 
service specific monitoring) is that the implementation (and perhaps 
even the recovery) needs to be handled in a way that may be service 
specific as well. For those cases, I'd think it's likely that some HA 
solution (outside of OpenStack) is already being leveraged....and 
perhaps the question becomes more about ensuring in the context of 
OpenStack that the right automation & orchestration exists to properly 
deploy & host that workload with its HA solution, and that the 
underlying infrastructure provisioned can meet requirements around 
isolation, performance, etc.

-Eric

On 12/3/2014 11:04 AM, Maish Saidel-Keesing wrote:
> I would agree with you whole heartedly Tyler.
>
> It is perhaps blasphemy - but Enterprise wants the same they can get 
> today with VMware - and the is standard HA.
>
> Maish
> On 03/12/2014 20:37, Britten, Tyler wrote:
>>
>> It seems like the main ask from the 'pets' side of the enterprise is 
>> not instance monitoring/recovery, but hypervisor monitoring for 
>> instance recovery- KVM host fails, something is checking for a 
>> heartbeat, and once that host is marked as offline, it would check 
>> the db for the instances running on that host and schedule them to 
>> start on other remaining hosts. Ovbiously this would require shared 
>> ephemeral storage (NFS) or limit recovery to boot from volume instances.
>>
>> Am I offbase?
>>
>> *Tyler Britten*
>>
>> Global Cloud Solutions | EMC^2
>>
>> 717.448.4057| tyler.britten at emc.com <mailto:tyler.britten at emc.com> | 
>> @VMTyler <https://twitter.com/vmtyler>
>>
>> *From:*Jason Venner [mailto:jvenner at mirantis.com]
>> *Sent:* Wednesday, December 03, 2014 13:31
>> *To:* Daniel P. Berrange
>> *Cc:* Enterprise-wg at lists.openstack.org; Stefano Maffulli
>> *Subject:* Re: [Win The Enterprise-wg] libvirtWatchdog status
>>
>> Would having an event we push in the bus be sufficient?
>>
>> I can see about having this added to our nova contributor's work queues.
>>
>> On Wed, Dec 3, 2014 at 2:19 AM, Daniel P. Berrange 
>> <berrange at redhat.com <mailto:berrange at redhat.com>> wrote:
>>
>> On Tue, Dec 02, 2014 at 08:48:52PM -0500, Steve Gordon wrote:
>> > ----- Original Message -----
>> > > From: "Stefano Maffulli" <stefano at openstack.org 
>> <mailto:stefano at openstack.org>>
>> > > To: "Daniel P. Berrange" <berrange at redhat.com 
>> <mailto:berrange at redhat.com>>, Enterprise-wg at lists.openstack.org 
>> <mailto:Enterprise-wg at lists.openstack.org>
>> > >
>> > > hi Daniel,
>> > >
>> > > during today's meeting for the Win The Enterprise working group we
>> > > noticed libvirtWatchdog. The wiki page
>> > > https://wiki.openstack.org/wiki/LibvirtWatchdog is authored by you
>> > > originally so I'm reaching out to learn more about the status of this
>> > > feature.
>> > >
>> > > In the WTE team, one of the priorities is to understand the status of
>> > > features that allow non-ephemeral (persistent) workloads on OpenStack
>> > > (aka the "pet" use case). libvirtWatchdog was mentioned during a 
>> session
>> > > in Paris, saying that it currently supports KVM and Linux guests 
>> only.
>> > >
>> > > What are the plans for its future (can/should it be extended to other
>> > > guests/hypervisors)? Who's maintaining it at the moment? Is there any
>> > > other documentation besides the wiki page?
>> >
>> > I'll take a crack at it and then Dan can tell me how wrong I am 
>> since it's probably my fault it was in the etherpad ;). The watchdog 
>> feature in OpenStack is exposing capabilities in the underlying 
>> Libvirt [1] and Qemu [2][3] layers which allow you to attach an 
>> i6300esb watchdog device to the guest and assign a lifecycle action 
>> to take if it is triggered. Fundamentally there's nothing preventing 
>> other hypervisor projects from implementing this, I'm not sure which 
>> ones if any actually have however (and when I cover the second part 
>> of your question below it might become clear why).
>> >
>> > As to why it only works with Linux guests (or more accurately why 
>> it doesn't work for Windows - I wouldn't be surprised if the BSD 
>> family or other OSes do support it to some degree but I've never 
>> checked) I believe it was originally intended to but there were a few 
>> issues uncovered during the chase, in particular:
>> >
>> > 1) The default Window's driver for the device only displays the PCI 
>> information for it (it doesn't actually do anything with the device).
>> >
>> > 2) The Intel driver for this device on Windows only ever worked 
>> with 32-bit editions of Windows.
>> >
>> > 3) The Intel driver for this device on Windows always assumes it's 
>> in a specific PCI slot.
>> >
>> > 4) There's no framework within Windows for triggering a watchdog 
>> device and we weren't able to determine if there were any Windows 
>> applications capable of  triggering one either.
>> >
>> > Basically while you can attach the device to a Windows guest for it 
>> to actually be used it would require someone to write a proper driver 
>> for the device that works on Windows and there would need to be 
>> applications that know how to actually make use of it. In the Linux 
>> case I believe there is wider support for it and it can be triggered 
>> by common panics and lockups (Rich's blog [3] gives some more examples).
>> >
>> > For the gorier details see: 
>> https://bugzilla.redhat.com/show_bug.cgi?id=610063 
>> <https://bugzilla.redhat.com/show_bug.cgi?id=610063>.
>>
>> Yep, that's pretty much it.
>>
>> Also note there's a missing feature in Nova in that we have no mechanism
>> to notify the end user when a watchdog fires on their VMs. Libvirt has
>> this notification ability but we've nowhere to send this info in 
>> OpenStack.
>> We need some kind of formal alerting system to get a message back to the
>> end user (or to an ochestration tool like Heat),  so they can take action
>> when it fires.
>>
>> Regards,
>> Daniel
>> --
>> |: http://berrange.com     -o- http://www.flickr.com/photos/dberrange/ :|
>> |: http://libvirt.org             -o- http://virt-manager.org :|
>> |: http://autobuild.org      -o- http://search.cpan.org/~danberr/ 
>> <http://search.cpan.org/%7Edanberr/> :|
>> |: http://entangle-photo.org      -o- http://live.gnome.org/gtk-vnc :|
>>
>>
>> _______________________________________________
>> Enterprise-wg mailing list
>> Enterprise-wg at lists.openstack.org 
>> <mailto:Enterprise-wg at lists.openstack.org>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/enterprise-wg
>>
>>
>>
>> -- 
>>
>> Jason Venner
>>
>> Vice President and Chief Architect
>> Mirantis Inc
>>
>>
>>
>> _______________________________________________
>> Enterprise-wg mailing list
>> Enterprise-wg at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/enterprise-wg
>
> -- 
> Maish Saidel-Keesing
>
>
> _______________________________________________
> Enterprise-wg mailing list
> Enterprise-wg at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/enterprise-wg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/enterprise-wg/attachments/20141203/6bdc78a7/attachment-0001.html>