[openstack-dev] [masakari] Intrusive Instance Monitoring
Waines, Greg
Greg.Waines at windriver.com
Mon May 29 14:48:10 UTC 2017
Hey Sam,
Was just trying to submit my spec for Intrusive Instance Monitoring for review.
And I get the following warning after committing when I do the ‘git review’
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs$ git review
You are about to submit multiple commits. This is expected if you are
submitting a commit that is dependent on one or more in-review
commits. Otherwise you should consider squashing your changes into one
commit before submitting.
The outstanding commits are:
f09deee (HEAD -> myBranch) Initial draft specification of Intrusive Instance Monitoring.
21aeb96 (origin/master, origin/HEAD, master) Prepare specs repository for Pike
83d1a0a Implement reserved_host, auto_priority and rh_priority recovery methods
4e746cb Add periodic task to clean up workflow failure
2c10be4 Add spec repo structure
a82016f Added .gitreview
Do you really want to submit the above commits?
Type 'yes' to confirm, other to cancel: no
Aborting.
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs$
Seems like my clone picked up someone else’s open commit ?
Any help would be appreciated,
The full log of my git session is below,
thanks
Greg.
----------------
gwaines at gwaines-VirtualBox:~/openstack$ git clone https://github.com/openstack/masakari-specs.git
Cloning into 'masakari-specs'...
remote: Counting objects: 61, done.
remote: Total 61 (delta 0), reused 0 (delta 0), pack-reused 61
Unpacking objects: 100% (61/61), done.
Checking connectivity... done.
gwaines at gwaines-VirtualBox:~/openstack$ cd masakari-specs/
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs$ git review -s
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs$ git checkout -b myBranch
Switched to a new branch 'myBranch'
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs$ git status
On branch myBranch
nothing to commit, working directory clean
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs$ cd doc/source/specs/pike/
approved/ implemented/
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs$ cd doc/source/specs/pike/implemented
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs/doc/source/specs/pike/implemented$ ls
pike-template.rst
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs/doc/source/specs/pike/implemented$ cp /tmp/vmHeartbeat.masakari.specfile.rst .
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs/doc/source/specs/pike/implemented$ ls -l
total 12
lrwxrwxrwx 1 gwaines gwaines 23 May 29 10:37 pike-template.rst -> ../../pike-template.rst
-rw-rw-r-- 1 gwaines gwaines 11115 May 29 10:38 vmHeartbeat.masakari.specfile.rst
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs/doc/source/specs/pike/implemented$ git status
On branch myBranch
Untracked files:
(use "git add <file>..." to include in what will be committed)
vmHeartbeat.masakari.specfile.rst
nothing added to commit but untracked files present (use "git add" to track)
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs/doc/source/specs/pike/implemented$ git add vmHeartbeat.masakari.specfile.rst
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs/doc/source/specs/pike/implemented$ git status
On branch myBranch
Changes to be committed:
(use "git reset HEAD <file>..." to unstage)
new file: vmHeartbeat.masakari.specfile.rst
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs/doc/source/specs/pike/implemented$ cd ../../../
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs/doc/source$ cd ../../
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs$ cd
gwaines at gwaines-VirtualBox:~$ cd openstack/masakari-specs/
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs$ ls
doc README.rst setup.cfg specs test-requirements.txt
LICENSE requirements.txt setup.py template.rst tox.ini
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs$ pwd
/home/gwaines/openstack/masakari-specs
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs$ git status
On branch myBranch
Changes to be committed:
(use "git reset HEAD <file>..." to unstage)
new file: specs/pike/implemented/vmHeartbeat.masakari.specfile.rst
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs$ git commit -s
[myBranch f09deee] Initial draft specification of Intrusive Instance Monitoring.
1 file changed, 264 insertions(+)
create mode 100644 specs/pike/implemented/vmHeartbeat.masakari.specfile.rst
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs$ git status
On branch myBranch
nothing to commit, working directory clean
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs$ git review
You are about to submit multiple commits. This is expected if you are
submitting a commit that is dependent on one or more in-review
commits. Otherwise you should consider squashing your changes into one
commit before submitting.
The outstanding commits are:
f09deee (HEAD -> myBranch) Initial draft specification of Intrusive Instance Monitoring.
21aeb96 (origin/master, origin/HEAD, master) Prepare specs repository for Pike
83d1a0a Implement reserved_host, auto_priority and rh_priority recovery methods
4e746cb Add periodic task to clean up workflow failure
2c10be4 Add spec repo structure
a82016f Added .gitreview
Do you really want to submit the above commits?
Type 'yes' to confirm, other to cancel: no
Aborting.
gwaines at gwaines-VirtualBox:~/openstack/masakari-specs$
From: Sam P <sam47priya at gmail.com>
Reply-To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Date: Thursday, May 18, 2017 at 2:06 PM
To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [masakari] Intrusive Instance Monitoring
Hi Greg,
Thank you for proposal.
#BTW, I replied to our discussion in [1].
Masakari mainly focuses on black box monitoring the VMs.
But that does not mean Masakari do not do white box type of monitoring.
There will be a configuration options for operators for whether to
use it or not and how to configure it.
For masakari, this is one of the ways to extend its instance
monitoring capabilities.
I really appreciate it if you could write a spec for this in [2], and
it will help masakari community and openstack-ha community to
understand the requirements and
support them in future developments.
[1] http://lists.openstack.org/pipermail/openstack-dev/2017-May/117003.html
[2] https://github.com/openstack/masakari-specs
--- Regards,
Sampath
On Thu, May 18, 2017 at 6:15 AM, Waines, Greg <Greg.Waines at windriver.com<mailto:Greg.Waines at windriver.com>> wrote:
( I have been having a discussion with Adam Spiers on
[openstack-dev][vitrage][nova] on this topic ... thought I would switchover
to [masakari] )
I am interested in contributing an implementation of Intrusive Instance
Monitoring,
initially specifically VM Heartbeat / Heath-check Monitoring thru the QEMU
Guest Agent (https://wiki.libvirt.org/page/Qemu_guest_agent).
I’d like to know whether Masakari project leaders would consider a blueprint
on “VM Heartbeat / Health-check Monitoring”.
See below for some more details,
Greg.
-------------------------------------
VM Heartbeating / Health-check Monitoring would introduce intrusive /
white-box type monitoring of VMs / Instances to Masakari.
Briefly, “VM Heartbeat / Health-check Monitoring”
· is optionally enabled thru a Nova flavor extra-spec,
· is a service that runs on an OpenStack Compute Node,
· it sends periodic Heartbeat / Health-check Challenge Requests to a
VM
over a virtio-serial-device setup between the Compute Node and the VM thru
QEMU,
( https://wiki.libvirt.org/page/Qemu_guest_agent )
· on loss of heartbeat or a failed health check status will result
in fault event, against the VM, being
reported to Masakari and any other registered reporting backends like
Mistral, or Vitrage.
I realize this is somewhat in the gray-zone of what a cloud should be
monitoring or not,
but I believe it provides an alternative for Applications deployed in VMs
that do not have an external monitoring/management entity like a VNF Manager
in the MANO architecture.
And even for VMs with VNF Managers, it provides a highly reliable alternate
monitoring path that does not rely on Tenant Networking.
VM HB/HC Monitoring would leverage
https://wiki.libvirt.org/page/Qemu_guest_agent
that would require the agent to be installed in the images for talking back
to the compute host.
( there are other examples of similar approaches in openstack ... the
murano-agent for installation, the swift-agent for object store management )
Although here, in the case of VM HB/HC Monitoring, via the QEMU Guest Agent,
the messaging path is internal thru a QEMU virtual serial device. i.e. a
very simple interface with very few dependencies ... it’s up and available
very early in VM lifecycle and virtually always up.
Wrt failure modes / use-cases
· a VM’s response to a Heartbeat Challenge Request can be as simple
as just ACK-ing,
this alone allows for detection of:
o a failed or hung QEMU/KVM instance, or
o a failed or hung VM’s OS, or
o a failure of the VM’s OS to schedule the QEMU Guest Agent daemon, or
o a failure of the VM to route basic IO via linux sockets.
· I have had feedback that this is similar to the virtual hardware
watchdog of QEMU/KVM (https://libvirt.org/formatdomain.html#elementsWatchdog
)
· However, the VM Heartbeat / Health-check Monitoring
o provides a higher-level (i.e. application-level) heartbeating
§ i.e. if the Heartbeat requests are being answered by the Application
running within the VM
o provides more than just heartbeating, as the Application can use it to
trigger a variety of audits,
o provides a mechanism for the Application within the VM to report a
Health Status / Info back to the Host / Cloud,
o provides notification of the Heartbeat / Health-check status to
higher-level cloud entities thru Masakari, Mistral and/or Vitrage
§ e.g. VM-Heartbeat-Monitor - to - Vitrage - (EventAlarm) - Aodh - ... -
VNF-Manager
- (StateChange) - Nova - ... - VNF Manager
NOTE: perhaps the reporting to Vitrage would be a separate blueprint within
Masakari.
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org<mailto:OpenStack-dev-request at lists.openstack.org>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org<mailto:OpenStack-dev-request at lists.openstack.org>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170529/fde498e2/attachment.html>
More information about the OpenStack-dev
mailing list