<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
I didn't mean a robust monitoring system, just something simpler.
Notifications is a good idea for FuelWeb.<br>
<br>
P.<br>
<br>
<div class="moz-cite-prefix">On 11/06/2014 09:59 AM, Anton Zemlyanov
wrote:<br>
</div>
<blockquote
cite="mid:CA+ABCu=nB0Z9SGDELsr9SRYTcHeRK3LEVK5MLh_fnSrvTgEG9Q@mail.gmail.com"
type="cite">
<div dir="ltr">We can add a notification to FuelWeb, no additional
software or user actions are required. I would not overestimate
this method though, it is in no way the robust monitoring
system. Forcing user to do something on a regular basis is
unlikely to work.
<div>
<div><br>
</div>
<div>Anton</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Nov 6, 2014 at 11:55 AM,
Przemyslaw Kaminski <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:pkaminski@mirantis.com" target="_blank">pkaminski@mirantis.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> I think we're missing
the point here. What I meant adding a simple monitoring
system that informed the user via UI/CLI/email/whatever of
low resources on fuel master node. That's it. HA here is
not an option -- if, despite of warnings, the user still
continues to use fuel and disk becomes full, it's the
user's fault. By adding these warnings we have a way of
saying "We told you so!" Without warnings we get bugs like
[1] I mentioned in the first post.<br>
<br>
Of course user can check disk space by hand but since we
do have a full-blown UI telling the user to periodically
log in to the console and check disks by hand seems a bit
of a burden.<br>
<br>
We can even implement such monitoring functionality as a
Nailgun plugin -- installing it would be optional and at
the same time we would grow our plugin ecosystem.<span
class="HOEnZb"><font color="#888888"><br>
<br>
P.</font></span>
<div>
<div class="h5"><br>
<br>
<div>On 11/05/2014 08:42 PM, Dmitry Borodaenko wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Even one additional hardware node
required to host the Fuel master is seen by many
users as excessive. Unless you can come up with an
architecture that adds HA capability to Fuel
without increasing its hardware footprint by 2
more nodes, it's just not worth it.<br>
<br>
The only operational aspect of the Fuel master
node that you don't want to lose even for a short
while is logging. You'd be better off redirecting
OpenStack environments' logs to a dedicated highly
available logging server (which, of course, you
already have in your environment), and deal with
Fuel master node failures by restoring it from
backups.<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Nov 5, 2014 at
8:26 AM, Anton Zemlyanov <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:azemlyanov@mirantis.com"
target="_blank">azemlyanov@mirantis.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">Monitoring of the
Fuel master's disk space is the special
case. I really wonder why Fuel master have
no HA option, disk overflow can be
predicted but many other failures cannot.
HA is a solution of the 'single point of
failure' problem.</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">The current
monitoring recommendations (<a
moz-do-not-send="true"
href="http://docs.openstack.org/openstack-ops/content/logging_monitoring.html"
target="_blank">http://docs.openstack.org/openstack-ops/content/logging_monitoring.html</a>)
are based on analyzing logs and manual
checks, that are rather reactive way of
fixing problems. Zabbix is quite good for
preventing failures that are predictable
but for the abrupt problems Zabbix just
reports them 'post mortem'.</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">The only way to
remove the single failure point is to
implement redundancy/HA</div>
<span><font color="#888888">
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">Anton</div>
</font></span>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Nov
4, 2014 at 6:26 PM, Przemyslaw
Kaminski <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:pkaminski@mirantis.com"
target="_blank">pkaminski@mirantis.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hello,<br>
<br>
In extension to my comment in this
bug [1] I'd like to discuss the
possibility of adding Fuel master
node monitoring. As I wrote in the
comment, when disk is full it
might be already too late to
perform any action since for
example Nailgun could be down
because DB shut itself down. So we
should somehow warn the user that
disk is running low (in the UI and
fuel CLI on stderr for example)
before it actually happens.<br>
<br>
For now the only meaningful value
to monitor would be disk usage --
do you have other suggestions? If
not then probably a simple API
endpoint with statvfs calls would
suffice. If you see other usages
of this then maybe it would be
better to have some daemon
collecting the stats we want.<br>
<br>
If we opted for a daemon, then I'm
aware that the user can optionally
install Zabbix server although
looking at blueprints in [2] I
don't see anything about
monitoring Fuel master itself --
is it possible to do? Though the
installation of Zabbix though is
not mandatory so it still doesn't
completely solve the problem.<br>
<br>
[1] <a moz-do-not-send="true"
href="https://bugs.launchpad.net/fuel/+bug/1371757"
target="_blank">https://bugs.launchpad.net/fuel/+bug/1371757</a><br>
[2] <a moz-do-not-send="true"
href="https://blueprints.launchpad.net/fuel/+spec/monitoring-system"
target="_blank">https://blueprints.launchpad.net/fuel/+spec/monitoring-system</a><br>
<br>
Przemek<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:OpenStack-dev@lists.openstack.org"
target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:OpenStack-dev@lists.openstack.org"
target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div>
<div dir="ltr">
<div>Dmitry Borodaenko</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
OpenStack-dev mailing list
<a moz-do-not-send="true" href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a>
<a moz-do-not-send="true" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>