<div dir="ltr">We can add a notification to FuelWeb, no additional software or user actions are required. I would not overestimate this method though, it is in no way the robust monitoring system. Forcing user to do something on a regular basis is unlikely to work. <div><div><br></div><div>Anton</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 6, 2014 at 11:55 AM, Przemyslaw Kaminski <span dir="ltr"><<a href="mailto:pkaminski@mirantis.com" target="_blank">pkaminski@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
I think we're missing the point here. What I meant adding a simple
monitoring system that informed the user via UI/CLI/email/whatever
of low resources on fuel master node. That's it. HA here is not an
option -- if, despite of warnings, the user still continues to use
fuel and disk becomes full, it's the user's fault. By adding these
warnings we have a way of saying "We told you so!" Without warnings
we get bugs like [1] I mentioned in the first post.<br>
<br>
Of course user can check disk space by hand but since we do have a
full-blown UI telling the user to periodically log in to the console
and check disks by hand seems a bit of a burden.<br>
<br>
We can even implement such monitoring functionality as a Nailgun
plugin -- installing it would be optional and at the same time we
would grow our plugin ecosystem.<span class="HOEnZb"><font color="#888888"><br>
<br>
P.</font></span><div><div class="h5"><br>
<br>
<div>On 11/05/2014 08:42 PM, Dmitry
Borodaenko wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Even one additional hardware node required to host
the Fuel master is seen by many users as excessive. Unless you
can come up with an architecture that adds HA capability to Fuel
without increasing its hardware footprint by 2 more nodes, it's
just not worth it.<br>
<br>
The only operational aspect of the Fuel master node that you
don't want to lose even for a short while is logging. You'd be
better off redirecting OpenStack environments' logs to a
dedicated highly available logging server (which, of course, you
already have in your environment), and deal with Fuel master
node failures by restoring it from backups.<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Nov 5, 2014 at 8:26 AM, Anton
Zemlyanov <span dir="ltr"><<a href="mailto:azemlyanov@mirantis.com" target="_blank">azemlyanov@mirantis.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">Monitoring of the Fuel master's
disk space is the special case. I really wonder why Fuel
master have no HA option, disk overflow can be predicted
but many other failures cannot. HA is a solution of the
'single point of failure' problem.</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">The current monitoring
recommendations (<a href="http://docs.openstack.org/openstack-ops/content/logging_monitoring.html" target="_blank">http://docs.openstack.org/openstack-ops/content/logging_monitoring.html</a>)
are based on analyzing logs and manual checks, that are
rather reactive way of fixing problems. Zabbix is quite
good for preventing failures that are predictable but
for the abrupt problems Zabbix just reports them 'post
mortem'.</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">The only way to remove the single
failure point is to implement redundancy/HA</div>
<span><font color="#888888">
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">Anton</div>
</font></span>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Nov 4, 2014 at 6:26
PM, Przemyslaw Kaminski <span dir="ltr"><<a href="mailto:pkaminski@mirantis.com" target="_blank">pkaminski@mirantis.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hello,<br>
<br>
In extension to my comment in this bug [1] I'd
like to discuss the possibility of adding Fuel
master node monitoring. As I wrote in the
comment, when disk is full it might be already
too late to perform any action since for example
Nailgun could be down because DB shut itself
down. So we should somehow warn the user that
disk is running low (in the UI and fuel CLI on
stderr for example) before it actually happens.<br>
<br>
For now the only meaningful value to monitor
would be disk usage -- do you have other
suggestions? If not then probably a simple API
endpoint with statvfs calls would suffice. If
you see other usages of this then maybe it would
be better to have some daemon collecting the
stats we want.<br>
<br>
If we opted for a daemon, then I'm aware that
the user can optionally install Zabbix server
although looking at blueprints in [2] I don't
see anything about monitoring Fuel master itself
-- is it possible to do? Though the installation
of Zabbix though is not mandatory so it still
doesn't completely solve the problem.<br>
<br>
[1] <a href="https://bugs.launchpad.net/fuel/+bug/1371757" target="_blank">https://bugs.launchpad.net/fuel/+bug/1371757</a><br>
[2] <a href="https://blueprints.launchpad.net/fuel/+spec/monitoring-system" target="_blank">https://blueprints.launchpad.net/fuel/+spec/monitoring-system</a><br>
<br>
Przemek<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div>
<div dir="ltr">
<div>Dmitry Borodaenko</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
OpenStack-dev mailing list
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</div></div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>