<html>
  <head>
    <meta content="text/html; charset=windows-1252"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    I think we're missing the point here. What I meant adding a simple
    monitoring system that informed the user via UI/CLI/email/whatever
    of low resources on fuel master node. That's it. HA here is not an
    option -- if, despite of warnings, the user still continues to use
    fuel and disk becomes full, it's the user's fault. By adding these
    warnings we have a way of saying "We told you so!" Without warnings
    we get bugs like [1] I mentioned in the first post.<br>
    <br>
    Of course user can check disk space by hand but since we do have a
    full-blown UI telling the user to periodically log in to the console
    and check disks by hand seems a bit of a burden.<br>
    <br>
    We can even implement such monitoring functionality as a Nailgun
    plugin -- installing it would be optional and at the same time we
    would grow our plugin ecosystem.<br>
    <br>
    P.<br>
    <br>
    <div class="moz-cite-prefix">On 11/05/2014 08:42 PM, Dmitry
      Borodaenko wrote:<br>
    </div>
    <blockquote
cite="mid:CAM0pNLMM5PAV9EyeR30EZtEJDJPiUstSHq6vi7EJNU3jrbofjw@mail.gmail.com"
      type="cite">
      <div dir="ltr">Even one additional hardware node required to host
        the Fuel master is seen by many users as excessive. Unless you
        can come up with an architecture that adds HA capability to Fuel
        without increasing its hardware footprint by 2 more nodes, it's
        just not worth it.<br>
        <br>
        The only operational aspect of the Fuel master node that you
        don't want to lose even for a short while is logging. You'd be
        better off redirecting OpenStack environments' logs to a
        dedicated highly available logging server (which, of course, you
        already have in your environment), and deal with Fuel master
        node failures by restoring it from backups.<br>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">On Wed, Nov 5, 2014 at 8:26 AM, Anton
          Zemlyanov <span dir="ltr"><<a moz-do-not-send="true"
              href="mailto:azemlyanov@mirantis.com" target="_blank">azemlyanov@mirantis.com</a>></span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div dir="ltr">
              <div class="gmail_extra">Monitoring of the Fuel master's
                disk space is the special case. I really wonder why Fuel
                master have no HA option, disk overflow can be predicted
                but many other failures cannot. HA is a solution of the
                'single point of failure' problem.</div>
              <div class="gmail_extra"><br>
              </div>
              <div class="gmail_extra">The current monitoring
                recommendations (<a moz-do-not-send="true"
href="http://docs.openstack.org/openstack-ops/content/logging_monitoring.html"
                  target="_blank">http://docs.openstack.org/openstack-ops/content/logging_monitoring.html</a>)
                are based on analyzing logs and manual checks, that are
                rather reactive way of fixing problems. Zabbix is quite
                good for preventing failures that are predictable but
                for the abrupt problems Zabbix just reports them 'post
                mortem'.</div>
              <div class="gmail_extra"><br>
              </div>
              <div class="gmail_extra">The only way to remove the single
                failure point is to implement redundancy/HA</div>
              <span class="HOEnZb"><font color="#888888">
                  <div class="gmail_extra"><br>
                  </div>
                  <div class="gmail_extra">Anton</div>
                </font></span>
              <div>
                <div class="h5">
                  <div class="gmail_extra"><br>
                    <div class="gmail_quote">On Tue, Nov 4, 2014 at 6:26
                      PM, Przemyslaw Kaminski <span dir="ltr"><<a
                          moz-do-not-send="true"
                          href="mailto:pkaminski@mirantis.com"
                          target="_blank">pkaminski@mirantis.com</a>></span>
                      wrote:<br>
                      <blockquote class="gmail_quote" style="margin:0px
                        0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hello,<br>
                        <br>
                        In extension to my comment in this bug [1] I'd
                        like to discuss the possibility of adding Fuel
                        master node monitoring. As I wrote in the
                        comment, when disk is full it might be already
                        too late to perform any action since for example
                        Nailgun could be down because DB shut itself
                        down. So we should somehow warn the user that
                        disk is running low (in the UI and fuel CLI on
                        stderr for example) before it actually happens.<br>
                        <br>
                        For now the only meaningful value to monitor
                        would be disk usage -- do you have other
                        suggestions? If not then probably a simple API
                        endpoint with statvfs calls would suffice. If
                        you see other usages of this then maybe it would
                        be better to have some daemon collecting the
                        stats we want.<br>
                        <br>
                        If we opted for a daemon, then I'm aware that
                        the user can optionally install Zabbix server
                        although looking at blueprints in [2] I don't
                        see anything about monitoring Fuel master itself
                        -- is it possible to do? Though the installation
                        of Zabbix though is not mandatory so it still
                        doesn't completely solve the problem.<br>
                        <br>
                        [1] <a moz-do-not-send="true"
                          href="https://bugs.launchpad.net/fuel/+bug/1371757"
                          target="_blank">https://bugs.launchpad.net/fuel/+bug/1371757</a><br>
                        [2] <a moz-do-not-send="true"
                          href="https://blueprints.launchpad.net/fuel/+spec/monitoring-system"
                          target="_blank">https://blueprints.launchpad.net/fuel/+spec/monitoring-system</a><br>
                        <br>
                        Przemek<br>
                        <br>
                        _______________________________________________<br>
                        OpenStack-dev mailing list<br>
                        <a moz-do-not-send="true"
                          href="mailto:OpenStack-dev@lists.openstack.org"
                          target="_blank">OpenStack-dev@lists.openstack.org</a><br>
                        <a moz-do-not-send="true"
                          href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
                          target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
                      </blockquote>
                    </div>
                    <br>
                  </div>
                </div>
              </div>
            </div>
            <br>
            _______________________________________________<br>
            OpenStack-dev mailing list<br>
            <a moz-do-not-send="true"
              href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
            <a moz-do-not-send="true"
              href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
              target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
            <br>
          </blockquote>
        </div>
        <br>
        <br clear="all">
        <br>
        -- <br>
        <div class="gmail_signature">
          <div dir="ltr">
            <div>Dmitry Borodaenko</div>
          </div>
        </div>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
    </blockquote>
    <br>
  </body>
</html>