Open Stack

On Wed, Jan 13, 2016 at 1:27 PM, Steven Dake (stdake) <stdake at cisco.com> wrote:
> Eric,


Hi Steven


>
> Apologies for top post, not really sure where in this thread to post this
> list of questions as its sort of a change in topic so I changed the
> subject line :)
>
> 1.
> Somewhere I read when researching this Heka topic, that Heka cannot log
> all details from /dev/log.  Some services like mariadb for example don't
> log to stdout as I think Heka requires to operate correctly.  Would you
> mind responding on the question "Would Heka be able to effectively log
> every piece of information coming off the system related to OpenStack (our
> infrastructure services like ceph/mariadb/etc as well as the OpenStack
> services)?


My first reaction to this is: if we have services, such as mariadb,
that can only send their logs to syslog then let's continue using
Rsyslog.  And with Rsyslog we can easily store logs on the local
filesystem as well (your requirement #3 below).

That being said, it appears that Heka supports reading logs from
/dev/log.  This can be done using the UdpInput plugin with "net" set
to "unixgram".  See
<https://github.com/mozilla-services/heka/issues/790> for the original
issue.  Heka also supports writing logs to files on the local
filesystem, through the FileOutput plugin.  We do not currently use
the UdpInput plugin, so we need to test it and see if it can work for
Kolla.  We will work on these tests, and report back to the list.



> 2.
> Also, I want to make sure we can fix up the backtrace defeciency.
> Currently rsyslog doesn't log backtraces in python code.  Perhaps Sam or
> inc0 know the reason behind it, but I want to make sure we can fix up this
> annoyance, because backtraces are mightily important.


I've had a look on my AIO Kolla.  And I do see Python tracebacks in
OpenStack log files created by Rsyslog (in
/var/lib/docker/volumes/rsyslog/_data/nova/nova-api.log for example).
They're just on a single line, with "#012" used as the separator [*].
So they are hard to read, but they are there.  I think that is
consistent with what SamYaple and inc0 said yesterday on IRC.

[*] This is related to Rsyslog's $EscapeControlCharactersOnReceive
setting. See <http://www.rsyslog.com/doc/master/configuration/input_directives/rsconf1_escapecontrolcharactersonreceive.html>.


> 3.
> Also I want to make sure each node ends up with log files in a data
> container (or data volume or whatever we just recently replaced the data
> containers with) for all the services for individual node diagnostics.
> This helps fill the gap of the Kibana visualization and Elasticsearch
> where we may not have a perfect diagnostic solution at the conclusion of
> Mitaka and in need of individual node inspection of the logs.  Can Heka be
> made to do this?  Our rsyslog implementation does today, and its a hard
> requirement for the moment.  If we need some special software to run in
> addition to Heka, I could live with that.


That "special software" could be Rsyslog :)  Seriously, Rsyslog
provides a solution for getting logs from services that only log to
syslog.  We can also easily configure Rsyslog to write logs on the
local filesystem, as done in Kolla already today.  And using Heka we
can easily make Python tracebacks look good in Kibana.

I would like to point out that our initial intent was not to remove
Rsyslog.  Our intent was to propose a scalable/decentralized log
processing architecture based on Heka running on each node, instead of
relying on a centralized Logstash instance.  Using Heka we eliminate
the need to deploy and manage a resilient Logstash/Redis cluster.  And
it is to be noted that Heka gives us a lot of flexibility.  In
particular, Heka makes it possible to collect logs from services that
don't speak syslog (RabbitMQ for example, whose logs are not currently
collected!).

As mentioned above Heka provides plugins that we could possibly
leverage to remove Rsyslog completely, but at this point we cannot
guarantee that they will do the job.  Our coming tests will tell.

Thanks.