Open Stack

Eric,

Comments inline.

On 1/14/16, 3:31 AM, "Eric LEMOINE" <elemoine at mirantis.com> wrote:

>On Wed, Jan 13, 2016 at 1:27 PM, Steven Dake (stdake) <stdake at cisco.com>
>wrote:
>> Eric,
>
>
>Hi Steven


Feel free to call me Steve

>
>
>>
>> Apologies for top post, not really sure where in this thread to post
>>this
>> list of questions as its sort of a change in topic so I changed the
>> subject line :)
>>
>> 1.
>> Somewhere I read when researching this Heka topic, that Heka cannot log
>> all details from /dev/log.  Some services like mariadb for example don't
>> log to stdout as I think Heka requires to operate correctly.  Would you
>> mind responding on the question "Would Heka be able to effectively log
>> every piece of information coming off the system related to OpenStack
>>(our
>> infrastructure services like ceph/mariadb/etc as well as the OpenStack
>> services)?
>
>
>My first reaction to this is: if we have services, such as mariadb,
>that can only send their logs to syslog then let's continue using
>Rsyslog.  And with Rsyslog we can easily store logs on the local
>filesystem as well (your requirement #3 below).

Rsyslog may be the best tool for this job.  I'll be looking to your POC to
get a feel for the validity of that claim :)

>
>That being said, it appears that Heka supports reading logs from
>/dev/log.  This can be done using the UdpInput plugin with "net" set
>to "unixgram".  See
><https://github.com/mozilla-services/heka/issues/790> for the original
>issue.  Heka also supports writing logs to files on the local
>filesystem, through the FileOutput plugin.  We do not currently use
>the UdpInput plugin, so we need to test it and see if it can work for
>Kolla.  We will work on these tests, and report back to the list.
>
>
>
>> 2.
>> Also, I want to make sure we can fix up the backtrace defeciency.
>> Currently rsyslog doesn't log backtraces in python code.  Perhaps Sam or
>> inc0 know the reason behind it, but I want to make sure we can fix up
>>this
>> annoyance, because backtraces are mightily important.
>
>
>I've had a look on my AIO Kolla.  And I do see Python tracebacks in
>OpenStack log files created by Rsyslog (in
>/var/lib/docker/volumes/rsyslog/_data/nova/nova-api.log for example).
>They're just on a single line, with "#012" used as the separator [*].
>So they are hard to read, but they are there.  I think that is
>consistent with what SamYaple and inc0 said yesterday on IRC.
>
>[*] This is related to Rsyslog's $EscapeControlCharactersOnReceive
>setting. See 
><http://www.rsyslog.com/doc/master/configuration/input_directives/rsconf1_
>escapecontrolcharactersonreceive.html>.
>

Apparently the one-lining of the tracebacks is a new feature because of a
bug fix in oslo.log that dims fixed for us.  So my original comments about
missing backtraces was dated information from December.

>
>> 3.
>> Also I want to make sure each node ends up with log files in a data
>> container (or data volume or whatever we just recently replaced the data
>> containers with) for all the services for individual node diagnostics.
>> This helps fill the gap of the Kibana visualization and Elasticsearch
>> where we may not have a perfect diagnostic solution at the conclusion of
>> Mitaka and in need of individual node inspection of the logs.  Can Heka
>>be
>> made to do this?  Our rsyslog implementation does today, and its a hard
>> requirement for the moment.  If we need some special software to run in
>> addition to Heka, I could live with that.
>
>
>That "special software" could be Rsyslog :)  Seriously, Rsyslog
>provides a solution for getting logs from services that only log to
>syslog.  We can also easily configure Rsyslog to write logs on the
>local filesystem, as done in Kolla already today.  And using Heka we
>can easily make Python tracebacks look good in Kibana.
>
>I would like to point out that our initial intent was not to remove
>Rsyslog.  Our intent was to propose a scalable/decentralized log
>processing architecture based on Heka running on each node, instead of
>relying on a centralized Logstash instance.  Using Heka we eliminate
>the need to deploy and manage a resilient Logstash/Redis cluster.  And
>it is to be noted that Heka gives us a lot of flexibility.  In
>particular, Heka makes it possible to collect logs from services that
>don't speak syslog (RabbitMQ for example, whose logs are not currently
>collected!).


FWIW the concern that logstash isn't scalable is unsubstantiated.  IMNSHO
the reason to use Heka is it does two things we need with one component
while removing JVMs from each compute node.  This is my main attraction to
the adoption of Heka.

>
>As mentioned above Heka provides plugins that we could possibly
>leverage to remove Rsyslog completely, but at this point we cannot
>guarantee that they will do the job.  Our coming tests will tell.

If the plugins work in a way in which we can remove rsyslog completely, I
would prefer that outcome.  Less dependencies in software systems are
better.

Regards
-steve

>
>Thanks.
>
>__________________________________________________________________________
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev