Open Stack

Just FYI.

Docker support multi log driver now[0].

Version New Logging Drivers
1.6.0 json-file, syslog, none
1.7.0 journald
1.8.0 fluentd, gelf
1.9.0 awslogs

For the stdout from container, we can use gelf/syslog driver to forward.

[0] https://docs.docker.com/engine/reference/logging/overview/

On Fri, Jan 15, 2016 at 8:28 AM, Steven Dake (stdake) <stdake at cisco.com>
wrote:

> Eric,
>
> Comments inline.
>
> On 1/14/16, 3:31 AM, "Eric LEMOINE" <elemoine at mirantis.com> wrote:
>
> >On Wed, Jan 13, 2016 at 1:27 PM, Steven Dake (stdake) <stdake at cisco.com>
> >wrote:
> >> Eric,
> >
> >
> >Hi Steven
>
>
> Feel free to call me Steve
>
> >
> >
> >>
> >> Apologies for top post, not really sure where in this thread to post
> >>this
> >> list of questions as its sort of a change in topic so I changed the
> >> subject line :)
> >>
> >> 1.
> >> Somewhere I read when researching this Heka topic, that Heka cannot log
> >> all details from /dev/log.  Some services like mariadb for example don't
> >> log to stdout as I think Heka requires to operate correctly.  Would you
> >> mind responding on the question "Would Heka be able to effectively log
> >> every piece of information coming off the system related to OpenStack
> >>(our
> >> infrastructure services like ceph/mariadb/etc as well as the OpenStack
> >> services)?
> >
> >
> >My first reaction to this is: if we have services, such as mariadb,
> >that can only send their logs to syslog then let's continue using
> >Rsyslog.  And with Rsyslog we can easily store logs on the local
> >filesystem as well (your requirement #3 below).
>
> Rsyslog may be the best tool for this job.  I'll be looking to your POC to
> get a feel for the validity of that claim :)
>
> >
> >That being said, it appears that Heka supports reading logs from
> >/dev/log.  This can be done using the UdpInput plugin with "net" set
> >to "unixgram".  See
> ><https://github.com/mozilla-services/heka/issues/790> for the original
> >issue.  Heka also supports writing logs to files on the local
> >filesystem, through the FileOutput plugin.  We do not currently use
> >the UdpInput plugin, so we need to test it and see if it can work for
> >Kolla.  We will work on these tests, and report back to the list.
> >
> >
> >
> >> 2.
> >> Also, I want to make sure we can fix up the backtrace defeciency.
> >> Currently rsyslog doesn't log backtraces in python code.  Perhaps Sam or
> >> inc0 know the reason behind it, but I want to make sure we can fix up
> >>this
> >> annoyance, because backtraces are mightily important.
> >
> >
> >I've had a look on my AIO Kolla.  And I do see Python tracebacks in
> >OpenStack log files created by Rsyslog (in
> >/var/lib/docker/volumes/rsyslog/_data/nova/nova-api.log for example).
> >They're just on a single line, with "#012" used as the separator [*].
> >So they are hard to read, but they are there.  I think that is
> >consistent with what SamYaple and inc0 said yesterday on IRC.
> >
> >[*] This is related to Rsyslog's $EscapeControlCharactersOnReceive
> >setting. See
> ><
> http://www.rsyslog.com/doc/master/configuration/input_directives/rsconf1_
> >escapecontrolcharactersonreceive.html>.
> >
>
> Apparently the one-lining of the tracebacks is a new feature because of a
> bug fix in oslo.log that dims fixed for us.  So my original comments about
> missing backtraces was dated information from December.
>
> >
> >> 3.
> >> Also I want to make sure each node ends up with log files in a data
> >> container (or data volume or whatever we just recently replaced the data
> >> containers with) for all the services for individual node diagnostics.
> >> This helps fill the gap of the Kibana visualization and Elasticsearch
> >> where we may not have a perfect diagnostic solution at the conclusion of
> >> Mitaka and in need of individual node inspection of the logs.  Can Heka
> >>be
> >> made to do this?  Our rsyslog implementation does today, and its a hard
> >> requirement for the moment.  If we need some special software to run in
> >> addition to Heka, I could live with that.
> >
> >
> >That "special software" could be Rsyslog :)  Seriously, Rsyslog
> >provides a solution for getting logs from services that only log to
> >syslog.  We can also easily configure Rsyslog to write logs on the
> >local filesystem, as done in Kolla already today.  And using Heka we
> >can easily make Python tracebacks look good in Kibana.
> >
> >I would like to point out that our initial intent was not to remove
> >Rsyslog.  Our intent was to propose a scalable/decentralized log
> >processing architecture based on Heka running on each node, instead of
> >relying on a centralized Logstash instance.  Using Heka we eliminate
> >the need to deploy and manage a resilient Logstash/Redis cluster.  And
> >it is to be noted that Heka gives us a lot of flexibility.  In
> >particular, Heka makes it possible to collect logs from services that
> >don't speak syslog (RabbitMQ for example, whose logs are not currently
> >collected!).
>
>
> FWIW the concern that logstash isn't scalable is unsubstantiated.  IMNSHO
> the reason to use Heka is it does two things we need with one component
> while removing JVMs from each compute node.  This is my main attraction to
> the adoption of Heka.
>
> >
> >As mentioned above Heka provides plugins that we could possibly
> >leverage to remove Rsyslog completely, but at this point we cannot
> >guarantee that they will do the job.  Our coming tests will tell.
>
> If the plugins work in a way in which we can remove rsyslog completely, I
> would prefer that outcome.  Less dependencies in software systems are
> better.
>
> Regards
> -steve
>
> >
> >Thanks.
> >
> >__________________________________________________________________________
> >OpenStack Development Mailing List (not for usage questions)
> >Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Jeffrey Zhang
Blog: http://xcodest.me
twitter/weibo: @jeffrey4l
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160125/780c2552/attachment.html>