Open Stack

On 11 January 2016 at 10:55, Eric LEMOINE <elemoine at mirantis.com> wrote:
> On Mon, Jan 11, 2016 at 5:01 PM, Michał Jastrzębski <inc007 at gmail.com> wrote:
>> On 11 January 2016 at 09:16, Eric LEMOINE <elemoine at mirantis.com> wrote:
>
>>> * Logstash was designed for logs processing.  Heka is a "unified data
>>> processing" software, designed to process streams of any type of data.
>>> So Heka is about running one service on each box instead of many.
>>> Using a single service for processing different types of data also
>>> makes it possible to do correlations, and derive metrics from logs and
>>> events.  See Rob Miller's presentation [4] for more details.
>>
>> Right now we use rsyslog for that.
>
>
> Currently the services running in containers send their logs to
> rsyslog. And rsyslog stores the logs in local files, located in the
> host's /var/log directory.

Yeah, however plan was to teach rsyslog to forward logs to central
logging stack once this thing is implemented.

>> As I understand Heka right now
>> would be actually alternative to rsyslog, and that is already
>> implemented. Also with Heka case, we might run into same problem we've
>> seen with rsyslog - transporting logs from service to heka. Keep in
>> mind we're in dockers and heka will be in different container than
>> service it's supposed to listen to. We do that by sharing faked
>> /dev/log across containers and rsyslog can handle that.
>
>
> I know. Our plan is to rely on Docker. Basically: containers write
> their logs to stdout. The logs are collected by Docker Engine, which
> makes them available through the unix:///var/run/docker.sock socket.
> The socket is mounted into the Heka container, which uses the Docker
> Log Input plugin [*] to reads the logs from that that socket.
>
> [*] <http://hekad.readthedocs.org/en/latest/config/inputs/docker_log.html>

So docker logs isn't best thing there is, however I'd suspect that's
mostly console output fault. If you can tap into stdout efficiently,
I'd say that's pretty good option.

>
>> Also Heka seems to be just processing mechanism while logstash is
>> well...stash, so it saves and persists logs, so it seems to me they're
>> different layers of log processing.
>
> No. Logstash typically stores the logs in Elasticsearch. And we'd do
> the same with Heka.
>
>
>>
>> Seems to me we need additional comparason of heka vs rsyslog;) Also
>> this would have to be hands down better because rsyslog is already
>> implemented, working and most of operators knows how to use it.
>
>
> We don't need to remove Rsyslog. Services running in containers can
> write their logs to both Rsyslog and stdout, which even is what they
> do today (at least for the OpenStack services).
>

There is no point for that imho. I don't want to have several systems
doing the same thing. Let's make decision of one, but optimal toolset.
Could you please describe bottoms up what would your logging stack
look like? Heka listening on stdout, transfering stuff to
elasticsearch and kibana on top of it?

> Hope that makes sense!
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev