Open Stack

So tracebacks sort of works, they're there just ugly. That's why I'm
also happy if we change rsyslog to heka.

Eric, I hope I wont ask too much, but could you please prepare PoC of
kolla+heka, for what I care heka can log to local log file same as
rsyslog for now. Would that be big problem?

On 11 January 2016 at 16:37, Sam Yaple <samuel at yaple.net> wrote:
> Here is why I am on board with this. As we have discovered, the logging with
> the syslog plugin leaves alot to be desired. It (to my understanding) still
> can't save tracebacks/stacktraces to the log files for whatever reason.
> stdout/stderr however works perfectly fine. That said the Docker log stuff
> has been a source of pain in the past, but it has gotten better. It does
> have the limitation of being only able to log one output at a time. This
> means, as an example, the neutron-dhcp-agent could send its logs to
> stdout/err but the dnsmasq process that it launch (that also has logs) would
> have to mix its logs in with the neutron logs in stdout/err. Can Heka handle
> this and separate them efficiently? Otherwise I see no choice but to stick
> with something that can handle multiple logs from a single container.
>
> Sam Yaple
>
> On Mon, Jan 11, 2016 at 10:16 PM, Eric LEMOINE <elemoine at mirantis.com>
> wrote:
>>
>>
>> Le 11 janv. 2016 18:45, "Michał Jastrzębski" <inc007 at gmail.com> a écrit :
>> >
>> > On 11 January 2016 at 10:55, Eric LEMOINE <elemoine at mirantis.com> wrote:
>> > > Currently the services running in containers send their logs to
>> > > rsyslog. And rsyslog stores the logs in local files, located in the
>> > > host's /var/log directory.
>> >
>> > Yeah, however plan was to teach rsyslog to forward logs to central
>> > logging stack once this thing is implemented.
>>
>> Yes. With the current ELK Change Request, Rsyslog sends logs to the
>> central Logstash instance. If you read my design doc you'll understand that
>> it's precisely what we're proposing changing.
>>
>> > > I know. Our plan is to rely on Docker. Basically: containers write
>> > > their logs to stdout. The logs are collected by Docker Engine, which
>> > > makes them available through the unix:///var/run/docker.sock socket.
>> > > The socket is mounted into the Heka container, which uses the Docker
>> > > Log Input plugin [*] to reads the logs from that that socket.
>> > >
>> > > [*]
>> > > <http://hekad.readthedocs.org/en/latest/config/inputs/docker_log.html>
>> >
>> > So docker logs isn't best thing there is, however I'd suspect that's
>> > mostly console output fault. If you can tap into stdout efficiently,
>> > I'd say that's pretty good option.
>>
>> I'm not following you. Could you please be more specific?
>>
>> > >> Seems to me we need additional comparason of heka vs rsyslog;) Also
>> > >> this would have to be hands down better because rsyslog is already
>> > >> implemented, working and most of operators knows how to use it.
>> > >
>> > >
>> > > We don't need to remove Rsyslog. Services running in containers can
>> > > write their logs to both Rsyslog and stdout, which even is what they
>> > > do today (at least for the OpenStack services).
>> > >
>> >
>> > There is no point for that imho. I don't want to have several systems
>> > doing the same thing. Let's make decision of one, but optimal toolset.
>> > Could you please describe bottoms up what would your logging stack
>> > look like? Heka listening on stdout, transfering stuff to
>> > elasticsearch and kibana on top of it?
>>
>> My plan is to provide details in the blueprint document, that I'll
>> continue working on if the core developers agree with the principles of the
>> proposed architecture and change.
>>
>> But here's our plan—as already described in my previous email: the Kolla
>> services, which run in containers, write their logs to stdout. Logs are
>> collected by the Docker engine. Heka's Docker Log Input plugin is used to
>> read the container logs from the Docker endpoint (Unix socket). Since Heka
>> will run in a container a volume is necessary for accessing the Docker
>> endpoint. The Docker Log Input plugin inserts the logs into the Heka
>> pipeline, at the end of which an Elasticsearch Output plugin will send the
>> log messages to Elasticsearch. Here's a blog post reporting on that
>> approach:
>> <http://www.ianneubert.com/wp/2015/03/03/how-to-use-heka-docker-and-tutum/>.
>> We haven't tested that approach yet, but we plan to experiment with it as we
>> work on the specs.
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>