Open Stack

k8s's offical way of dealing with logs is to ensure use of the docker json logger, not the journald one. then all the k8s log shippers have a standard way to gather the logs. Docker supports log rotation and other options too. seems to work out pretty well in practice.

log shipping with other cri drivers such as containerd seems to work well too.

Thanks,
Kevin
________________________________________
From: Sean Mooney [smooney at redhat.com]
Sent: Wednesday, January 30, 2019 8:23 AM
To: Emilien Macchi; Juan Antonio Osorio Robles
Cc: openstack-discuss at lists.openstack.org
Subject: Re: [TripleO] containers logging to stdout

On Wed, 2019-01-30 at 07:37 -0500, Emilien Macchi wrote:
>
>
> On Wed, Jan 30, 2019 at 5:53 AM Juan Antonio Osorio Robles <jaosorior at redhat.com> wrote:
> > Hello!
> >
> >
> > In Queens, the a spec to provide the option to make containers log to
> > standard output was proposed [1] [2]. Some work was done on that side,
> > but due to the lack of traction, it wasn't completed. With the Train
> > release coming, I think it would be a good idea to revive this effort,
> > but make logging to stdout the default in that release.
> >
> > This would allow several benefits:
> >
> > * All logging from the containers would en up in journald; this would
> > make it easier for us to forward the logs, instead of having to keep
> > track of the different directories in /var/log/containers
> >
> > * The journald driver would add metadata to the logs about the container
> > (we would automatically get what container ID issued the logs).
> >
> > * This wouldo also simplify the stacks (removing the Logging nested
> > stack which is present in several templates).
> >
> > * Finally... if at some point we move towards kubernetes (or something
> > in between), managing our containers, it would work with their logging
> > tooling as well
>
> Also, I would add that it'll be aligned with what we did for Paunch-managed containers (with Podman backend) where
> each ("long life") container has its own SystemD service (+ SystemD timer sometimes); so using journald makes total
> sense to me.
one thing to keep in mind is that journald apparently has rate limiting so if you contaiern are very verbose journald
will actully slowdown the execution of the contaienr application as it slows down the rate at wich it can log.
this came form a downstream conversation on irc were they were recommending that such applciation bypass journald and
log to a file for best performacne.
> --
> Emilien Macchi