k8s's offical way of dealing with logs is to ensure use of the docker json logger, not the journald one. then all the k8s log shippers have a standard way to gather the logs. Docker supports log rotation and other options too. seems to work out pretty well in practice.
log shipping with other cri drivers such as containerd seems to work well too.
Thanks, Kevin ________________________________________ From: Sean Mooney [smooney@redhat.com] Sent: Wednesday, January 30, 2019 8:23 AM To: Emilien Macchi; Juan Antonio Osorio Robles Cc: openstack-discuss@lists.openstack.org Subject: Re: [TripleO] containers logging to stdout
On Wed, 2019-01-30 at 07:37 -0500, Emilien Macchi wrote:
On Wed, Jan 30, 2019 at 5:53 AM Juan Antonio Osorio Robles jaosorior@redhat.com wrote:
Hello!
In Queens, the a spec to provide the option to make containers log to standard output was proposed [1] [2]. Some work was done on that side, but due to the lack of traction, it wasn't completed. With the Train release coming, I think it would be a good idea to revive this effort, but make logging to stdout the default in that release.
This would allow several benefits:
- All logging from the containers would en up in journald; this would
make it easier for us to forward the logs, instead of having to keep track of the different directories in /var/log/containers
- The journald driver would add metadata to the logs about the container
(we would automatically get what container ID issued the logs).
- This wouldo also simplify the stacks (removing the Logging nested
stack which is present in several templates).
- Finally... if at some point we move towards kubernetes (or something
in between), managing our containers, it would work with their logging tooling as well
Also, I would add that it'll be aligned with what we did for Paunch-managed containers (with Podman backend) where each ("long life") container has its own SystemD service (+ SystemD timer sometimes); so using journald makes total sense to me.
one thing to keep in mind is that journald apparently has rate limiting so if you contaiern are very verbose journald will actully slowdown the execution of the contaienr application as it slows down the rate at wich it can log. this came form a downstream conversation on irc were they were recommending that such applciation bypass journald and log to a file for best performacne.
-- Emilien Macchi