On 1/30/19 6:26 PM, Fox, Kevin M wrote:
k8s's offical way of dealing with logs is to ensure use of the docker json logger, not the journald one. then all the k8s log shippers have a standard way to gather the logs. Docker supports log rotation and other options too. seems to work out pretty well in practice. sending directly to a file looks a good option indeed. Journald and (r)syslog have both some throttle issue, and it might create some issues in case of service restarting and the like.
Pushing logs directly from the container engine (podman does actually support the same options) might be the way to go.
As long as we have a common, easy way to output the logs, it's all for the best. The only concern I have with the "not-journald" path is the possible lack of "journalctl -f CONTAINER_NAME=foo". But, compared to the risks exposed in this thread about the possible crash if journald isn't available, and throttling, I think it's fine.
Also, a small note regarding "log re-shipping": some people might want to push their logs to some elk/kelk/others - pushing the logs directly as json in plain files might help a log for that, as (r)syslog can then read them (and there, no bottleneck with throttle) and send it in the proper format to the remote logging infra.
Soooo... yeah. imho the "direct writing as json" might be the way to go :). That is just fine IMO. the runtime engine usually allows you to configure the logging driver (docker in CentOS defaults... or used to default, to journald); but if we find out that file is a better choice;
On 2/1/19 8:44 AM, Cédric Jeanneret wrote: that's entirely fine. The whole point is to let the runtime engine do its job, and handle the logging with the driver.
log shipping with other cri drivers such as containerd seems to work well too. Not tested yet, but at least podman has the option (as a work for this engine integration is done).
Cheers,
C.
Thanks, Kevin ________________________________________ From: Sean Mooney [smooney@redhat.com] Sent: Wednesday, January 30, 2019 8:23 AM To: Emilien Macchi; Juan Antonio Osorio Robles Cc: openstack-discuss@lists.openstack.org Subject: Re: [TripleO] containers logging to stdout
On Wed, 2019-01-30 at 07:37 -0500, Emilien Macchi wrote:
On Wed, Jan 30, 2019 at 5:53 AM Juan Antonio Osorio Robles <jaosorior@redhat.com> wrote:
Hello!
In Queens, the a spec to provide the option to make containers log to standard output was proposed [1] [2]. Some work was done on that side, but due to the lack of traction, it wasn't completed. With the Train release coming, I think it would be a good idea to revive this effort, but make logging to stdout the default in that release.
This would allow several benefits:
* All logging from the containers would en up in journald; this would make it easier for us to forward the logs, instead of having to keep track of the different directories in /var/log/containers
* The journald driver would add metadata to the logs about the container (we would automatically get what container ID issued the logs).
* This wouldo also simplify the stacks (removing the Logging nested stack which is present in several templates).
* Finally... if at some point we move towards kubernetes (or something in between), managing our containers, it would work with their logging tooling as well Also, I would add that it'll be aligned with what we did for Paunch-managed containers (with Podman backend) where each ("long life") container has its own SystemD service (+ SystemD timer sometimes); so using journald makes total sense to me.
one thing to keep in mind is that journald apparently has rate limiting so if you contaiern are very verbose journald will actully slowdown the execution of the contaienr application as it slows down the rate at wich it can log. this came form a downstream conversation on irc were they were recommending that such applciation bypass journald and log to a file for best performacne.
-- Emilien Macchi