Open Stack

On 1/30/19 6:26 PM, Fox, Kevin M wrote:
> k8s's offical way of dealing with logs is to ensure use of the docker json logger, not the journald one. then all the k8s log shippers have a standard way to gather the logs. Docker supports log rotation and other options too. seems to work out pretty well in practice.

sending directly to a file looks a good option indeed. Journald and
(r)syslog have both some throttle issue, and it might create some issues
in case of service restarting and the like.

Pushing logs directly from the container engine (podman does actually
support the same options) might be the way to go.

As long as we have a common, easy way to output the logs, it's all for
the best.
The only concern I have with the "not-journald" path is the possible
lack of "journalctl -f CONTAINER_NAME=foo". But, compared to the risks
exposed in this thread about the possible crash if journald isn't
available, and throttling, I think it's fine.

Also, a small note regarding "log re-shipping": some people might want
to push their logs to some elk/kelk/others - pushing the logs directly
as json in plain files might help a log for that, as (r)syslog can then
read them (and there, no bottleneck with throttle) and send it in the
proper format to the remote logging infra.

Soooo... yeah. imho the "direct writing as json" might be the way to go :).

> 
> log shipping with other cri drivers such as containerd seems to work well too.

Not tested yet, but at least podman has the option (as a work for this
engine integration is done).

Cheers,

C.

> 
> Thanks,
> Kevin
> ________________________________________
> From: Sean Mooney [smooney at redhat.com]
> Sent: Wednesday, January 30, 2019 8:23 AM
> To: Emilien Macchi; Juan Antonio Osorio Robles
> Cc: openstack-discuss at lists.openstack.org
> Subject: Re: [TripleO] containers logging to stdout
> 
> On Wed, 2019-01-30 at 07:37 -0500, Emilien Macchi wrote:
>>
>>
>> On Wed, Jan 30, 2019 at 5:53 AM Juan Antonio Osorio Robles <jaosorior at redhat.com> wrote:
>>> Hello!
>>>
>>>
>>> In Queens, the a spec to provide the option to make containers log to
>>> standard output was proposed [1] [2]. Some work was done on that side,
>>> but due to the lack of traction, it wasn't completed. With the Train
>>> release coming, I think it would be a good idea to revive this effort,
>>> but make logging to stdout the default in that release.
>>>
>>> This would allow several benefits:
>>>
>>> * All logging from the containers would en up in journald; this would
>>> make it easier for us to forward the logs, instead of having to keep
>>> track of the different directories in /var/log/containers
>>>
>>> * The journald driver would add metadata to the logs about the container
>>> (we would automatically get what container ID issued the logs).
>>>
>>> * This wouldo also simplify the stacks (removing the Logging nested
>>> stack which is present in several templates).
>>>
>>> * Finally... if at some point we move towards kubernetes (or something
>>> in between), managing our containers, it would work with their logging
>>> tooling as well
>>
>> Also, I would add that it'll be aligned with what we did for Paunch-managed containers (with Podman backend) where
>> each ("long life") container has its own SystemD service (+ SystemD timer sometimes); so using journald makes total
>> sense to me.
> one thing to keep in mind is that journald apparently has rate limiting so if you contaiern are very verbose journald
> will actully slowdown the execution of the contaienr application as it slows down the rate at wich it can log.
> this came form a downstream conversation on irc were they were recommending that such applciation bypass journald and
> log to a file for best performacne.
>> --
>> Emilien Macchi
> 
> 
> 

-- 
Cédric Jeanneret
Software Engineer
DFG:DF

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190201/ef3323f2/attachment.sig>