Hello! In Queens, the a spec to provide the option to make containers log to standard output was proposed [1] [2]. Some work was done on that side, but due to the lack of traction, it wasn't completed. With the Train release coming, I think it would be a good idea to revive this effort, but make logging to stdout the default in that release. This would allow several benefits: * All logging from the containers would en up in journald; this would make it easier for us to forward the logs, instead of having to keep track of the different directories in /var/log/containers * The journald driver would add metadata to the logs about the container (we would automatically get what container ID issued the logs). * This wouldo also simplify the stacks (removing the Logging nested stack which is present in several templates). * Finally... if at some point we move towards kubernetes (or something in between), managing our containers, it would work with their logging tooling as well. Any thoughts? [1] https://specs.openstack.org/openstack/tripleo-specs/specs/queens/logging-std... [2] https://blueprints.launchpad.net/tripleo/+spec/logging-stdout-rsyslog
On 30.01.2019 11:49, Juan Antonio Osorio Robles wrote:
Hello!
In Queens, the a spec to provide the option to make containers log to standard output was proposed [1] [2]. Some work was done on that side, but due to the lack of traction, it wasn't completed. With the Train release coming, I think it would be a good idea to revive this effort, but make logging to stdout the default in that release.
+1
This would allow several benefits:
* All logging from the containers would en up in journald; this would make it easier for us to forward the logs, instead of having to keep track of the different directories in /var/log/containers
and simplifies logs rotation a lot!
* The journald driver would add metadata to the logs about the container (we would automatically get what container ID issued the logs).
* This wouldo also simplify the stacks (removing the Logging nested stack which is present in several templates).
* Finally... if at some point we move towards kubernetes (or something in between), managing our containers, it would work with their logging tooling as well.
Any thoughts?
[1] https://specs.openstack.org/openstack/tripleo-specs/specs/queens/logging-std...
[2] https://blueprints.launchpad.net/tripleo/+spec/logging-stdout-rsyslog
-- Best regards, Bogdan Dobrelya, Irc #bogdando
On Wed, Jan 30, 2019 at 5:53 AM Juan Antonio Osorio Robles < jaosorior@redhat.com> wrote:
Hello!
In Queens, the a spec to provide the option to make containers log to standard output was proposed [1] [2]. Some work was done on that side, but due to the lack of traction, it wasn't completed. With the Train release coming, I think it would be a good idea to revive this effort, but make logging to stdout the default in that release.
This would allow several benefits:
* All logging from the containers would en up in journald; this would make it easier for us to forward the logs, instead of having to keep track of the different directories in /var/log/containers
* The journald driver would add metadata to the logs about the container (we would automatically get what container ID issued the logs).
* This wouldo also simplify the stacks (removing the Logging nested stack which is present in several templates).
* Finally... if at some point we move towards kubernetes (or something in between), managing our containers, it would work with their logging tooling as well
Also, I would add that it'll be aligned with what we did for Paunch-managed containers (with Podman backend) where each ("long life") container has its own SystemD service (+ SystemD timer sometimes); so using journald makes total sense to me. -- Emilien Macchi
On Wed, 2019-01-30 at 07:37 -0500, Emilien Macchi wrote:
On Wed, Jan 30, 2019 at 5:53 AM Juan Antonio Osorio Robles <jaosorior@redhat.com> wrote:
Hello!
In Queens, the a spec to provide the option to make containers log to standard output was proposed [1] [2]. Some work was done on that side, but due to the lack of traction, it wasn't completed. With the Train release coming, I think it would be a good idea to revive this effort, but make logging to stdout the default in that release.
This would allow several benefits:
* All logging from the containers would en up in journald; this would make it easier for us to forward the logs, instead of having to keep track of the different directories in /var/log/containers
* The journald driver would add metadata to the logs about the container (we would automatically get what container ID issued the logs).
* This wouldo also simplify the stacks (removing the Logging nested stack which is present in several templates).
* Finally... if at some point we move towards kubernetes (or something in between), managing our containers, it would work with their logging tooling as well
Also, I would add that it'll be aligned with what we did for Paunch-managed containers (with Podman backend) where each ("long life") container has its own SystemD service (+ SystemD timer sometimes); so using journald makes total sense to me.
one thing to keep in mind is that journald apparently has rate limiting so if you contaiern are very verbose journald will actully slowdown the execution of the contaienr application as it slows down the rate at wich it can log. this came form a downstream conversation on irc were they were recommending that such applciation bypass journald and log to a file for best performacne.
-- Emilien Macchi
On 1/30/19 5:23 PM, Sean Mooney wrote:
On Wed, 2019-01-30 at 07:37 -0500, Emilien Macchi wrote:
On Wed, Jan 30, 2019 at 5:53 AM Juan Antonio Osorio Robles <jaosorior@redhat.com> wrote:
Hello!
In Queens, the a spec to provide the option to make containers log to standard output was proposed [1] [2]. Some work was done on that side, but due to the lack of traction, it wasn't completed. With the Train release coming, I think it would be a good idea to revive this effort, but make logging to stdout the default in that release.
This would allow several benefits:
* All logging from the containers would en up in journald; this would make it easier for us to forward the logs, instead of having to keep track of the different directories in /var/log/containers
* The journald driver would add metadata to the logs about the container (we would automatically get what container ID issued the logs).
* This wouldo also simplify the stacks (removing the Logging nested stack which is present in several templates).
* Finally... if at some point we move towards kubernetes (or something in between), managing our containers, it would work with their logging tooling as well
Also, I would add that it'll be aligned with what we did for Paunch-managed containers (with Podman backend) where each ("long life") container has its own SystemD service (+ SystemD timer sometimes); so using journald makes total sense to me.
one thing to keep in mind is that journald apparently has rate limiting so if you contaiern are very verbose journald will actully slowdown the execution of the contaienr application as it slows down the rate at wich it can log. this came form a downstream conversation on irc were they were recommending that such applciation bypass journald and log to a file for best performacne.
Another thing to check (if you haven't already) is what happens when journald restarts. We had an issue with os-collect-config where it died when journald was restarted because it started to get EPIPE responses when logging. I don't know if that would be an issue here, but it's something to check.
-- Emilien Macchi
k8s's offical way of dealing with logs is to ensure use of the docker json logger, not the journald one. then all the k8s log shippers have a standard way to gather the logs. Docker supports log rotation and other options too. seems to work out pretty well in practice. log shipping with other cri drivers such as containerd seems to work well too. Thanks, Kevin ________________________________________ From: Sean Mooney [smooney@redhat.com] Sent: Wednesday, January 30, 2019 8:23 AM To: Emilien Macchi; Juan Antonio Osorio Robles Cc: openstack-discuss@lists.openstack.org Subject: Re: [TripleO] containers logging to stdout On Wed, 2019-01-30 at 07:37 -0500, Emilien Macchi wrote:
On Wed, Jan 30, 2019 at 5:53 AM Juan Antonio Osorio Robles <jaosorior@redhat.com> wrote:
Hello!
In Queens, the a spec to provide the option to make containers log to standard output was proposed [1] [2]. Some work was done on that side, but due to the lack of traction, it wasn't completed. With the Train release coming, I think it would be a good idea to revive this effort, but make logging to stdout the default in that release.
This would allow several benefits:
* All logging from the containers would en up in journald; this would make it easier for us to forward the logs, instead of having to keep track of the different directories in /var/log/containers
* The journald driver would add metadata to the logs about the container (we would automatically get what container ID issued the logs).
* This wouldo also simplify the stacks (removing the Logging nested stack which is present in several templates).
* Finally... if at some point we move towards kubernetes (or something in between), managing our containers, it would work with their logging tooling as well
Also, I would add that it'll be aligned with what we did for Paunch-managed containers (with Podman backend) where each ("long life") container has its own SystemD service (+ SystemD timer sometimes); so using journald makes total sense to me.
one thing to keep in mind is that journald apparently has rate limiting so if you contaiern are very verbose journald will actully slowdown the execution of the contaienr application as it slows down the rate at wich it can log. this came form a downstream conversation on irc were they were recommending that such applciation bypass journald and log to a file for best performacne.
-- Emilien Macchi
On 1/30/19 6:26 PM, Fox, Kevin M wrote:
k8s's offical way of dealing with logs is to ensure use of the docker json logger, not the journald one. then all the k8s log shippers have a standard way to gather the logs. Docker supports log rotation and other options too. seems to work out pretty well in practice.
sending directly to a file looks a good option indeed. Journald and (r)syslog have both some throttle issue, and it might create some issues in case of service restarting and the like. Pushing logs directly from the container engine (podman does actually support the same options) might be the way to go. As long as we have a common, easy way to output the logs, it's all for the best. The only concern I have with the "not-journald" path is the possible lack of "journalctl -f CONTAINER_NAME=foo". But, compared to the risks exposed in this thread about the possible crash if journald isn't available, and throttling, I think it's fine. Also, a small note regarding "log re-shipping": some people might want to push their logs to some elk/kelk/others - pushing the logs directly as json in plain files might help a log for that, as (r)syslog can then read them (and there, no bottleneck with throttle) and send it in the proper format to the remote logging infra. Soooo... yeah. imho the "direct writing as json" might be the way to go :).
log shipping with other cri drivers such as containerd seems to work well too.
Not tested yet, but at least podman has the option (as a work for this engine integration is done). Cheers, C.
Thanks, Kevin ________________________________________ From: Sean Mooney [smooney@redhat.com] Sent: Wednesday, January 30, 2019 8:23 AM To: Emilien Macchi; Juan Antonio Osorio Robles Cc: openstack-discuss@lists.openstack.org Subject: Re: [TripleO] containers logging to stdout
On Wed, 2019-01-30 at 07:37 -0500, Emilien Macchi wrote:
On Wed, Jan 30, 2019 at 5:53 AM Juan Antonio Osorio Robles <jaosorior@redhat.com> wrote:
Hello!
In Queens, the a spec to provide the option to make containers log to standard output was proposed [1] [2]. Some work was done on that side, but due to the lack of traction, it wasn't completed. With the Train release coming, I think it would be a good idea to revive this effort, but make logging to stdout the default in that release.
This would allow several benefits:
* All logging from the containers would en up in journald; this would make it easier for us to forward the logs, instead of having to keep track of the different directories in /var/log/containers
* The journald driver would add metadata to the logs about the container (we would automatically get what container ID issued the logs).
* This wouldo also simplify the stacks (removing the Logging nested stack which is present in several templates).
* Finally... if at some point we move towards kubernetes (or something in between), managing our containers, it would work with their logging tooling as well
Also, I would add that it'll be aligned with what we did for Paunch-managed containers (with Podman backend) where each ("long life") container has its own SystemD service (+ SystemD timer sometimes); so using journald makes total sense to me.
one thing to keep in mind is that journald apparently has rate limiting so if you contaiern are very verbose journald will actully slowdown the execution of the contaienr application as it slows down the rate at wich it can log. this came form a downstream conversation on irc were they were recommending that such applciation bypass journald and log to a file for best performacne.
-- Emilien Macchi
-- Cédric Jeanneret Software Engineer DFG:DF
On 1/30/19 6:26 PM, Fox, Kevin M wrote:
k8s's offical way of dealing with logs is to ensure use of the docker json logger, not the journald one. then all the k8s log shippers have a standard way to gather the logs. Docker supports log rotation and other options too. seems to work out pretty well in practice. sending directly to a file looks a good option indeed. Journald and (r)syslog have both some throttle issue, and it might create some issues in case of service restarting and the like.
Pushing logs directly from the container engine (podman does actually support the same options) might be the way to go.
As long as we have a common, easy way to output the logs, it's all for the best. The only concern I have with the "not-journald" path is the possible lack of "journalctl -f CONTAINER_NAME=foo". But, compared to the risks exposed in this thread about the possible crash if journald isn't available, and throttling, I think it's fine.
Also, a small note regarding "log re-shipping": some people might want to push their logs to some elk/kelk/others - pushing the logs directly as json in plain files might help a log for that, as (r)syslog can then read them (and there, no bottleneck with throttle) and send it in the proper format to the remote logging infra.
Soooo... yeah. imho the "direct writing as json" might be the way to go :). That is just fine IMO. the runtime engine usually allows you to configure the logging driver (docker in CentOS defaults... or used to default, to journald); but if we find out that file is a better choice;
On 2/1/19 8:44 AM, Cédric Jeanneret wrote: that's entirely fine. The whole point is to let the runtime engine do its job, and handle the logging with the driver.
log shipping with other cri drivers such as containerd seems to work well too. Not tested yet, but at least podman has the option (as a work for this engine integration is done).
Cheers,
C.
Thanks, Kevin ________________________________________ From: Sean Mooney [smooney@redhat.com] Sent: Wednesday, January 30, 2019 8:23 AM To: Emilien Macchi; Juan Antonio Osorio Robles Cc: openstack-discuss@lists.openstack.org Subject: Re: [TripleO] containers logging to stdout
On Wed, 2019-01-30 at 07:37 -0500, Emilien Macchi wrote:
On Wed, Jan 30, 2019 at 5:53 AM Juan Antonio Osorio Robles <jaosorior@redhat.com> wrote:
Hello!
In Queens, the a spec to provide the option to make containers log to standard output was proposed [1] [2]. Some work was done on that side, but due to the lack of traction, it wasn't completed. With the Train release coming, I think it would be a good idea to revive this effort, but make logging to stdout the default in that release.
This would allow several benefits:
* All logging from the containers would en up in journald; this would make it easier for us to forward the logs, instead of having to keep track of the different directories in /var/log/containers
* The journald driver would add metadata to the logs about the container (we would automatically get what container ID issued the logs).
* This wouldo also simplify the stacks (removing the Logging nested stack which is present in several templates).
* Finally... if at some point we move towards kubernetes (or something in between), managing our containers, it would work with their logging tooling as well Also, I would add that it'll be aligned with what we did for Paunch-managed containers (with Podman backend) where each ("long life") container has its own SystemD service (+ SystemD timer sometimes); so using journald makes total sense to me.
one thing to keep in mind is that journald apparently has rate limiting so if you contaiern are very verbose journald will actully slowdown the execution of the contaienr application as it slows down the rate at wich it can log. this came form a downstream conversation on irc were they were recommending that such applciation bypass journald and log to a file for best performacne.
-- Emilien Macchi
Hello there! small thoughts: - we might already push the stdout logging, in parallel of the current existing one - that would already point some weakness and issues, without making the whole thing crash, since there aren't that many logs in stdout for now - that would already allow to check what's the best way to do it, and what's the best format for re-usability (thinking: sending logs to some (k)elk and the like) This would also allow devs to actually test that for their services. And thus going forward on this topic. Any thoughts? Cheers, C. On 1/30/19 11:49 AM, Juan Antonio Osorio Robles wrote:
Hello!
In Queens, the a spec to provide the option to make containers log to standard output was proposed [1] [2]. Some work was done on that side, but due to the lack of traction, it wasn't completed. With the Train release coming, I think it would be a good idea to revive this effort, but make logging to stdout the default in that release.
This would allow several benefits:
* All logging from the containers would en up in journald; this would make it easier for us to forward the logs, instead of having to keep track of the different directories in /var/log/containers
* The journald driver would add metadata to the logs about the container (we would automatically get what container ID issued the logs).
* This wouldo also simplify the stacks (removing the Logging nested stack which is present in several templates).
* Finally... if at some point we move towards kubernetes (or something in between), managing our containers, it would work with their logging tooling as well.
Any thoughts?
[1] https://specs.openstack.org/openstack/tripleo-specs/specs/queens/logging-std...
[2] https://blueprints.launchpad.net/tripleo/+spec/logging-stdout-rsyslog
-- Cédric Jeanneret Software Engineer DFG:DF
Hello, I'm currently testing things, related to this LP: https://bugs.launchpad.net/tripleo/+bug/1814897 We might hit some issues: - With docker, json-file log driver doesn't support any "path" options, and it outputs the files inside the container namespace (/var/lib/docker/container/ID/ID-json.log) - With podman, we actually have a "path" option, and it works nice. But the json-file isn't a JSON at all. - Docker supports journald and some other outputs - Podman doesn't support anything else than json-file Apparently, Docker seems to support a failing "journald" backend. So we might end with two ways of logging, if we're to keep docker in place. Cheers, C. On 2/5/19 11:11 AM, Cédric Jeanneret wrote:
Hello there!
small thoughts: - we might already push the stdout logging, in parallel of the current existing one
- that would already point some weakness and issues, without making the whole thing crash, since there aren't that many logs in stdout for now
- that would already allow to check what's the best way to do it, and what's the best format for re-usability (thinking: sending logs to some (k)elk and the like)
This would also allow devs to actually test that for their services. And thus going forward on this topic.
Any thoughts?
Cheers,
C.
On 1/30/19 11:49 AM, Juan Antonio Osorio Robles wrote:
Hello!
In Queens, the a spec to provide the option to make containers log to standard output was proposed [1] [2]. Some work was done on that side, but due to the lack of traction, it wasn't completed. With the Train release coming, I think it would be a good idea to revive this effort, but make logging to stdout the default in that release.
This would allow several benefits:
* All logging from the containers would en up in journald; this would make it easier for us to forward the logs, instead of having to keep track of the different directories in /var/log/containers
* The journald driver would add metadata to the logs about the container (we would automatically get what container ID issued the logs).
* This wouldo also simplify the stacks (removing the Logging nested stack which is present in several templates).
* Finally... if at some point we move towards kubernetes (or something in between), managing our containers, it would work with their logging tooling as well.
Any thoughts?
[1] https://specs.openstack.org/openstack/tripleo-specs/specs/queens/logging-std...
[2] https://blueprints.launchpad.net/tripleo/+spec/logging-stdout-rsyslog
-- Cédric Jeanneret Software Engineer DFG:DF
k8s only supports the json driver too. So if its the end goal, sticking to that might be easier. Thanks, Kevin ________________________________________ From: Cédric Jeanneret [cjeanner@redhat.com] Sent: Wednesday, February 06, 2019 10:11 PM To: openstack-discuss@lists.openstack.org Subject: Re: [TripleO] containers logging to stdout Hello, I'm currently testing things, related to this LP: https://bugs.launchpad.net/tripleo/+bug/1814897 We might hit some issues: - With docker, json-file log driver doesn't support any "path" options, and it outputs the files inside the container namespace (/var/lib/docker/container/ID/ID-json.log) - With podman, we actually have a "path" option, and it works nice. But the json-file isn't a JSON at all. - Docker supports journald and some other outputs - Podman doesn't support anything else than json-file Apparently, Docker seems to support a failing "journald" backend. So we might end with two ways of logging, if we're to keep docker in place. Cheers, C. On 2/5/19 11:11 AM, Cédric Jeanneret wrote:
Hello there!
small thoughts: - we might already push the stdout logging, in parallel of the current existing one
- that would already point some weakness and issues, without making the whole thing crash, since there aren't that many logs in stdout for now
- that would already allow to check what's the best way to do it, and what's the best format for re-usability (thinking: sending logs to some (k)elk and the like)
This would also allow devs to actually test that for their services. And thus going forward on this topic.
Any thoughts?
Cheers,
C.
On 1/30/19 11:49 AM, Juan Antonio Osorio Robles wrote:
Hello!
In Queens, the a spec to provide the option to make containers log to standard output was proposed [1] [2]. Some work was done on that side, but due to the lack of traction, it wasn't completed. With the Train release coming, I think it would be a good idea to revive this effort, but make logging to stdout the default in that release.
This would allow several benefits:
* All logging from the containers would en up in journald; this would make it easier for us to forward the logs, instead of having to keep track of the different directories in /var/log/containers
* The journald driver would add metadata to the logs about the container (we would automatically get what container ID issued the logs).
* This wouldo also simplify the stacks (removing the Logging nested stack which is present in several templates).
* Finally... if at some point we move towards kubernetes (or something in between), managing our containers, it would work with their logging tooling as well.
Any thoughts?
[1] https://specs.openstack.org/openstack/tripleo-specs/specs/queens/logging-std...
[2] https://blueprints.launchpad.net/tripleo/+spec/logging-stdout-rsyslog
-- Cédric Jeanneret Software Engineer DFG:DF
On 2/7/19 5:32 PM, Fox, Kevin M wrote:
k8s only supports the json driver too. So if its the end goal, sticking to that might be easier.
Cool then - the only big difference being the path, it shouldn't be that hard: docker outputs its json directly in a container-related path, while podman needs a parameter for it (in tripleo world, I've set it to /var/lib/containers/stdouts - we can change it if needed). Oh, not to mention the format - podman doesn't output a proper JSON, it's more a "kubernetes-like-ish" format iiuc[1]... A first patch has been merged by the way: https://review.openstack.org/635437 A second is waiting for reviews: https://review.openstack.org/635438 And a third will hit tripleo-heat-templates once we get the new paunch, in order to inject "--container-log-path /var/log/containers/stdouts". I suppose it would be best to push a parameter in heat (ContainerLogPath for example), I'll check how to do that and reflect its value in docker-puppet.py. Cheers, C. [1] https://github.com/containers/libpod/issues/2265#issuecomment-461060541
Thanks, Kevin ________________________________________ From: Cédric Jeanneret [cjeanner@redhat.com] Sent: Wednesday, February 06, 2019 10:11 PM To: openstack-discuss@lists.openstack.org Subject: Re: [TripleO] containers logging to stdout
Hello,
I'm currently testing things, related to this LP: https://bugs.launchpad.net/tripleo/+bug/1814897
We might hit some issues: - With docker, json-file log driver doesn't support any "path" options, and it outputs the files inside the container namespace (/var/lib/docker/container/ID/ID-json.log)
- With podman, we actually have a "path" option, and it works nice. But the json-file isn't a JSON at all.
- Docker supports journald and some other outputs
- Podman doesn't support anything else than json-file
Apparently, Docker seems to support a failing "journald" backend. So we might end with two ways of logging, if we're to keep docker in place.
Cheers,
C.
On 2/5/19 11:11 AM, Cédric Jeanneret wrote:
Hello there!
small thoughts: - we might already push the stdout logging, in parallel of the current existing one
- that would already point some weakness and issues, without making the whole thing crash, since there aren't that many logs in stdout for now
- that would already allow to check what's the best way to do it, and what's the best format for re-usability (thinking: sending logs to some (k)elk and the like)
This would also allow devs to actually test that for their services. And thus going forward on this topic.
Any thoughts?
Cheers,
C.
On 1/30/19 11:49 AM, Juan Antonio Osorio Robles wrote:
Hello!
In Queens, the a spec to provide the option to make containers log to standard output was proposed [1] [2]. Some work was done on that side, but due to the lack of traction, it wasn't completed. With the Train release coming, I think it would be a good idea to revive this effort, but make logging to stdout the default in that release.
This would allow several benefits:
* All logging from the containers would en up in journald; this would make it easier for us to forward the logs, instead of having to keep track of the different directories in /var/log/containers
* The journald driver would add metadata to the logs about the container (we would automatically get what container ID issued the logs).
* This wouldo also simplify the stacks (removing the Logging nested stack which is present in several templates).
* Finally... if at some point we move towards kubernetes (or something in between), managing our containers, it would work with their logging tooling as well.
Any thoughts?
[1] https://specs.openstack.org/openstack/tripleo-specs/specs/queens/logging-std...
[2] https://blueprints.launchpad.net/tripleo/+spec/logging-stdout-rsyslog
-- Cédric Jeanneret Software Engineer DFG:DF
-- Cédric Jeanneret Software Engineer DFG:DF
participants (7)
-
Ben Nemec
-
Bogdan Dobrelya
-
Cédric Jeanneret
-
Emilien Macchi
-
Fox, Kevin M
-
Juan Antonio Osorio Robles
-
Sean Mooney