[OpenStack-Ansible] Log Aggregation in Yoga
Hello, I've been told that Openstack now uses journald rather than rsyslog, but the Yoga docs still show a log aggregation host. Although I still prefer rsyslog, what I really prefer is to have logs for all hosts in the cluster collected in one place. How do I configure this for a fresh installation? Is it reasonable to assign this duty to an infrastructure host? Thanks. -Dave -- Dave Hall Binghamton University kdhall@binghamton.edu
Hey, Yes, indeed, we do store all logs with journald nowadays and rsyslog is not being used. These roles and documentation is our technical debt and we should have deprecated that back in Victoria. Journald for containers is bind mounted and you can check for all container logs on host. At the same time there plenty of tools to convert journald to any format of your taste including rsyslog. To have that said, I would discourage using rsyslog as with that you loose tons of important metadata and it's hard to parse them properly. If you're using any central logging tool, like elk or graylog, there re ways to forward journal to these as well. We also have roles for elk or graylog in our ops repo https://opendev.org/openstack/openstack-ansible-ops Though we don't provide support for them, thus don't guarantee they're working as expected and some effort might be needed to update them. сб, 9 июл. 2022 г., 23:33 Dave Hall <kdhall@binghamton.edu>:
Hello,
I've been told that Openstack now uses journald rather than rsyslog, but the Yoga docs still show a log aggregation host. Although I still prefer rsyslog, what I really prefer is to have logs for all hosts in the cluster collected in one place. How do I configure this for a fresh installation? Is it reasonable to assign this duty to an infrastructure host?
Thanks.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu
Two further questions: After looking a bit further, journald seems to have the ability to forward to an journald aggregation target. Is there an option in OpenStack-Ansible to configure this in all of the deployed containers? Are there any relevant services deployed in a typical OSA deployment that don't log to journald? Thanks. -Dave On Sun, Jul 10, 2022, 6:02 AM Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
Hey,
Yes, indeed, we do store all logs with journald nowadays and rsyslog is not being used. These roles and documentation is our technical debt and we should have deprecated that back in Victoria.
Journald for containers is bind mounted and you can check for all container logs on host. At the same time there plenty of tools to convert journald to any format of your taste including rsyslog. To have that said, I would discourage using rsyslog as with that you loose tons of important metadata and it's hard to parse them properly. If you're using any central logging tool, like elk or graylog, there re ways to forward journal to these as well. We also have roles for elk or graylog in our ops repo https://opendev.org/openstack/openstack-ansible-ops Though we don't provide support for them, thus don't guarantee they're working as expected and some effort might be needed to update them.
сб, 9 июл. 2022 г., 23:33 Dave Hall <kdhall@binghamton.edu>:
Hello,
I've been told that Openstack now uses journald rather than rsyslog, but the Yoga docs still show a log aggregation host. Although I still prefer rsyslog, what I really prefer is to have logs for all hosts in the cluster collected in one place. How do I configure this for a fresh installation? Is it reasonable to assign this duty to an infrastructure host?
Thanks.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu
Yes, we have a role openstack.osa.journald_remote that should be shipped and installed during bootstrap: https://opendev.org/openstack/openstack-ansible-plugins/src/branch/master/ro... I think as of today only ceph can not log into journald. вс, 10 июл. 2022 г., 15:43 Dave Hall <kdhall@binghamton.edu>:
Two further questions:
After looking a bit further, journald seems to have the ability to forward to an journald aggregation target. Is there an option in OpenStack-Ansible to configure this in all of the deployed containers?
Are there any relevant services deployed in a typical OSA deployment that don't log to journald?
Thanks.
-Dave
On Sun, Jul 10, 2022, 6:02 AM Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
Hey,
Yes, indeed, we do store all logs with journald nowadays and rsyslog is not being used. These roles and documentation is our technical debt and we should have deprecated that back in Victoria.
Journald for containers is bind mounted and you can check for all container logs on host. At the same time there plenty of tools to convert journald to any format of your taste including rsyslog. To have that said, I would discourage using rsyslog as with that you loose tons of important metadata and it's hard to parse them properly. If you're using any central logging tool, like elk or graylog, there re ways to forward journal to these as well. We also have roles for elk or graylog in our ops repo https://opendev.org/openstack/openstack-ansible-ops Though we don't provide support for them, thus don't guarantee they're working as expected and some effort might be needed to update them.
сб, 9 июл. 2022 г., 23:33 Dave Hall <kdhall@binghamton.edu>:
Hello,
I've been told that Openstack now uses journald rather than rsyslog, but the Yoga docs still show a log aggregation host. Although I still prefer rsyslog, what I really prefer is to have logs for all hosts in the cluster collected in one place. How do I configure this for a fresh installation? Is it reasonable to assign this duty to an infrastructure host?
Thanks.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu
Btw journald-remote on itself has quite a few long-lasting bugs, and it's development seems quite absent as of today. Most significant one is regarding rotation of logs on the remote server, check https://github.com/systemd/systemd/issues/5242 вс, 10 июл. 2022 г., 15:43 Dave Hall <kdhall@binghamton.edu>:
Two further questions:
After looking a bit further, journald seems to have the ability to forward to an journald aggregation target. Is there an option in OpenStack-Ansible to configure this in all of the deployed containers?
Are there any relevant services deployed in a typical OSA deployment that don't log to journald?
Thanks.
-Dave
On Sun, Jul 10, 2022, 6:02 AM Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
Hey,
Yes, indeed, we do store all logs with journald nowadays and rsyslog is not being used. These roles and documentation is our technical debt and we should have deprecated that back in Victoria.
Journald for containers is bind mounted and you can check for all container logs on host. At the same time there plenty of tools to convert journald to any format of your taste including rsyslog. To have that said, I would discourage using rsyslog as with that you loose tons of important metadata and it's hard to parse them properly. If you're using any central logging tool, like elk or graylog, there re ways to forward journal to these as well. We also have roles for elk or graylog in our ops repo https://opendev.org/openstack/openstack-ansible-ops Though we don't provide support for them, thus don't guarantee they're working as expected and some effort might be needed to update them.
сб, 9 июл. 2022 г., 23:33 Dave Hall <kdhall@binghamton.edu>:
Hello,
I've been told that Openstack now uses journald rather than rsyslog, but the Yoga docs still show a log aggregation host. Although I still prefer rsyslog, what I really prefer is to have logs for all hosts in the cluster collected in one place. How do I configure this for a fresh installation? Is it reasonable to assign this duty to an infrastructure host?
Thanks.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu
I had been poking at some updated documentation for OSA around remote journaling/centralized logging that I have been meaning to put in for review which may be useful here. I'll try and get to tidying it up in the next couple weeks. --Neil On Sun, Jul 10, 2022, 10:04 Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
Btw journald-remote on itself has quite a few long-lasting bugs, and it's development seems quite absent as of today. Most significant one is regarding rotation of logs on the remote server, check https://github.com/systemd/systemd/issues/5242
вс, 10 июл. 2022 г., 15:43 Dave Hall <kdhall@binghamton.edu>:
Two further questions:
After looking a bit further, journald seems to have the ability to forward to an journald aggregation target. Is there an option in OpenStack-Ansible to configure this in all of the deployed containers?
Are there any relevant services deployed in a typical OSA deployment that don't log to journald?
Thanks.
-Dave
On Sun, Jul 10, 2022, 6:02 AM Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
Hey,
Yes, indeed, we do store all logs with journald nowadays and rsyslog is not being used. These roles and documentation is our technical debt and we should have deprecated that back in Victoria.
Journald for containers is bind mounted and you can check for all container logs on host. At the same time there plenty of tools to convert journald to any format of your taste including rsyslog. To have that said, I would discourage using rsyslog as with that you loose tons of important metadata and it's hard to parse them properly. If you're using any central logging tool, like elk or graylog, there re ways to forward journal to these as well. We also have roles for elk or graylog in our ops repo https://opendev.org/openstack/openstack-ansible-ops Though we don't provide support for them, thus don't guarantee they're working as expected and some effort might be needed to update them.
сб, 9 июл. 2022 г., 23:33 Dave Hall <kdhall@binghamton.edu>:
Hello,
I've been told that Openstack now uses journald rather than rsyslog, but the Yoga docs still show a log aggregation host. Although I still prefer rsyslog, what I really prefer is to have logs for all hosts in the cluster collected in one place. How do I configure this for a fresh installation? Is it reasonable to assign this duty to an infrastructure host?
Thanks.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu
participants (3)
-
Dave Hall
-
Dmitriy Rabotyagov
-
Neil Hanlon