[kolla] [kolla-ansible] fluentd doesn't forward OpenStack logs to Elasticsearch
I might have found a bug in Kolla-Ansible (Victoria version) but don't know where to file it. This is about central logging. In my installation, none of the interesting logs (Nova, Cinder, Neutron...) are sent to Elasticsearch. I confirmed that using tcpdump. I found that fluentd's config file /etc/kolla/fluentd/td-agent.conf tags these logs with "kolla.*". But later in the file, one finds filters like this: # Included from conf/filter/01-rewrite-0.14.conf.j2: <match kolla.var.log.kolla.*.*.log> @type rewrite_tag_filter capitalize_regex_backreference yes ... <rule> key programname pattern ^(nova-api|nova-compute|nova-compute-ironic|nova-conductor|nova-manage|nova-novncproxy|nova-scheduler|nova-placement-api|placement-api|privsep-helper)$ tag openstack_python </rule> If I understand this right, this basically re-tags all nova logs with "openstack_python". The same config file has an output rule at the very end. I think the intention is to make this a catch-all rule (or "match anything else"): # Included from conf/output/01-es.conf.j2: <match *.**> @type copy <store> @type elasticsearch host 192.168.122.209 port 9200 scheme http etc. Unfortunately, the /openstack_python/ tag doesn't match /*.**/, since it contains no dot. I fixed this with <match **>. Now I receive all logs, but I am not sure if this is the right way to fix it. The error, if it is one, is in https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/.... If you want me to file a bug, please let me know how. Bernd.
On Sat, 29 May 2021 at 11:24, Bernd Bausch <berndbausch@gmail.com> wrote:
I might have found a bug in Kolla-Ansible (Victoria version) but don't know where to file it.
Hi Bernd, you can file kolla-ansible bugs on Launchpad [1]. [1] https://bugs.launchpad.net/kolla-ansible/+filebug
This is about central logging. In my installation, none of the interesting logs (Nova, Cinder, Neutron...) are sent to Elasticsearch. I confirmed that using tcpdump.
I found that fluentd's config file /etc/kolla/fluentd/td-agent.conf tags these logs with "kolla.*". But later in the file, one finds filters like this:
# Included from conf/filter/01-rewrite-0.14.conf.j2: <match kolla.var.log.kolla.*.*.log> @type rewrite_tag_filter capitalize_regex_backreference yes ... <rule> key programname pattern ^(nova-api|nova-compute|nova-compute-ironic|nova-conductor|nova-manage|nova-novncproxy|nova-scheduler|nova-placement-api|placement-api|privsep-helper)$ tag openstack_python </rule>
If I understand this right, this basically re-tags all nova logs with "openstack_python".
The same config file has an output rule at the very end. I think the intention is to make this a catch-all rule (or "match anything else"):
# Included from conf/output/01-es.conf.j2: <match *.**> @type copy <store> @type elasticsearch host 192.168.122.209 port 9200 scheme http
etc.
Unfortunately, the openstack_python tag doesn't match *.**, since it contains no dot. I fixed this with <match **>. Now I receive all logs, but I am not sure if this is the right way to fix it.
I have seen log aggregation working, although possibly haven't tried it with Victoria. I can't see any obviously relevant changes, so please file a bug.
The error, if it is one, is in https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/....
If you want me to file a bug, please let me know how.
Bernd.
On Thu, 3 Jun 2021 at 09:24, Mark Goddard <mark@stackhpc.com> wrote:
On Sat, 29 May 2021 at 11:24, Bernd Bausch <berndbausch@gmail.com> wrote:
I might have found a bug in Kolla-Ansible (Victoria version) but don't know where to file it.
Hi Bernd, you can file kolla-ansible bugs on Launchpad [1].
[1] https://bugs.launchpad.net/kolla-ansible/+filebug
This is about central logging. In my installation, none of the interesting logs (Nova, Cinder, Neutron...) are sent to Elasticsearch. I confirmed that using tcpdump.
I found that fluentd's config file /etc/kolla/fluentd/td-agent.conf tags these logs with "kolla.*". But later in the file, one finds filters like this:
# Included from conf/filter/01-rewrite-0.14.conf.j2: <match kolla.var.log.kolla.*.*.log> @type rewrite_tag_filter capitalize_regex_backreference yes ... <rule> key programname pattern ^(nova-api|nova-compute|nova-compute-ironic|nova-conductor|nova-manage|nova-novncproxy|nova-scheduler|nova-placement-api|placement-api|privsep-helper)$ tag openstack_python </rule>
If I understand this right, this basically re-tags all nova logs with "openstack_python".
The same config file has an output rule at the very end. I think the intention is to make this a catch-all rule (or "match anything else"):
# Included from conf/output/01-es.conf.j2: <match *.**> @type copy <store> @type elasticsearch host 192.168.122.209 port 9200 scheme http
etc.
Unfortunately, the openstack_python tag doesn't match *.**, since it contains no dot. I fixed this with <match **>. Now I receive all logs, but I am not sure if this is the right way to fix it.
I have seen log aggregation working, although possibly haven't tried it with Victoria. I can't see any obviously relevant changes, so please file a bug.
I tried this out on a recent (CentOS) Victoria deployment. I couldn't reproduce the issue. My test case was nova-scheduler. I restarted it and verified that shutdown/startup logs appear in Elastic. Could you verify whether that case also works for you, and if so, provide a broken case.
The error, if it is one, is in https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/....
If you want me to file a bug, please let me know how.
Bernd.
On Thu, 3 Jun 2021 at 09:53, Mark Goddard <mark@stackhpc.com> wrote:
On Thu, 3 Jun 2021 at 09:24, Mark Goddard <mark@stackhpc.com> wrote:
On Sat, 29 May 2021 at 11:24, Bernd Bausch <berndbausch@gmail.com> wrote:
I might have found a bug in Kolla-Ansible (Victoria version) but don't know where to file it.
Hi Bernd, you can file kolla-ansible bugs on Launchpad [1].
[1] https://bugs.launchpad.net/kolla-ansible/+filebug
This is about central logging. In my installation, none of the interesting logs (Nova, Cinder, Neutron...) are sent to Elasticsearch. I confirmed that using tcpdump.
I found that fluentd's config file /etc/kolla/fluentd/td-agent.conf tags these logs with "kolla.*". But later in the file, one finds filters like this:
# Included from conf/filter/01-rewrite-0.14.conf.j2: <match kolla.var.log.kolla.*.*.log> @type rewrite_tag_filter capitalize_regex_backreference yes ... <rule> key programname pattern ^(nova-api|nova-compute|nova-compute-ironic|nova-conductor|nova-manage|nova-novncproxy|nova-scheduler|nova-placement-api|placement-api|privsep-helper)$ tag openstack_python </rule>
If I understand this right, this basically re-tags all nova logs with "openstack_python".
The same config file has an output rule at the very end. I think the intention is to make this a catch-all rule (or "match anything else"):
# Included from conf/output/01-es.conf.j2: <match *.**> @type copy <store> @type elasticsearch host 192.168.122.209 port 9200 scheme http
etc.
Unfortunately, the openstack_python tag doesn't match *.**, since it contains no dot. I fixed this with <match **>. Now I receive all logs, but I am not sure if this is the right way to fix it.
I have seen log aggregation working, although possibly haven't tried it with Victoria. I can't see any obviously relevant changes, so please file a bug.
I tried this out on a recent (CentOS) Victoria deployment. I couldn't reproduce the issue. My test case was nova-scheduler. I restarted it and verified that shutdown/startup logs appear in Elastic. Could you verify whether that case also works for you, and if so, provide a broken case.
Could you provide your version of fluentd/td-agent? docker exec -it fluentd td-agent --version I have 1.11.2, although we have just confirmed a broken case with 1.12.1. John Garbutt is planning to develop a patch based on your suggested fix.
The error, if it is one, is in https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/....
If you want me to file a bug, please let me know how.
Bernd.
participants (2)
-
Bernd Bausch
-
Mark Goddard