Open Stack

On Thu, 3 Jun 2021 at 09:53, Mark Goddard <mark at stackhpc.com> wrote:
>
> On Thu, 3 Jun 2021 at 09:24, Mark Goddard <mark at stackhpc.com> wrote:
> >
> > On Sat, 29 May 2021 at 11:24, Bernd Bausch <berndbausch at gmail.com> wrote:
> > >
> > > I might have found a bug in Kolla-Ansible (Victoria version) but don't know where to file it.
> >
> > Hi Bernd, you can file kolla-ansible bugs on Launchpad [1].
> >
> > [1] https://bugs.launchpad.net/kolla-ansible/+filebug
> >
> > >
> > > This is about central logging. In my installation, none of the interesting logs (Nova, Cinder, Neutron...) are sent to Elasticsearch. I confirmed that using tcpdump.
> > >
> > > I found that fluentd's config file /etc/kolla/fluentd/td-agent.conf tags these logs with "kolla.*". But later in the file, one finds filters like this:
> > >
> > > # Included from conf/filter/01-rewrite-0.14.conf.j2:
> > > <match kolla.var.log.kolla.*.*.log>
> > >     @type rewrite_tag_filter
> > >     capitalize_regex_backreference yes
> > > ...
> > >   <rule>
> > >     key     programname
> > >     pattern ^(nova-api|nova-compute|nova-compute-ironic|nova-conductor|nova-manage|nova-novncproxy|nova-scheduler|nova-placement-api|placement-api|privsep-helper)$
> > >     tag openstack_python
> > >   </rule>
> > >
> > > If I understand this right, this basically re-tags all nova logs with "openstack_python".
> > >
> > > The same config file has an output rule at the very end. I think the intention is to make this a catch-all rule (or "match anything else"):
> > >
> > > # Included from conf/output/01-es.conf.j2:
> > > <match *.**>
> > >     @type copy
> > >     <store>
> > >        @type elasticsearch
> > >        host 192.168.122.209
> > >        port 9200
> > >        scheme http
> > >
> > > etc.
> > >
> > > Unfortunately, the openstack_python tag doesn't match *.**, since it contains no dot.  I fixed this with <match **>. Now I receive all logs, but I am not sure if this is the right way to fix it.
> >
> > I have seen log aggregation working, although possibly haven't tried
> > it with Victoria. I can't see any obviously relevant changes, so
> > please file a bug.
>
> I tried this out on a recent (CentOS) Victoria deployment. I couldn't
> reproduce the issue. My test case was nova-scheduler. I restarted it
> and verified that shutdown/startup logs appear in Elastic. Could you
> verify whether that case also works for you, and if so, provide a
> broken case.

Could you provide your version of fluentd/td-agent?

docker exec -it fluentd td-agent --version

I have 1.11.2, although we have just confirmed a broken case with 1.12.1.

John Garbutt is planning to develop a patch based on your suggested fix.

>
> >
> > >
> > > The error, if it is one, is in https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/common/templates/conf/output/01-es.conf.j2.
> > >
> > > If you want me to file a bug, please let me know how.
> > >
> > > Bernd.
> > >
> > >