Open Stack

Pierre,

You are right, I was running an old fluentd image, after building and
deploying a new fluentd image for yoga everything started working.
Thank you for your suggestion.

Thanks
Vish


On Mon, Jul 25, 2022 at 2:26 PM Pierre Riteau <pierre at stackhpc.com> wrote:

> Are you sure you are running a Yoga fluentd image? I am seeing higher
> version numbers in logs of my Wallaby image:
>
> 2022-07-25 14:03:29 +0000 [info]: parsing config file is succeeded
> path="/etc/td-agent/td-agent.conf"
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluent-plugin-calyptia-monitoring'
> version '0.1.3'
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluent-plugin-elasticsearch'
> version '5.2.3'
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluent-plugin-elasticsearch'
> version '5.2.2'
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluent-plugin-flowcounter-simple'
> version '0.1.0'
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluent-plugin-grep' version '0.3.4'
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluent-plugin-grok-parser' version
> '2.6.2'
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluent-plugin-kafka' version
> '0.17.5'
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluent-plugin-metrics-cmetrics'
> version '0.1.2'
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluent-plugin-opensearch' version
> '1.0.4'
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluent-plugin-parser' version
> '0.6.1'
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluent-plugin-prometheus' version
> '2.0.3'
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluent-plugin-prometheus' version
> '2.0.2'
> 2022-07-25 14:03:29 +0000 [info]: gem
> 'fluent-plugin-prometheus_pushgateway' version '0.1.0'
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluent-plugin-record-modifier'
> version '2.1.0'
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter'
> version '2.4.0'
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluent-plugin-s3' version '1.6.1'
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluent-plugin-sd-dns' version
> '0.1.0'
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluent-plugin-systemd' version
> '1.0.5'
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluent-plugin-td' version '1.1.0'
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluent-plugin-utmpx' version '0.5.0'
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluent-plugin-webhdfs' version
> '1.5.0'
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluentd' version '1.14.6'
> 2022-07-25 14:03:29 +0000 [info]: gem 'fluentd' version '0.12.43'
>
> On Mon, 25 Jul 2022 at 19:50, Vishwanath <vishwanath.ne at gmail.com> wrote:
>
>> Hi Pierre,
>>
>> As far as i know we are not using any custom configs, Attached is
>> td-agent.conf file from one of the controller nodes. Please check attached.
>> thanks
>>
>>
>>
>>
>> Thanks
>> Vish
>> (408)-471-8579
>>
>>
>> On Thu, Jul 7, 2022 at 1:49 AM Pierre Riteau <pierre at stackhpc.com> wrote:
>>
>>> Hello Vish,
>>>
>>> Are you using a custom configuration for fluentd? Could you please share
>>> your generated td-agent.conf?
>>>
>>> Best wishes,
>>> Pierre Riteau (priteau)
>>>
>>> On Wed, 6 Jul 2022 at 23:34, Vishwanath <vishwanath.ne at gmail.com> wrote:
>>>
>>>> Hello all,
>>>>
>>>> I have upgraded openstack, we are currently running on 14.1.0. I
>>>> noticed fluentd container  restarting indefinitely. The message I see under
>>>> /var/log/kolla/fluentd/fluentd.log is as follows, Any thoughts on how to
>>>> fix this ? I noticed a similar issue back in 2017 from this post  -
>>>> https://bugs.launchpad.net/kolla-ansible/+bug/1663126
>>>>
>>>> *error log:*
>>>> *2022-07-06 21:20:42 +0000 [error]: config error
>>>> file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError
>>>> error="'format' parameter is required"*
>>>>
>>>>
>>>> *full logs:*
>>>> 2022-07-06 21:20:42 +0000 [info]: parsing config file is succeeded
>>>> path="/etc/td-agent/td-agent.conf"
>>>> 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-elasticsearch'
>>>> version '5.2.3'
>>>> 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-elasticsearch'
>>>> version '4.1.1'
>>>> 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-grep' version
>>>> '0.3.4'
>>>> 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-grok-parser'
>>>> version '2.6.2'
>>>> 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-kafka' version
>>>> '0.14.1'
>>>> 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-parser' version
>>>> '0.6.1'
>>>> 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-prometheus'
>>>> version '2.0.3'
>>>> 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-prometheus'
>>>> version '1.8.2'
>>>> 2022-07-06 21:20:42 +0000 [info]: gem
>>>> 'fluent-plugin-prometheus_pushgateway' version '0.0.2'
>>>> 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-record-modifier'
>>>> version '2.1.0'
>>>> 2022-07-06 21:20:42 +0000 [info]: gem
>>>> 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
>>>> 2022-07-06 21:20:42 +0000 [info]: gem
>>>> 'fluent-plugin-rewrite-tag-filter' version '2.3.0'
>>>> 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-s3' version '1.4.0'
>>>> 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-systemd' version
>>>> '1.0.2'
>>>> 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-td' version '1.1.0'
>>>> 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-webhdfs' version
>>>> '1.2.5'
>>>> 2022-07-06 21:20:42 +0000 [info]: gem 'fluentd' version '1.11.2'
>>>> 2022-07-06 21:20:42 +0000 [info]: gem 'fluentd' version '0.12.43'
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf97620
>>>> @keys="programname">,
>>>> /^(cinder-api-access|cloudkitty-api-access|gnocchi-api-access|horizon-access|keystone-apache-admin-access|keystone-apache-public-access|monasca-api-access|octavia-api-access|placement-api-access)$/,
>>>> "", "apache_access", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf96f90
>>>> @keys="programname">,
>>>> /^(aodh_wsgi_access|barbican_api_uwsgi_access|zun_api_wsgi_access|vitrage_wsgi_access)$/,
>>>> "", "wsgi_access", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf96950
>>>> @keys="programname">,
>>>> /^(nova-api|nova-compute|nova-compute-ironic|nova-conductor|nova-manage|nova-novncproxy|nova-scheduler|nova-placement-api|placement-api|privsep-helper)$/,
>>>> "", "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf96310
>>>> @keys="programname">, /^(sahara-api|sahara-engine)$/, "",
>>>> "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf95cd0
>>>> @keys="programname">,
>>>> /^(neutron-server|neutron-openvswitch-agent|neutron-ns-metadata-proxy|neutron-metadata-agent|neutron-l3-agent|neutron-dhcp-agent)$/,
>>>> "", "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf95690
>>>> @keys="programname">, /^(magnum-conductor|magnum-api)$/, "",
>>>> "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf95050
>>>> @keys="programname">, /^(keystone)$/, "", "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf94a10
>>>> @keys="programname">, /^(heat-engine|heat-api|heat-api-cfn)$/, "",
>>>> "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf943d0
>>>> @keys="programname">, /^(glance-api)$/, "", "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adabf850
>>>> @keys="programname">,
>>>> /^(cloudkitty-storage-init|cloudkitty-processor|cloudkitty-dbsync|cloudkitty-api)$/,
>>>> "", "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adabe978
>>>> @keys="programname">,
>>>> /^(ceilometer-polling|ceilometer-agent-notification)$/, "",
>>>> "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adabdcd0
>>>> @keys="programname">,
>>>> /^(barbican-api|barbican-worker|barbican-keystone-listener|barbican-db-manage|app)$/,
>>>> "", "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adabc4e8
>>>> @keys="programname">,
>>>> /^(aodh-notifier|aodh-listener|aodh-evaluator|aodh-dbsync)$/, "",
>>>> "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9fa50
>>>> @keys="programname">, /^(cyborg-api|cyborg-conductor|cyborg-agent)$/, "",
>>>> "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9f410
>>>> @keys="programname">,
>>>> /^(cinder-api|cinder-scheduler|cinder-manage|cinder-volume|cinder-backup|privsep-helper)$/,
>>>> "", "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9edd0
>>>> @keys="programname">, /^(mistral-server|mistral-engine|mistral-executor)$/,
>>>> "", "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9e790
>>>> @keys="programname">,
>>>> /^(designate-api|designate-central|designate-manage|designate-mdns|designate-sink|designate-worker)$/,
>>>> "", "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9e150
>>>> @keys="programname">,
>>>> /^(manila-api|manila-data|manila-manage|manila-share|manila-scheduler)$/,
>>>> "", "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9db10
>>>> @keys="programname">,
>>>> /^(trove-api|trove-conductor|trove-manage|trove-taskmanager)$/, "",
>>>> "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9d4d0
>>>> @keys="programname">, /^(murano-api|murano-engine)$/, "",
>>>> "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9ce90
>>>> @keys="programname">,
>>>> /^(senlin-api|senlin-conductor|senlin-engine|senlin-health-manager)$/, "",
>>>> "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9c850
>>>> @keys="programname">,
>>>> /^(watcher-api|watcher-applier|watcher-db-manage|watcher-decision-engine)$/,
>>>> "", "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9c210
>>>> @keys="programname">, /^(freezer-api|freezer-api_access|freezer-manage)$/,
>>>> "", "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adab7880
>>>> @keys="programname">,
>>>> /^(octavia-api|octavia-health-manager|octavia-housekeeping|octavia-worker)$/,
>>>> "", "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adab6ef8
>>>> @keys="programname">, /^(zun-api|zun-compute|zun-cni-daemon)$/, "",
>>>> "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adab5918
>>>> @keys="programname">, /^(kuryr-server)$/, "", "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfabe90
>>>> @keys="programname">,
>>>> /^(gnocchi-api|gnocchi-statsd|gnocchi-metricd|gnocchi-upgrade)$/, "",
>>>> "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfab850
>>>> @keys="programname">, /^(ironic-api|ironic-conductor|ironic-inspector)$/,
>>>> "", "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfab210
>>>> @keys="programname">, /^(tacker-server|tacker-conductor)$/, "",
>>>> "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfaabd0
>>>> @keys="programname">,
>>>> /^(vitrage-ml|vitrage-notifier|vitrage-graph|vitrage-persistor)$/, "",
>>>> "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfaa590
>>>> @keys="programname">, /^(blazar-api|blazar-manager)$/, "",
>>>> "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfa9f50
>>>> @keys="programname">,
>>>> /^(monasca-api|monasca-notification|monasca-persister|agent-collector|agent-forwarder|agent-statsd)$/,
>>>> "", "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfa9910
>>>> @keys="programname">, /^(masakari-engine|masakari-api)$/, "",
>>>> "openstack_python", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> programname
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfa92d0
>>>> @keys="programname">, /.+/, "", "unmatched", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> Payload
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfc9cb0
>>>> @keys="Payload">, /^\d{6}/, "", "infra.mariadb.mysqld_safe", nil]
>>>> 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule:
>>>> Payload
>>>> [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfc9670
>>>> @keys="Payload">, /^\d{4}-\d{2}-\d{2}/, "", "infra.mariadb.mysqld", nil]
>>>> *2022-07-06 21:20:42 +0000 [error]: config error
>>>> file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError
>>>> error="'format' parameter is required"*
>>>>
>>>>
>>>> Thanks
>>>> Vish
>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20220726/943d6511/attachment-0001.htm>