<div dir="ltr">Hi Pierre,<div><br></div><div>As far as i know we are not using any custom configs, Attached is td-agent.conf file from one of the controller nodes. Please check attached. thanks</div><div><br></div><div><br></div><div><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><br></div><div>Thanks</div><div>Vish</div><div>(408)-471-8579</div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jul 7, 2022 at 1:49 AM Pierre Riteau <<a href="mailto:pierre@stackhpc.com">pierre@stackhpc.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Hello Vish,<div><br></div><div>Are you using a custom configuration for fluentd? Could you please share your generated td-agent.conf?</div><div><br></div><div>Best wishes,</div><div>Pierre Riteau (priteau)</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 6 Jul 2022 at 23:34, Vishwanath <<a href="mailto:vishwanath.ne@gmail.com" target="_blank">vishwanath.ne@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello all,<div><br></div><div>I have upgraded openstack, we are currently running on 14.1.0. I noticed fluentd container  restarting indefinitely. The message I see under /var/log/kolla/fluentd/fluentd.log is as follows, Any thoughts on how to fix this ? I noticed a similar issue back in 2017 from this post  - <a href="https://bugs.launchpad.net/kolla-ansible/+bug/1663126" target="_blank">https://bugs.launchpad.net/kolla-ansible/+bug/1663126</a> </div><div><br></div><div><b>error log:</b></div><div><b>2022-07-06 21:20:42 +0000 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="'format' parameter is required"</b><br></div><div><br></div><div><br></div><div><b><u>full logs:</u></b></div><div>2022-07-06 21:20:42 +0000 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"<br>2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.3'<br>2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '4.1.1'<br>2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-grep' version '0.3.4'<br>2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'<br>2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-kafka' version '0.14.1'<br>2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-parser' version '0.6.1'<br>2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'<br>2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.8.2'<br>2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.0.2'<br>2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'<br>2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'<br>2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.3.0'<br>2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-s3' version '1.4.0'<br>2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.2'<br>2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-td' version '1.1.0'<br>2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.2.5'<br>2022-07-06 21:20:42 +0000 [info]: gem 'fluentd' version '1.11.2'<br>2022-07-06 21:20:42 +0000 [info]: gem 'fluentd' version '0.12.43'<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf97620 @keys="programname">, /^(cinder-api-access|cloudkitty-api-access|gnocchi-api-access|horizon-access|keystone-apache-admin-access|keystone-apache-public-access|monasca-api-access|octavia-api-access|placement-api-access)$/, "", "apache_access", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf96f90 @keys="programname">, /^(aodh_wsgi_access|barbican_api_uwsgi_access|zun_api_wsgi_access|vitrage_wsgi_access)$/, "", "wsgi_access", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf96950 @keys="programname">, /^(nova-api|nova-compute|nova-compute-ironic|nova-conductor|nova-manage|nova-novncproxy|nova-scheduler|nova-placement-api|placement-api|privsep-helper)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf96310 @keys="programname">, /^(sahara-api|sahara-engine)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf95cd0 @keys="programname">, /^(neutron-server|neutron-openvswitch-agent|neutron-ns-metadata-proxy|neutron-metadata-agent|neutron-l3-agent|neutron-dhcp-agent)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf95690 @keys="programname">, /^(magnum-conductor|magnum-api)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf95050 @keys="programname">, /^(keystone)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf94a10 @keys="programname">, /^(heat-engine|heat-api|heat-api-cfn)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf943d0 @keys="programname">, /^(glance-api)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adabf850 @keys="programname">, /^(cloudkitty-storage-init|cloudkitty-processor|cloudkitty-dbsync|cloudkitty-api)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adabe978 @keys="programname">, /^(ceilometer-polling|ceilometer-agent-notification)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adabdcd0 @keys="programname">, /^(barbican-api|barbican-worker|barbican-keystone-listener|barbican-db-manage|app)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adabc4e8 @keys="programname">, /^(aodh-notifier|aodh-listener|aodh-evaluator|aodh-dbsync)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9fa50 @keys="programname">, /^(cyborg-api|cyborg-conductor|cyborg-agent)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9f410 @keys="programname">, /^(cinder-api|cinder-scheduler|cinder-manage|cinder-volume|cinder-backup|privsep-helper)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9edd0 @keys="programname">, /^(mistral-server|mistral-engine|mistral-executor)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9e790 @keys="programname">, /^(designate-api|designate-central|designate-manage|designate-mdns|designate-sink|designate-worker)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9e150 @keys="programname">, /^(manila-api|manila-data|manila-manage|manila-share|manila-scheduler)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9db10 @keys="programname">, /^(trove-api|trove-conductor|trove-manage|trove-taskmanager)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9d4d0 @keys="programname">, /^(murano-api|murano-engine)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9ce90 @keys="programname">, /^(senlin-api|senlin-conductor|senlin-engine|senlin-health-manager)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9c850 @keys="programname">, /^(watcher-api|watcher-applier|watcher-db-manage|watcher-decision-engine)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9c210 @keys="programname">, /^(freezer-api|freezer-api_access|freezer-manage)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adab7880 @keys="programname">, /^(octavia-api|octavia-health-manager|octavia-housekeeping|octavia-worker)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adab6ef8 @keys="programname">, /^(zun-api|zun-compute|zun-cni-daemon)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adab5918 @keys="programname">, /^(kuryr-server)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfabe90 @keys="programname">, /^(gnocchi-api|gnocchi-statsd|gnocchi-metricd|gnocchi-upgrade)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfab850 @keys="programname">, /^(ironic-api|ironic-conductor|ironic-inspector)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfab210 @keys="programname">, /^(tacker-server|tacker-conductor)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfaabd0 @keys="programname">, /^(vitrage-ml|vitrage-notifier|vitrage-graph|vitrage-persistor)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfaa590 @keys="programname">, /^(blazar-api|blazar-manager)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfa9f50 @keys="programname">, /^(monasca-api|monasca-notification|monasca-persister|agent-collector|agent-forwarder|agent-statsd)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfa9910 @keys="programname">, /^(masakari-engine|masakari-api)$/, "", "openstack_python", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfa92d0 @keys="programname">, /.+/, "", "unmatched", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: Payload [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfc9cb0 @keys="Payload">, /^\d{6}/, "", "infra.mariadb.mysqld_safe", nil]<br>2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: Payload [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfc9670 @keys="Payload">, /^\d{4}-\d{2}-\d{2}/, "", "infra.mariadb.mysqld", nil]<br><b>2022-07-06 21:20:42 +0000 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="'format' parameter is required"</b><br></div><div><br clear="all"><div><div dir="ltr"><div dir="ltr"><div><br></div><div>Thanks</div><div>Vish</div><div><br></div></div></div></div></div></div>
</blockquote></div></div>
</blockquote></div>