Hi;

I did update another update of my platforme, and still having problems with fluentd container (it is still restarting) but with different error :

Running command: '/usr/sbin/td-agent -o /var/log/kolla/fluentd/fluentd.log'
/opt/td-agent/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.3.0, because faraday-1.10.3 conflicts with faraday (>= 2.0.0), faraday-excon-1.1.0 conflicts with faraday-excon (>= 2.0.0) (Gem::ConflictError)
        from /opt/td-agent/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
        from /opt/td-agent/lib/ruby/2.7.0/rubygems.rb:217:in `rescue in try_activate'
        from /opt/td-agent/lib/ruby/2.7.0/rubygems.rb:210:in `try_activate'
        from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
        from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:20:in `<top (required)>'
        from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:103:in `block in search'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:100:in `each'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:100:in `search'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:44:in `lookup'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:169:in `new_impl'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:114:in `new_output'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/multi_output.rb:108:in `block in configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/multi_output.rb:99:in `each'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/multi_output.rb:99:in `configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/out_copy.rb:39:in `configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:187:in `configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:132:in `add_match'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:74:in `block in configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in `each'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in `configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:149:in `configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:105:in `configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:80:in `run_configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:571:in `run_supervisor'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
        from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/bin/fluentd:15:in `<top (required)>'
        from /opt/td-agent/bin/fluentd:23:in `load'
        from /opt/td-agent/bin/fluentd:23:in `<top (required)>'
        from /usr/sbin/td-agent:15:in `load'
        from /usr/sbin/td-agent:15:in `<main>'
/opt/td-agent/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.3.0, because faraday-1.10.3 conflicts with faraday (>= 2.0.0), faraday-excon-1.1.0 conflicts with faraday-excon (>= 2.0.0) (Gem::ConflictError)
        from /opt/td-agent/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
        from /opt/td-agent/lib/ruby/2.7.0/rubygems.rb:211:in `try_activate'
        from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
        from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:20:in `<top (required)>'
        from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:103:in `block in search'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:100:in `each'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:100:in `search'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:44:in `lookup'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:169:in `new_impl'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:114:in `new_output'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/multi_output.rb:108:in `block in configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/multi_output.rb:99:in `each'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/multi_output.rb:99:in `configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/out_copy.rb:39:in `configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:187:in `configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:132:in `add_match'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:74:in `block in configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in `each'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in `configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:149:in `configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:105:in `configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:80:in `run_configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:571:in `run_supervisor'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
        from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/bin/fluentd:15:in `<top (required)>'
        from /opt/td-agent/bin/fluentd:23:in `load'
        from /opt/td-agent/bin/fluentd:23:in `<top (required)>'
        from /usr/sbin/td-agent:15:in `load'
        from /usr/sbin/td-agent:15:in `<main>'
/opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require': cannot load such file -- fluent/log-ext (LoadError)
        from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:20:in `<top (required)>'
        from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:103:in `block in search'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:100:in `each'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:100:in `search'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:44:in `lookup'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:169:in `new_impl'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:114:in `new_output'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/multi_output.rb:108:in `block in configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/multi_output.rb:99:in `each'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/multi_output.rb:99:in `configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/out_copy.rb:39:in `configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:187:in `configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:132:in `add_match'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:74:in `block in configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in `each'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in `configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:149:in `configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:105:in `configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:80:in `run_configure'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:571:in `run_supervisor'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
        from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/bin/fluentd:15:in `<top (required)>'
        from /opt/td-agent/bin/fluentd:23:in `load'
        from /opt/td-agent/bin/fluentd:23:in `<top (required)>'
        from /usr/sbin/td-agent:15:in `load'
        from /usr/sbin/td-agent:15:in `<main>'
Regards.

Virus-free.www.avast.com

Le lun. 8 mai 2023 à 14:59, Michał Nasiadka <mnasiadka@gmail.com> a écrit :
Hello,

Then probably the only solution is to rebuild the container image with this patch: https://review.opendev.org/c/openstack/kolla/+/882289
It will take a couple days before it’s merged upstream.

Best regards,
Michal

On 8 May 2023, at 14:06, wodel youchi <wodel.youchi@gmail.com> wrote:

Hi,

I cannot access the fluentd container, it crashes almost instantly after each restart.
Is there a workaround to make the container start and wait until the commands are executed?

Regards.

Le jeu. 4 mai 2023 à 14:12, Michał Nasiadka <mnasiadka@gmail.com> a écrit :
Hello,

Kolla-Ansible is not supporting both opensearch and elasticsearch running at the same time - so if you’re using cloudkitty - it’s better to stick for Elasticsearch for now (CK does not support OpenSearch yet).

I started working on the bug - will let you know in the bug report when a fix will be merged and images published.
In the meantime you can try to uninstall the too-new elasticsearch gems using td-agent-gem uninstall in your running container image.

Best regards,
Michal

On 4 May 2023, at 14:33, wodel youchi <wodel.youchi@gmail.com> wrote:

Hi,

I'll try to open a bug for this.

I am using elasticsearch also with Cloudkitty : cloudkitty_storage_backend: "elasticsearch" instead of influxdb to get some HA.
Will I still get the fluentd problem even if I migrate to Opensearch leaving  Cloudkitty with  elasticsearch???

Regards.

Virus-free.www.avast.com

Le jeu. 4 mai 2023 à 07:42, Michał Nasiadka <mnasiadka@gmail.com> a écrit :
Hello,

That probably is a Kolla bug - can you please raise a bug in launchpad.net?
The other alternative is to migrate to OpenSearch (we’ve back ported this functionality recently) - https://docs.openstack.org/kolla-ansible/yoga/reference/logging-and-monitoring/central-logging-guide-opensearch.html#migration

Best regards,
Michal

On 3 May 2023, at 17:13, wodel youchi <wodel.youchi@gmail.com> wrote:

Hi,

I have finished the update of my openstack platform with newer containers.

While verifying I noticed that fluentd container keeps restarting.

In the log file I am having this :
2023-05-03 16:07:59 +0100 [error]: #0 config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Using Elasticsearch client 8.7.1 is not compatible for your Elasticsearch server. Please check your using elasticsearch gem version and Elasticsearch server."
2023-05-03 16:07:59 +0100 [error]: Worker 0 finished unexpectedly with status 2
2023-05-03 16:07:59 +0100 [info]: Received graceful stop

Those are the images I am using :
(yogavenv) [deployer@rscdeployer ~]$ sudo docker images | grep -i elas
192.168.1.16:4000/openstack.kolla/centos-source-prometheus-elasticsearch-exporter   yoga030523       b48f63ed0072   12 hours ago   539MB
192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch                       yoga030523       3558611b0cf4   12 hours ago   1.2GB
192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch-curator               yoga030523       83a6b48339ea   12 hours ago   637MB

(yogavenv) [deployer@rscdeployer ~]$ sudo docker images | grep -i fluen
192.168.1.16:4000/openstack.kolla/centos-source-fluentd                             yoga030523       bf6596e139e2   12 hours ago   847MB

Any ideas?

Regards.

Virus-free.www.avast.com