[kolla-ansible][yoga] fluentd problem with elasticsearch after update
Hi, I have finished the update of my openstack platform with newer containers. While verifying I noticed that fluentd container keeps restarting. In the log file I am having this :
2023-05-03 16:07:59 +0100 [error]: #0 config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Using Elasticsearch client 8.7.1 is not compatible for your Elasticsearch server. Please check your using elasticsearch gem version and Elasticsearch server." 2023-05-03 16:07:59 +0100 [error]: Worker 0 finished unexpectedly with status 2 2023-05-03 16:07:59 +0100 [info]: Received graceful stop
Those are the images I am using : (yogavenv) [deployer@rscdeployer ~]$ sudo docker images | grep -i elas 192.168.1.16:4000/openstack.kolla/centos-source-prometheus-elasticsearch-exporter yoga030523 b48f63ed0072 12 hours ago 539MB 192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch yoga030523 3558611b0cf4 12 hours ago 1.2GB 192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch-curator yoga030523 83a6b48339ea 12 hours ago 637MB (yogavenv) [deployer@rscdeployer ~]$ sudo docker images | grep -i fluen 192.168.1.16:4000/openstack.kolla/centos-source-fluentd yoga030523 bf6596e139e2 12 hours ago 847MB Any ideas? Regards. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Hello, That probably is a Kolla bug - can you please raise a bug in launchpad.net <http://launchpad.net/>? The other alternative is to migrate to OpenSearch (we’ve back ported this functionality recently) - https://docs.openstack.org/kolla-ansible/yoga/reference/logging-and-monitori... Best regards, Michal
On 3 May 2023, at 17:13, wodel youchi <wodel.youchi@gmail.com> wrote:
Hi,
I have finished the update of my openstack platform with newer containers.
While verifying I noticed that fluentd container keeps restarting.
In the log file I am having this :
2023-05-03 16:07:59 +0100 [error]: #0 config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Using Elasticsearch client 8.7.1 is not compatible for your Elasticsearch server. Please check your using elasticsearch gem version and Elasticsearch server." 2023-05-03 16:07:59 +0100 [error]: Worker 0 finished unexpectedly with status 2 2023-05-03 16:07:59 +0100 [info]: Received graceful stop
Those are the images I am using : (yogavenv) [deployer@rscdeployer ~]$ sudo docker images | grep -i elas 192.168.1.16:4000/openstack.kolla/centos-source-prometheus-elasticsearch-exporter <http://192.168.1.16:4000/openstack.kolla/centos-source-prometheus-elasticsearch-exporter> yoga030523 b48f63ed0072 12 hours ago 539MB 192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch <http://192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch> yoga030523 3558611b0cf4 12 hours ago 1.2GB 192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch-curator <http://192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch-curator> yoga030523 83a6b48339ea 12 hours ago 637MB
(yogavenv) [deployer@rscdeployer ~]$ sudo docker images | grep -i fluen 192.168.1.16:4000/openstack.kolla/centos-source-fluentd <http://192.168.1.16:4000/openstack.kolla/centos-source-fluentd> yoga030523 bf6596e139e2 12 hours ago 847MB
Any ideas?
Regards.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <x-msg://4/#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Hi, I'll try to open a bug for this. I am using elasticsearch also with Cloudkitty : cloudkitty_storage_backend: "elasticsearch" instead of influxdb to get some HA. Will I still get the fluentd problem even if I migrate to Opensearch leaving Cloudkitty with elasticsearch??? Regards. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> Le jeu. 4 mai 2023 à 07:42, Michał Nasiadka <mnasiadka@gmail.com> a écrit :
Hello,
That probably is a Kolla bug - can you please raise a bug in launchpad.net ? The other alternative is to migrate to OpenSearch (we’ve back ported this functionality recently) - https://docs.openstack.org/kolla-ansible/yoga/reference/logging-and-monitori...
Best regards, Michal
On 3 May 2023, at 17:13, wodel youchi <wodel.youchi@gmail.com> wrote:
Hi,
I have finished the update of my openstack platform with newer containers.
While verifying I noticed that fluentd container keeps restarting.
In the log file I am having this :
2023-05-03 16:07:59 +0100 [error]: #0 config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Using Elasticsearch client 8.7.1 is not compatible for your Elasticsearch server. Please check your using elasticsearch gem version and Elasticsearch server." 2023-05-03 16:07:59 +0100 [error]: Worker 0 finished unexpectedly with status 2 2023-05-03 16:07:59 +0100 [info]: Received graceful stop
Those are the images I am using : (yogavenv) [deployer@rscdeployer ~]$ sudo docker images | grep -i elas
192.168.1.16:4000/openstack.kolla/centos-source-prometheus-elasticsearch-exporter yoga030523 b48f63ed0072 12 hours ago 539MB 192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch yoga030523 3558611b0cf4 12 hours ago 1.2GB 192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch-curator yoga030523 83a6b48339ea 12 hours ago 637MB
(yogavenv) [deployer@rscdeployer ~]$ sudo docker images | grep -i fluen 192.168.1.16:4000/openstack.kolla/centos-source-fluentd yoga030523 bf6596e139e2 12 hours ago 847MB
Any ideas?
Regards.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Hello, Kolla-Ansible is not supporting both opensearch and elasticsearch running at the same time - so if you’re using cloudkitty - it’s better to stick for Elasticsearch for now (CK does not support OpenSearch yet). I started working on the bug - will let you know in the bug report when a fix will be merged and images published. In the meantime you can try to uninstall the too-new elasticsearch gems using td-agent-gem uninstall in your running container image. Best regards, Michal
On 4 May 2023, at 14:33, wodel youchi <wodel.youchi@gmail.com> wrote:
Hi,
I'll try to open a bug for this.
I am using elasticsearch also with Cloudkitty : cloudkitty_storage_backend: "elasticsearch" instead of influxdb to get some HA. Will I still get the fluentd problem even if I migrate to Opensearch leaving Cloudkitty with elasticsearch???
Regards.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <x-msg://6/#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> Le jeu. 4 mai 2023 à 07:42, Michał Nasiadka <mnasiadka@gmail.com <mailto:mnasiadka@gmail.com>> a écrit :
Hello,
That probably is a Kolla bug - can you please raise a bug in launchpad.net <http://launchpad.net/>? The other alternative is to migrate to OpenSearch (we’ve back ported this functionality recently) - https://docs.openstack.org/kolla-ansible/yoga/reference/logging-and-monitori...
Best regards, Michal
On 3 May 2023, at 17:13, wodel youchi <wodel.youchi@gmail.com <mailto:wodel.youchi@gmail.com>> wrote:
Hi,
I have finished the update of my openstack platform with newer containers.
While verifying I noticed that fluentd container keeps restarting.
In the log file I am having this :
2023-05-03 16:07:59 +0100 [error]: #0 config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Using Elasticsearch client 8.7.1 is not compatible for your Elasticsearch server. Please check your using elasticsearch gem version and Elasticsearch server." 2023-05-03 16:07:59 +0100 [error]: Worker 0 finished unexpectedly with status 2 2023-05-03 16:07:59 +0100 [info]: Received graceful stop
Those are the images I am using : (yogavenv) [deployer@rscdeployer ~]$ sudo docker images | grep -i elas 192.168.1.16:4000/openstack.kolla/centos-source-prometheus-elasticsearch-exporter <http://192.168.1.16:4000/openstack.kolla/centos-source-prometheus-elasticsearch-exporter> yoga030523 b48f63ed0072 12 hours ago 539MB 192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch <http://192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch> yoga030523 3558611b0cf4 12 hours ago 1.2GB 192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch-curator <http://192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch-curator> yoga030523 83a6b48339ea 12 hours ago 637MB
(yogavenv) [deployer@rscdeployer ~]$ sudo docker images | grep -i fluen 192.168.1.16:4000/openstack.kolla/centos-source-fluentd <http://192.168.1.16:4000/openstack.kolla/centos-source-fluentd> yoga030523 bf6596e139e2 12 hours ago 847MB
Any ideas?
Regards.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Hi, I cannot access the fluentd container, it crashes almost instantly after each restart. Is there a workaround to make the container start and wait until the commands are executed? Regards. Le jeu. 4 mai 2023 à 14:12, Michał Nasiadka <mnasiadka@gmail.com> a écrit :
Hello,
Kolla-Ansible is not supporting both opensearch and elasticsearch running at the same time - so if you’re using cloudkitty - it’s better to stick for Elasticsearch for now (CK does not support OpenSearch yet).
I started working on the bug - will let you know in the bug report when a fix will be merged and images published. In the meantime you can try to uninstall the too-new elasticsearch gems using td-agent-gem uninstall in your running container image.
Best regards, Michal
On 4 May 2023, at 14:33, wodel youchi <wodel.youchi@gmail.com> wrote:
Hi,
I'll try to open a bug for this.
I am using elasticsearch also with Cloudkitty : cloudkitty_storage_backend: "elasticsearch" instead of influxdb to get some HA. Will I still get the fluentd problem even if I migrate to Opensearch leaving Cloudkitty with elasticsearch???
Regards.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Le jeu. 4 mai 2023 à 07:42, Michał Nasiadka <mnasiadka@gmail.com> a écrit :
Hello,
That probably is a Kolla bug - can you please raise a bug in launchpad.net? The other alternative is to migrate to OpenSearch (we’ve back ported this functionality recently) - https://docs.openstack.org/kolla-ansible/yoga/reference/logging-and-monitori...
Best regards, Michal
On 3 May 2023, at 17:13, wodel youchi <wodel.youchi@gmail.com> wrote:
Hi,
I have finished the update of my openstack platform with newer containers.
While verifying I noticed that fluentd container keeps restarting.
In the log file I am having this :
2023-05-03 16:07:59 +0100 [error]: #0 config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Using Elasticsearch client 8.7.1 is not compatible for your Elasticsearch server. Please check your using elasticsearch gem version and Elasticsearch server." 2023-05-03 16:07:59 +0100 [error]: Worker 0 finished unexpectedly with status 2 2023-05-03 16:07:59 +0100 [info]: Received graceful stop
Those are the images I am using : (yogavenv) [deployer@rscdeployer ~]$ sudo docker images | grep -i elas
192.168.1.16:4000/openstack.kolla/centos-source-prometheus-elasticsearch-exporter yoga030523 b48f63ed0072 12 hours ago 539MB 192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch yoga030523 3558611b0cf4 12 hours ago 1.2GB 192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch-curator yoga030523 83a6b48339ea 12 hours ago 637MB
(yogavenv) [deployer@rscdeployer ~]$ sudo docker images | grep -i fluen 192.168.1.16:4000/openstack.kolla/centos-source-fluentd yoga030523 bf6596e139e2 12 hours ago 847MB
Any ideas?
Regards.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Hello, Then probably the only solution is to rebuild the container image with this patch: https://review.opendev.org/c/openstack/kolla/+/882289 It will take a couple days before it’s merged upstream. Best regards, Michal
On 8 May 2023, at 14:06, wodel youchi <wodel.youchi@gmail.com> wrote:
Hi,
I cannot access the fluentd container, it crashes almost instantly after each restart. Is there a workaround to make the container start and wait until the commands are executed?
Regards.
Le jeu. 4 mai 2023 à 14:12, Michał Nasiadka <mnasiadka@gmail.com <mailto:mnasiadka@gmail.com>> a écrit :
Hello,
Kolla-Ansible is not supporting both opensearch and elasticsearch running at the same time - so if you’re using cloudkitty - it’s better to stick for Elasticsearch for now (CK does not support OpenSearch yet).
I started working on the bug - will let you know in the bug report when a fix will be merged and images published. In the meantime you can try to uninstall the too-new elasticsearch gems using td-agent-gem uninstall in your running container image.
Best regards, Michal
On 4 May 2023, at 14:33, wodel youchi <wodel.youchi@gmail.com <mailto:wodel.youchi@gmail.com>> wrote:
Hi,
I'll try to open a bug for this.
I am using elasticsearch also with Cloudkitty : cloudkitty_storage_backend: "elasticsearch" instead of influxdb to get some HA. Will I still get the fluentd problem even if I migrate to Opensearch leaving Cloudkitty with elasticsearch???
Regards.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Le jeu. 4 mai 2023 à 07:42, Michał Nasiadka <mnasiadka@gmail.com <mailto:mnasiadka@gmail.com>> a écrit :
Hello,
That probably is a Kolla bug - can you please raise a bug in launchpad.net <http://launchpad.net/>? The other alternative is to migrate to OpenSearch (we’ve back ported this functionality recently) - https://docs.openstack.org/kolla-ansible/yoga/reference/logging-and-monitori...
Best regards, Michal
On 3 May 2023, at 17:13, wodel youchi <wodel.youchi@gmail.com <mailto:wodel.youchi@gmail.com>> wrote:
Hi,
I have finished the update of my openstack platform with newer containers.
While verifying I noticed that fluentd container keeps restarting.
In the log file I am having this :
2023-05-03 16:07:59 +0100 [error]: #0 config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Using Elasticsearch client 8.7.1 is not compatible for your Elasticsearch server. Please check your using elasticsearch gem version and Elasticsearch server." 2023-05-03 16:07:59 +0100 [error]: Worker 0 finished unexpectedly with status 2 2023-05-03 16:07:59 +0100 [info]: Received graceful stop
Those are the images I am using : (yogavenv) [deployer@rscdeployer ~]$ sudo docker images | grep -i elas 192.168.1.16:4000/openstack.kolla/centos-source-prometheus-elasticsearch-exporter <http://192.168.1.16:4000/openstack.kolla/centos-source-prometheus-elasticsearch-exporter> yoga030523 b48f63ed0072 12 hours ago 539MB 192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch <http://192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch> yoga030523 3558611b0cf4 12 hours ago 1.2GB 192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch-curator <http://192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch-curator> yoga030523 83a6b48339ea 12 hours ago 637MB
(yogavenv) [deployer@rscdeployer ~]$ sudo docker images | grep -i fluen 192.168.1.16:4000/openstack.kolla/centos-source-fluentd <http://192.168.1.16:4000/openstack.kolla/centos-source-fluentd> yoga030523 bf6596e139e2 12 hours ago 847MB
Any ideas?
Regards.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Hi; I did update another update of my platforme, and still having problems with fluentd container (it is still restarting) but with different error :
*Running command: '/usr/sbin/td-agent -o /var/log/kolla/fluentd/fluentd.log'/opt/td-agent/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.3.0, because faraday-1.10.3 conflicts with faraday (>= 2.0.0), faraday-excon-1.1.0 conflicts with faraday-excon (>= 2.0.0) (Gem::ConflictError)* from /opt/td-agent/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate' from /opt/td-agent/lib/ruby/2.7.0/rubygems.rb:217:in `rescue in try_activate' from /opt/td-agent/lib/ruby/2.7.0/rubygems.rb:210:in `try_activate' from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require' from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:20:in `<top (required)>' from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require' from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:103:in `block in search' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:100:in `each' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:100:in `search' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:44:in `lookup' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:169:in `new_impl' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:114:in `new_output' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/multi_output.rb:108:in `block in configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/multi_output.rb:99:in `each' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/multi_output.rb:99:in `configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/out_copy.rb:39:in `configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:187:in `configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:132:in `add_match' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:74:in `block in configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in `each' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in `configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:149:in `configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:105:in `configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:80:in `run_configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:571:in `run_supervisor' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/command/fluentd.rb:352:in `<top (required)>' from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require' from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/bin/fluentd:15:in `<top (required)>' from /opt/td-agent/bin/fluentd:23:in `load' from /opt/td-agent/bin/fluentd:23:in `<top (required)>' from /usr/sbin/td-agent:15:in `load' from /usr/sbin/td-agent:15:in `<main>' */opt/td-agent/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.3.0, because faraday-1.10.3 conflicts with faraday (>= 2.0.0), faraday-excon-1.1.0 conflicts with faraday-excon (>= 2.0.0) (Gem::ConflictError)* from /opt/td-agent/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate' from /opt/td-agent/lib/ruby/2.7.0/rubygems.rb:211:in `try_activate' from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require' from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:20:in `<top (required)>' from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require' from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:103:in `block in search' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:100:in `each' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:100:in `search' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:44:in `lookup' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:169:in `new_impl' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:114:in `new_output' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/multi_output.rb:108:in `block in configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/multi_output.rb:99:in `each' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/multi_output.rb:99:in `configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/out_copy.rb:39:in `configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:187:in `configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:132:in `add_match' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:74:in `block in configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in `each' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in `configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:149:in `configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:105:in `configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:80:in `run_configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:571:in `run_supervisor' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/command/fluentd.rb:352:in `<top (required)>' from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require' from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/bin/fluentd:15:in `<top (required)>' from /opt/td-agent/bin/fluentd:23:in `load' from /opt/td-agent/bin/fluentd:23:in `<top (required)>' from /usr/sbin/td-agent:15:in `load' from /usr/sbin/td-agent:15:in `<main>'
*/opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require': cannot load such file -- fluent/log-ext (LoadError) from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:20:in `<top (required)>'* from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require' from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:103:in `block in search' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:100:in `each' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:100:in `search' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:44:in `lookup' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:169:in `new_impl' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:114:in `new_output' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/multi_output.rb:108:in `block in configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/multi_output.rb:99:in `each' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/multi_output.rb:99:in `configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/out_copy.rb:39:in `configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:187:in `configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:132:in `add_match' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:74:in `block in configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in `each' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in `configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:149:in `configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:105:in `configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:80:in `run_configure' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:571:in `run_supervisor' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/command/fluentd.rb:352:in `<top (required)>' from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require' from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require' from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/bin/fluentd:15:in `<top (required)>' from /opt/td-agent/bin/fluentd:23:in `load' from /opt/td-agent/bin/fluentd:23:in `<top (required)>' from /usr/sbin/td-agent:15:in `load' from /usr/sbin/td-agent:15:in `<main>'
Regards. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> Le lun. 8 mai 2023 à 14:59, Michał Nasiadka <mnasiadka@gmail.com> a écrit :
Hello,
Then probably the only solution is to rebuild the container image with this patch: https://review.opendev.org/c/openstack/kolla/+/882289 It will take a couple days before it’s merged upstream.
Best regards, Michal
On 8 May 2023, at 14:06, wodel youchi <wodel.youchi@gmail.com> wrote:
Hi,
I cannot access the fluentd container, it crashes almost instantly after each restart. Is there a workaround to make the container start and wait until the commands are executed?
Regards.
Le jeu. 4 mai 2023 à 14:12, Michał Nasiadka <mnasiadka@gmail.com> a écrit :
Hello,
Kolla-Ansible is not supporting both opensearch and elasticsearch running at the same time - so if you’re using cloudkitty - it’s better to stick for Elasticsearch for now (CK does not support OpenSearch yet).
I started working on the bug - will let you know in the bug report when a fix will be merged and images published. In the meantime you can try to uninstall the too-new elasticsearch gems using td-agent-gem uninstall in your running container image.
Best regards, Michal
On 4 May 2023, at 14:33, wodel youchi <wodel.youchi@gmail.com> wrote:
Hi,
I'll try to open a bug for this.
I am using elasticsearch also with Cloudkitty : cloudkitty_storage_backend: "elasticsearch" instead of influxdb to get some HA. Will I still get the fluentd problem even if I migrate to Opensearch leaving Cloudkitty with elasticsearch???
Regards.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Le jeu. 4 mai 2023 à 07:42, Michał Nasiadka <mnasiadka@gmail.com> a écrit :
Hello,
That probably is a Kolla bug - can you please raise a bug in launchpad.net? The other alternative is to migrate to OpenSearch (we’ve back ported this functionality recently) - https://docs.openstack.org/kolla-ansible/yoga/reference/logging-and-monitori...
Best regards, Michal
On 3 May 2023, at 17:13, wodel youchi <wodel.youchi@gmail.com> wrote:
Hi,
I have finished the update of my openstack platform with newer containers.
While verifying I noticed that fluentd container keeps restarting.
In the log file I am having this :
2023-05-03 16:07:59 +0100 [error]: #0 config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Using Elasticsearch client 8.7.1 is not compatible for your Elasticsearch server. Please check your using elasticsearch gem version and Elasticsearch server." 2023-05-03 16:07:59 +0100 [error]: Worker 0 finished unexpectedly with status 2 2023-05-03 16:07:59 +0100 [info]: Received graceful stop
Those are the images I am using : (yogavenv) [deployer@rscdeployer ~]$ sudo docker images | grep -i elas
192.168.1.16:4000/openstack.kolla/centos-source-prometheus-elasticsearch-exporter yoga030523 b48f63ed0072 12 hours ago 539MB 192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch yoga030523 3558611b0cf4 12 hours ago 1.2GB 192.168.1.16:4000/openstack.kolla/centos-source-elasticsearch-curator yoga030523 83a6b48339ea 12 hours ago 637MB
(yogavenv) [deployer@rscdeployer ~]$ sudo docker images | grep -i fluen 192.168.1.16:4000/openstack.kolla/centos-source-fluentd yoga030523 bf6596e139e2 12 hours ago 847MB
Any ideas?
Regards.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
participants (2)
-
Michał Nasiadka
-
wodel youchi