[kolla][monasca] thresh keeps dying
can someone direct me where to search for the cause? as I understand failing to submit the topology to storm cluster shouldn't be final, and the process just dissapers there... + exec /opt/storm/bin/storm jar /monasca-thresh-source/monasca-thresh-stable-victoria/thresh/target/monasca-thresh-2.4.0-SNAPSHOT-shaded.jar -Djava.io.tmpdir=/var/lib/monasca-thresh/data monasca.thresh.ThresholdingEngine /etc/monasca/thresh-config.yml monasca-thresh Running: /usr/lib/jvm/java-8-openjdk-amd64/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/opt/storm -Dstorm.log.dir=/var/log/kolla/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /opt/storm/*:/opt/storm/lib/*:/opt/storm/extlib/*:/monasca-thresh-source/monasca-thresh-stable-victoria/thresh/target/monasca-thresh-2.4.0-SNAPSHOT-shaded.jar:/opt/storm/conf:/opt/storm/bin -Dstorm.jar=/monasca-thresh-source/monasca-thresh-stable-victoria/thresh/target/monasca-thresh-2.4.0-SNAPSHOT-shaded.jar -Dstorm.dependency.jars= -Dstorm.dependency.artifacts={} -Djava.io.tmpdir=/var/lib/monasca-thresh/data monasca.thresh.ThresholdingEngine /etc/monasca/thresh-config.yml monasca-thresh 687 [main] INFO m.t.ThresholdingEngine - -------- Version Information -------- 692 [main] INFO m.t.ThresholdingEngine - monasca-thresh-2.4.0-SNAPSHOT-2021-06-06T08:32:08-${buildNumber} 693 [main] INFO m.t.ThresholdingEngine - Instantiating ThresholdingEngine with config file: /etc/monasca/thresh-config.yml, topology: monasca-thresh 1000 [main] INFO o.h.v.i.u.Version - HV000001: Hibernate Validator 5.2.1.Final 1197 [main] INFO m.t.ThresholdingEngine - local set to false 1312 [main] INFO m.t.i.t.MetricSpout - Created 1340 [main] INFO m.t.i.t.EventSpout - EventSpout created 1516 [main] WARN o.a.s.u.Utils - STORM-VERSION new 1.2.2 old null 1516 [main] INFO m.t.ThresholdingEngine - submitting topology monasca-thresh to non-local storm cluster 1549 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -7012431400424907995:-9108807134284416946 1728 [main] INFO o.a.s.u.NimbusClient - Found leader nimbus : storm1:6627 1751 [main] INFO o.a.s.s.a.AuthUtils - Got AutoCreds [] 1756 [main] INFO o.a.s.u.NimbusClient - Found leader nimbus : storm1:6627 Exception in thread "main" java.lang.RuntimeException: Topology with name `monasca-thresh` already exists on cluster at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:237) at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:387) at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:159) at monasca.thresh.ThresholdingEngine.run(ThresholdingEngine.java:111) at monasca.thresh.ThresholdingEngine.main(ThresholdingEngine.java:82) regards -- Tomasz Rutkowski
On 16/06/2021 09:15, Tomasz Rutkowski wrote:
can someone direct me where to search for the cause?
Have you seen this? https://bugs.launchpad.net/kolla-ansible/+bug/1808805 The current behaviour without that fix on a multi-node cluster is for only one of the thresh containers to run in local mode and the others continually restart.
as I understand failing to submit the topology to storm cluster shouldn't be final, and the process just dissapers there...
+ exec /opt/storm/bin/storm jar /monasca-thresh-source/monasca-thresh-stable-victoria/thresh/target/monasca-thresh-2.4.0-SNAPSHOT-shaded.jar -Djava.io.tmpdir=/var/lib/monasca-thresh/data monasca.thresh.ThresholdingEngine /etc/monasca/thresh-config.yml monasca-thresh Running: /usr/lib/jvm/java-8-openjdk-amd64/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/opt/storm -Dstorm.log.dir=/var/log/kolla/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /opt/storm/*:/opt/storm/lib/*:/opt/storm/extlib/*:/monasca-thresh-source/monasca-thresh-stable-victoria/thresh/target/monasca-thresh-2.4.0-SNAPSHOT-shaded.jar:/opt/storm/conf:/opt/storm/bin -Dstorm.jar=/monasca-thresh-source/monasca-thresh-stable-victoria/thresh/target/monasca-thresh-2.4.0-SNAPSHOT-shaded.jar -Dstorm.dependency.jars= -Dstorm.dependency.artifacts={} -Djava.io.tmpdir=/var/lib/monasca-thresh/data monasca.thresh.ThresholdingEngine /etc/monasca/thresh-config.yml monasca-thresh 687 [main] INFO m.t.ThresholdingEngine - -------- Version Information -------- 692 [main] INFO m.t.ThresholdingEngine - monasca-thresh-2.4.0-SNAPSHOT-2021-06-06T08:32:08-${buildNumber} 693 [main] INFO m.t.ThresholdingEngine - Instantiating ThresholdingEngine with config file: /etc/monasca/thresh-config.yml, topology: monasca-thresh 1000 [main] INFO o.h.v.i.u.Version - HV000001: Hibernate Validator 5.2.1.Final 1197 [main] INFO m.t.ThresholdingEngine - local set to false 1312 [main] INFO m.t.i.t.MetricSpout - Created 1340 [main] INFO m.t.i.t.EventSpout - EventSpout created 1516 [main] WARN o.a.s.u.Utils - STORM-VERSION new 1.2.2 old null 1516 [main] INFO m.t.ThresholdingEngine - submitting topology monasca-thresh to non-local storm cluster 1549 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -7012431400424907995:-9108807134284416946 1728 [main] INFO o.a.s.u.NimbusClient - Found leader nimbus : storm1:6627 1751 [main] INFO o.a.s.s.a.AuthUtils - Got AutoCreds [] 1756 [main] INFO o.a.s.u.NimbusClient - Found leader nimbus : storm1:6627 Exception in thread "main" java.lang.RuntimeException: Topology with name `monasca-thresh` already exists on cluster at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:237) at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:387) at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:159) at monasca.thresh.ThresholdingEngine.run(ThresholdingEngine.java:111) at monasca.thresh.ThresholdingEngine.main(ThresholdingEngine.java:82)
regards
W dniu pon, 28.06.2021 o godzinie 09∶28 +0100, użytkownik Doug napisał:
On 16/06/2021 09:15, Tomasz Rutkowski wrote:
can someone direct me where to search for the cause?
Have you seen this?
https://bugs.launchpad.net/kolla-ansible/+bug/1808805
continually restart.
thanks, I haven't found this, however I managed to overcome the problem with two changes (one mentioned there): 1. delete "local" from the end of the command (connects to storm) 2. change the remaining "monasca-thresh" to "thresh-cluster" (without that the topology name is present but with empty config) then the containers die as before, however the topology is put in storm cluster and everything (alarms so far ;)) works as expected regards -- Tomasz Rutkowski Dział Rozwoju Systemów
participants (2)
-
Doug
-
Tomasz Rutkowski