Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Storm Nimbus is not running in HDF cluster

avatar
Expert Contributor

Nimbus and Supervisor were stopping immediately after starting. i am getting error message as " org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts [hostname]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?"

Also i am not able to change the nimbus.seed to localhost.

Please help me.

1 ACCEPTED SOLUTION

avatar
Expert Contributor

Hi @Jay Kumar SenSharma the problem was with nimbus.childopts and supervisor.chilopts.

i executed the below commands and restarted the storm services. Storm Nimbus and Supervisor started working.

For Nimbus:

/var/lib/ambari-server/resources/scripts/configs.sh set ambariserver.example.com test_cluster "nimbus.childopts" "'-Xmx1024m'"

For Supervisor:

/var/lib/ambari-server/resources/scripts/configs.sh set ambariserver.example.com test_cluster "supervisor.childopts" "'-Xmx1024m'"

The issue is resolved now.

Thanks for your valuable efforts.

View solution in original post

16 REPLIES 16

avatar
Master Mentor

@Raghu Ramamoorthi

Regarding your query: "Is there any ways to change the port?"

You can try the same API call to change the port (6627) to something else that is not blocked:

# /var/lib/ambari-server/resources/scripts/configs.sh set ambariserver.example.com test_cluster storm-site "nimbus.thrift.port" "7727"

.

avatar
Expert Contributor

Hi @Jay Kumar SenSharma i am able to change the port to 6667. But still the issue persists.

Not sure where i am getting wrong.

thank you

avatar
Expert Contributor

Hi @Jay Kumar SenSharma I am getting the below error log while running the Storm service check

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/STORM/0.9.1/package/scripts/service_check.py", line 79, in <module>
    ServiceCheck().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/STORM/0.9.1/package/scripts/service_check.py", line 70, in service_check
    user=params.storm_user
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'storm jar /tmp/wordCount.jar storm.starter.WordCountTopology WordCountidb034d964_date410217' returned 1. Running: /usr/jdk64/jdk1.8.0_112/bin/java -server -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.6.3.0-235/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/hdp/2.6.3.0-235/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.3.0-235/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.3.0-235/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.3.0-235/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.3.0-235/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.3.0-235/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.3.0-235/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.3.0-235/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.3.0-235/storm/lib/storm-core-1.1.0.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/storm/lib/storm-rename-hack-1.1.0.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/storm/lib/zookeeper.jar:/usr/hdp/2.6.3.0-235/storm/lib/ambari-metrics-storm-sink.jar org.apache.storm.daemon.ClientJarTransformerRunner org.apache.storm.hack.StormShadeTransformer /tmp/wordCount.jar /tmp/f931c92ebfb111e78489000d3a92188e.jar
834  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/base/BaseBasicBolt to org/apache/storm/topology/base/BaseBasicBolt in storm/starter/BasicDRPCTopology$ExclaimBolt.class. please modify your code to use the new namespace
838  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/tuple/Tuple to org/apache/storm/tuple/Tuple in storm/starter/BasicDRPCTopology$ExclaimBolt.class. please modify your code to use the new namespace
838  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/BasicOutputCollector to org/apache/storm/topology/BasicOutputCollector in storm/starter/BasicDRPCTopology$ExclaimBolt.class. please modify your code to use the new namespace
838  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/tuple/Values to org/apache/storm/tuple/Values in storm/starter/BasicDRPCTopology$ExclaimBolt.class. please modify your code to use the new namespace
839  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/OutputFieldsDeclarer to org/apache/storm/topology/OutputFieldsDeclarer in storm/starter/BasicDRPCTopology$ExclaimBolt.class. please modify your code to use the new namespace
839  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/tuple/Fields to org/apache/storm/tuple/Fields in storm/starter/BasicDRPCTopology$ExclaimBolt.class. please modify your code to use the new namespace
839  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/drpc/LinearDRPCTopologyBuilder to org/apache/storm/drpc/LinearDRPCTopologyBuilder in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
840  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/IBasicBolt to org/apache/storm/topology/IBasicBolt in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
840  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/drpc/LinearDRPCInputDeclarer to org/apache/storm/drpc/LinearDRPCInputDeclarer in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
840  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/Config to org/apache/storm/Config in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
841  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/LocalDRPC to org/apache/storm/LocalDRPC in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
841  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/LocalCluster to org/apache/storm/LocalCluster in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
841  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/ILocalDRPC to org/apache/storm/ILocalDRPC in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
841  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/generated/StormTopology to org/apache/storm/generated/StormTopology in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
842  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/StormSubmitter to org/apache/storm/StormSubmitter in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
846  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/base/BaseRichBolt to org/apache/storm/topology/base/BaseRichBolt in storm/starter/bolt/RollingCountBolt.class. please modify your code to use the new namespace
846  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/task/OutputCollector to org/apache/storm/task/OutputCollector in storm/starter/bolt/RollingCountBolt.class. please modify your code to use the new namespace
846  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/task/TopologyContext to org/apache/storm/task/TopologyContext in storm/starter/bolt/RollingCountBolt.class. please modify your code to use the new namespace
848  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/utils/TimeCacheMap$ExpiredCallback to org/apache/storm/utils/TimeCacheMap$ExpiredCallback in storm/starter/bolt/SingleJoinBolt$ExpireCallback.class. please modify your code to use the new namespace
848  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/generated/GlobalStreamId to org/apache/storm/generated/GlobalStreamId in storm/starter/bolt/SingleJoinBolt$ExpireCallback.class. please modify your code to use the new namespace
849  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/utils/TimeCacheMap to org/apache/storm/utils/TimeCacheMap in storm/starter/bolt/SingleJoinBolt$ExpireCallback.class. please modify your code to use the new namespace
855  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/spout/ISpout to org/apache/storm/spout/ISpout in storm/starter/clj/word_count$sentence_spout__$fn$reify__23.class. please modify your code to use the new namespace
858  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/task/IBolt to org/apache/storm/task/IBolt in storm/starter/clj/word_count$split_sentence__$fn$reify__42.class. please modify your code to use the new namespace
894  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/TopologyBuilder to org/apache/storm/topology/TopologyBuilder in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
894  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/testing/TestWordSpout to org/apache/storm/testing/TestWordSpout in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
894  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/IRichSpout to org/apache/storm/topology/IRichSpout in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
894  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/SpoutDeclarer to org/apache/storm/topology/SpoutDeclarer in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
894  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/IRichBolt to org/apache/storm/topology/IRichBolt in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
894  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/BoltDeclarer to org/apache/storm/topology/BoltDeclarer in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
895  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/InputDeclarer to org/apache/storm/topology/InputDeclarer in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
895  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/utils/Utils to org/apache/storm/utils/Utils in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
896  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/drpc/DRPCSpout to org/apache/storm/drpc/DRPCSpout in storm/starter/ManualDRPC.class. please modify your code to use the new namespace
896  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/drpc/ReturnResults to org/apache/storm/drpc/ReturnResults in storm/starter/ManualDRPC.class. please modify your code to use the new namespace
897  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/base/BaseBatchBolt to org/apache/storm/topology/base/BaseBatchBolt in storm/starter/ReachTopology$CountAggregator.class. please modify your code to use the new namespace
897  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/coordination/BatchOutputCollector to org/apache/storm/coordination/BatchOutputCollector in storm/starter/ReachTopology$CountAggregator.class. please modify your code to use the new namespace
900  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/coordination/IBatchBolt to org/apache/storm/coordination/IBatchBolt in storm/starter/ReachTopology.class. please modify your code to use the new namespace
901  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/testing/FeederSpout to org/apache/storm/testing/FeederSpout in storm/starter/SingleJoinExample.class. please modify your code to use the new namespace
902  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/base/BaseRichSpout to org/apache/storm/topology/base/BaseRichSpout in storm/starter/spout/RandomSentenceSpout.class. please modify your code to use the new namespace
902  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/spout/SpoutOutputCollector to org/apache/storm/spout/SpoutOutputCollector in storm/starter/spout/RandomSentenceSpout.class. please modify your code to use the new namespace
903  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/utils/Time to org/apache/storm/utils/Time in storm/starter/tools/NthLastModifiedTimeTracker.class. please modify your code to use the new namespace
910  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/base/BaseTransactionalBolt to org/apache/storm/topology/base/BaseTransactionalBolt in storm/starter/TransactionalGlobalCount$UpdateGlobalCount.class. please modify your code to use the new namespace
910  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/transactional/ICommitter to org/apache/storm/transactional/ICommitter in storm/starter/TransactionalGlobalCount$UpdateGlobalCount.class. please modify your code to use the new namespace
910  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/transactional/TransactionAttempt to org/apache/storm/transactional/TransactionAttempt in storm/starter/TransactionalGlobalCount$UpdateGlobalCount.class. please modify your code to use the new namespace
911  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/testing/MemoryTransactionalSpout to org/apache/storm/testing/MemoryTransactionalSpout in storm/starter/TransactionalGlobalCount.class. please modify your code to use the new namespace
911  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/transactional/TransactionalTopologyBuilder to org/apache/storm/transactional/TransactionalTopologyBuilder in storm/starter/TransactionalGlobalCount.class. please modify your code to use the new namespace
912  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/transactional/partitioned/IPartitionedTransactionalSpout to org/apache/storm/transactional/partitioned/IPartitionedTransactionalSpout in storm/starter/TransactionalGlobalCount.class. please modify your code to use the new namespace
919  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/BaseFunction to org/apache/storm/trident/operation/BaseFunction in storm/starter/trident/TridentReach$ExpandList.class. please modify your code to use the new namespace
919  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/tuple/TridentTuple to org/apache/storm/trident/tuple/TridentTuple in storm/starter/trident/TridentReach$ExpandList.class. please modify your code to use the new namespace
919  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/TridentCollector to org/apache/storm/trident/operation/TridentCollector in storm/starter/trident/TridentReach$ExpandList.class. please modify your code to use the new namespace
920  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/CombinerAggregator to org/apache/storm/trident/operation/CombinerAggregator in storm/starter/trident/TridentReach$One.class. please modify your code to use the new namespace
920  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/state/StateFactory to org/apache/storm/trident/state/StateFactory in storm/starter/trident/TridentReach$StaticSingleKeyMapState$Factory.class. please modify your code to use the new namespace
920  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/task/IMetricsContext to org/apache/storm/task/IMetricsContext in storm/starter/trident/TridentReach$StaticSingleKeyMapState$Factory.class. please modify your code to use the new namespace
920  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/state/State to org/apache/storm/trident/state/State in storm/starter/trident/TridentReach$StaticSingleKeyMapState$Factory.class. please modify your code to use the new namespace
921  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/state/ReadOnlyState to org/apache/storm/trident/state/ReadOnlyState in storm/starter/trident/TridentReach$StaticSingleKeyMapState.class. please modify your code to use the new namespace
921  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/state/map/ReadOnlyMapState to org/apache/storm/trident/state/map/ReadOnlyMapState in storm/starter/trident/TridentReach$StaticSingleKeyMapState.class. please modify your code to use the new namespace
921  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/TridentTopology to org/apache/storm/trident/TridentTopology in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
922  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/TridentState to org/apache/storm/trident/TridentState in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
922  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/Stream to org/apache/storm/trident/Stream in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
922  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/builtin/MapGet to org/apache/storm/trident/operation/builtin/MapGet in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
922  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/state/QueryFunction to org/apache/storm/trident/state/QueryFunction in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
922  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/Function to org/apache/storm/trident/operation/Function in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
922  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/fluent/GroupedStream to org/apache/storm/trident/fluent/GroupedStream in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
922  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/builtin/Sum to org/apache/storm/trident/operation/builtin/Sum in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
924  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/testing/MemoryMapState$Factory to org/apache/storm/trident/testing/MemoryMapState$Factory in storm/starter/trident/TridentWordCount.class. please modify your code to use the new namespace
924  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/testing/MemoryMapState to org/apache/storm/trident/testing/MemoryMapState in storm/starter/trident/TridentWordCount.class. please modify your code to use the new namespace
924  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/testing/FixedBatchSpout to org/apache/storm/trident/testing/FixedBatchSpout in storm/starter/trident/TridentWordCount.class. please modify your code to use the new namespace
924  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/spout/IBatchSpout to org/apache/storm/trident/spout/IBatchSpout in storm/starter/trident/TridentWordCount.class. please modify your code to use the new namespace
925  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/builtin/Count to org/apache/storm/trident/operation/builtin/Count in storm/starter/trident/TridentWordCount.class. please modify your code to use the new namespace
925  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/builtin/FilterNull to org/apache/storm/trident/operation/builtin/FilterNull in storm/starter/trident/TridentWordCount.class. please modify your code to use the new namespace
925  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/Filter to org/apache/storm/trident/operation/Filter in storm/starter/trident/TridentWordCount.class. please modify your code to use the new namespace
927  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/task/ShellBolt to org/apache/storm/task/ShellBolt in storm/starter/WordCountTopology$SplitSentence.class. please modify your code to use the new namespace
Running: /usr/jdk64/jdk1.8.0_112/bin/java -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.6.3.0-235/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /usr/hdp/2.6.3.0-235/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.3.0-235/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.3.0-235/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.3.0-235/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.3.0-235/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.3.0-235/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.3.0-235/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.3.0-235/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.3.0-235/storm/lib/storm-core-1.1.0.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/storm/lib/storm-rename-hack-1.1.0.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/storm/lib/zookeeper.jar:/usr/hdp/2.6.3.0-235/storm/lib/ambari-metrics-storm-sink.jar:/tmp/f931c92ebfb111e78489000d3a92188e.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/2.6.3.0-235/storm/bin -Dstorm.jar=/tmp/f931c92ebfb111e78489000d3a92188e.jar -Dstorm.dependency.jars= -Dstorm.dependency.artifacts={} storm.starter.WordCountTopology WordCountidb034d964_date410217
603  [main] INFO  o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -4726212338922243552:-8562052777460553859
695  [main] WARN  o.a.s.u.NimbusClient - Ignoring exception while trying to get leader nimbus info from realtimevm.ey.com. will retry with a different seed host.
org.apache.storm.thrift.transport.TTransportException: null
	at org.apache.storm.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.generated.Nimbus$Client.recv_getLeader(Nimbus.java:1193) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.generated.Nimbus$Client.getLeader(Nimbus.java:1181) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:85) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:58) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.blobstore.NimbusBlobStore.prepare(NimbusBlobStore.java:268) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.StormSubmitter.getListOfKeysFromBlobStore(StormSubmitter.java:598) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.StormSubmitter.validateConfs(StormSubmitter.java:564) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:210) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:390) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:162) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at storm.starter.WordCountTopology.main(WordCountTopology.java:77) [f931c92ebfb111e78489000d3a92188e.jar:?]
Exception in thread "main" org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts [realtimevm.ey.com]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?
	at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:112)
	at org.apache.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:58)
	at org.apache.storm.blobstore.NimbusBlobStore.prepare(NimbusBlobStore.java:268)
	at org.apache.storm.StormSubmitter.getListOfKeysFromBlobStore(StormSubmitter.java:598)
	at org.apache.storm.StormSubmitter.validateConfs(StormSubmitter.java:564)
	at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:210)
	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:390)
	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:162)
	at storm.starter.WordCountTopology.main(WordCountTopology.java:77)

avatar
Expert Contributor

Hi @Jay Kumar SenSharma,

The below log is found in /var/log/storm/nimbus.out

I guess the jar file (jmxetric-1.0.4.jar) is missing under the directory (/usr/hdp/current/storm-nimbus/contrib/storm-jmxetric/lib/). I checked in the path. It does not exists.

[root@realtimevm ~]# cat /var/log/storm/nimbus.out Running: /usr/jdk64/jdk1.8.0_112/bin/java -server -Ddaemon.name=nimbus -Dstorm.options= -Dstorm.home=/usr/hdp/2.6.3.0-235/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /usr/hdp/2.6.3.0-235/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.3.0-235/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.3.0-235/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.3.0-235/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.3.0-235/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.3.0-235/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.3.0-235/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.3.0-235/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.3.0-235/storm/lib/storm-core-1.1.0.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/storm/lib/storm-rename-hack-1.1.0.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/storm/lib/zookeeper.jar:/usr/hdp/2.6.3.0-235/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.6.3.0-235/storm/extlib-daemon/ojdbc6.jar:/usr/hdp/2.6.3.0-235/storm/extlib-daemon/ranger-plugin-classloader-0.7.0.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/storm/extlib-daemon/ranger-storm-plugin-shim-0.7.0.2.6.3.0-235.jar:/usr/hdp/current/storm-nimbus/conf -Xmx1024m -javaagent:/usr/hdp/current/storm-nimbus/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8649,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-nimbus/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Nimbus_JVM -Dlogfile.name=nimbus.log -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dlog4j.configurationFile=/usr/hdp/2.6.3.0-235/storm/log4j2/cluster.xml org.apache.storm.daemon.nimbus Error opening zip file or JAR manifest missing : /usr/hdp/current/storm-nimbus/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar Error occurred during initialization of VM agent library failed to init: instrument

Any help on this would be appreciated.

thank you

avatar
Expert Contributor

Hi @Jay Kumar SenSharma the problem was with nimbus.childopts and supervisor.chilopts.

i executed the below commands and restarted the storm services. Storm Nimbus and Supervisor started working.

For Nimbus:

/var/lib/ambari-server/resources/scripts/configs.sh set ambariserver.example.com test_cluster "nimbus.childopts" "'-Xmx1024m'"

For Supervisor:

/var/lib/ambari-server/resources/scripts/configs.sh set ambariserver.example.com test_cluster "supervisor.childopts" "'-Xmx1024m'"

The issue is resolved now.

Thanks for your valuable efforts.

avatar
Master Mentor

@Raghu Ramamoorthi

Wonderful finding Raghu, Thank you for sharing the solution.

avatar
New Contributor

Thank you very much!! I resolved this issue using your solution!