Support Questions

raghu7711 · ‎11-01-2017

Nimbus and Supervisor were stopping immediately after starting. i am getting error message as " org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts [hostname]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?"

Also i am not able to change the nimbus.seed to localhost.

Please help me.

raghu7711 · ‎11-03-2017

Hi @Jay Kumar SenSharma the problem was with nimbus.childopts and supervisor.chilopts.

i executed the below commands and restarted the storm services. Storm Nimbus and Supervisor started working.

For Nimbus:

/var/lib/ambari-server/resources/scripts/configs.sh set ambariserver.example.com test_cluster "nimbus.childopts" "'-Xmx1024m'"

For Supervisor:

/var/lib/ambari-server/resources/scripts/configs.sh set ambariserver.example.com test_cluster "supervisor.childopts" "'-Xmx1024m'"

The issue is resolved now.

Thanks for your valuable efforts.

View solution in original post

jsensharma · ‎11-02-2017

@Raghu Ramamoorthi

Regarding your query: "Is there any ways to change the port?"

You can try the same API call to change the port (6627) to something else that is not blocked:

# /var/lib/ambari-server/resources/scripts/configs.sh set ambariserver.example.com test_cluster storm-site "nimbus.thrift.port" "7727"

.

raghu7711 · ‎11-02-2017

Hi @Jay Kumar SenSharma i am able to change the port to 6667. But still the issue persists.

Not sure where i am getting wrong.

thank you

raghu7711 · ‎11-02-2017

Hi @Jay Kumar SenSharma I am getting the below error log while running the Storm service check

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/STORM/0.9.1/package/scripts/service_check.py", line 79, in <module>
    ServiceCheck().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/STORM/0.9.1/package/scripts/service_check.py", line 70, in service_check
    user=params.storm_user
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'storm jar /tmp/wordCount.jar storm.starter.WordCountTopology WordCountidb034d964_date410217' returned 1. Running: /usr/jdk64/jdk1.8.0_112/bin/java -server -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.6.3.0-235/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/hdp/2.6.3.0-235/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.3.0-235/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.3.0-235/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.3.0-235/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.3.0-235/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.3.0-235/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.3.0-235/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.3.0-235/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.3.0-235/storm/lib/storm-core-1.1.0.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/storm/lib/storm-rename-hack-1.1.0.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/storm/lib/zookeeper.jar:/usr/hdp/2.6.3.0-235/storm/lib/ambari-metrics-storm-sink.jar org.apache.storm.daemon.ClientJarTransformerRunner org.apache.storm.hack.StormShadeTransformer /tmp/wordCount.jar /tmp/f931c92ebfb111e78489000d3a92188e.jar
834  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/base/BaseBasicBolt to org/apache/storm/topology/base/BaseBasicBolt in storm/starter/BasicDRPCTopology$ExclaimBolt.class. please modify your code to use the new namespace
838  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/tuple/Tuple to org/apache/storm/tuple/Tuple in storm/starter/BasicDRPCTopology$ExclaimBolt.class. please modify your code to use the new namespace
838  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/BasicOutputCollector to org/apache/storm/topology/BasicOutputCollector in storm/starter/BasicDRPCTopology$ExclaimBolt.class. please modify your code to use the new namespace
838  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/tuple/Values to org/apache/storm/tuple/Values in storm/starter/BasicDRPCTopology$ExclaimBolt.class. please modify your code to use the new namespace
839  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/OutputFieldsDeclarer to org/apache/storm/topology/OutputFieldsDeclarer in storm/starter/BasicDRPCTopology$ExclaimBolt.class. please modify your code to use the new namespace
839  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/tuple/Fields to org/apache/storm/tuple/Fields in storm/starter/BasicDRPCTopology$ExclaimBolt.class. please modify your code to use the new namespace
839  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/drpc/LinearDRPCTopologyBuilder to org/apache/storm/drpc/LinearDRPCTopologyBuilder in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
840  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/IBasicBolt to org/apache/storm/topology/IBasicBolt in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
840  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/drpc/LinearDRPCInputDeclarer to org/apache/storm/drpc/LinearDRPCInputDeclarer in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
840  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/Config to org/apache/storm/Config in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
841  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/LocalDRPC to org/apache/storm/LocalDRPC in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
841  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/LocalCluster to org/apache/storm/LocalCluster in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
841  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/ILocalDRPC to org/apache/storm/ILocalDRPC in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
841  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/generated/StormTopology to org/apache/storm/generated/StormTopology in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
842  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/StormSubmitter to org/apache/storm/StormSubmitter in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
846  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/base/BaseRichBolt to org/apache/storm/topology/base/BaseRichBolt in storm/starter/bolt/RollingCountBolt.class. please modify your code to use the new namespace
846  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/task/OutputCollector to org/apache/storm/task/OutputCollector in storm/starter/bolt/RollingCountBolt.class. please modify your code to use the new namespace
846  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/task/TopologyContext to org/apache/storm/task/TopologyContext in storm/starter/bolt/RollingCountBolt.class. please modify your code to use the new namespace
848  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/utils/TimeCacheMap$ExpiredCallback to org/apache/storm/utils/TimeCacheMap$ExpiredCallback in storm/starter/bolt/SingleJoinBolt$ExpireCallback.class. please modify your code to use the new namespace
848  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/generated/GlobalStreamId to org/apache/storm/generated/GlobalStreamId in storm/starter/bolt/SingleJoinBolt$ExpireCallback.class. please modify your code to use the new namespace
849  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/utils/TimeCacheMap to org/apache/storm/utils/TimeCacheMap in storm/starter/bolt/SingleJoinBolt$ExpireCallback.class. please modify your code to use the new namespace
855  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/spout/ISpout to org/apache/storm/spout/ISpout in storm/starter/clj/word_count$sentence_spout__$fn$reify__23.class. please modify your code to use the new namespace
858  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/task/IBolt to org/apache/storm/task/IBolt in storm/starter/clj/word_count$split_sentence__$fn$reify__42.class. please modify your code to use the new namespace
894  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/TopologyBuilder to org/apache/storm/topology/TopologyBuilder in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
894  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/testing/TestWordSpout to org/apache/storm/testing/TestWordSpout in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
894  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/IRichSpout to org/apache/storm/topology/IRichSpout in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
894  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/SpoutDeclarer to org/apache/storm/topology/SpoutDeclarer in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
894  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/IRichBolt to org/apache/storm/topology/IRichBolt in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
894  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/BoltDeclarer to org/apache/storm/topology/BoltDeclarer in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
895  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/InputDeclarer to org/apache/storm/topology/InputDeclarer in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
895  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/utils/Utils to org/apache/storm/utils/Utils in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
896  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/drpc/DRPCSpout to org/apache/storm/drpc/DRPCSpout in storm/starter/ManualDRPC.class. please modify your code to use the new namespace
896  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/drpc/ReturnResults to org/apache/storm/drpc/ReturnResults in storm/starter/ManualDRPC.class. please modify your code to use the new namespace
897  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/base/BaseBatchBolt to org/apache/storm/topology/base/BaseBatchBolt in storm/starter/ReachTopology$CountAggregator.class. please modify your code to use the new namespace
897  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/coordination/BatchOutputCollector to org/apache/storm/coordination/BatchOutputCollector in storm/starter/ReachTopology$CountAggregator.class. please modify your code to use the new namespace
900  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/coordination/IBatchBolt to org/apache/storm/coordination/IBatchBolt in storm/starter/ReachTopology.class. please modify your code to use the new namespace
901  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/testing/FeederSpout to org/apache/storm/testing/FeederSpout in storm/starter/SingleJoinExample.class. please modify your code to use the new namespace
902  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/base/BaseRichSpout to org/apache/storm/topology/base/BaseRichSpout in storm/starter/spout/RandomSentenceSpout.class. please modify your code to use the new namespace
902  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/spout/SpoutOutputCollector to org/apache/storm/spout/SpoutOutputCollector in storm/starter/spout/RandomSentenceSpout.class. please modify your code to use the new namespace
903  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/utils/Time to org/apache/storm/utils/Time in storm/starter/tools/NthLastModifiedTimeTracker.class. please modify your code to use the new namespace
910  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/base/BaseTransactionalBolt to org/apache/storm/topology/base/BaseTransactionalBolt in storm/starter/TransactionalGlobalCount$UpdateGlobalCount.class. please modify your code to use the new namespace
910  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/transactional/ICommitter to org/apache/storm/transactional/ICommitter in storm/starter/TransactionalGlobalCount$UpdateGlobalCount.class. please modify your code to use the new namespace
910  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/transactional/TransactionAttempt to org/apache/storm/transactional/TransactionAttempt in storm/starter/TransactionalGlobalCount$UpdateGlobalCount.class. please modify your code to use the new namespace
911  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/testing/MemoryTransactionalSpout to org/apache/storm/testing/MemoryTransactionalSpout in storm/starter/TransactionalGlobalCount.class. please modify your code to use the new namespace
911  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/transactional/TransactionalTopologyBuilder to org/apache/storm/transactional/TransactionalTopologyBuilder in storm/starter/TransactionalGlobalCount.class. please modify your code to use the new namespace
912  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/transactional/partitioned/IPartitionedTransactionalSpout to org/apache/storm/transactional/partitioned/IPartitionedTransactionalSpout in storm/starter/TransactionalGlobalCount.class. please modify your code to use the new namespace
919  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/BaseFunction to org/apache/storm/trident/operation/BaseFunction in storm/starter/trident/TridentReach$ExpandList.class. please modify your code to use the new namespace
919  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/tuple/TridentTuple to org/apache/storm/trident/tuple/TridentTuple in storm/starter/trident/TridentReach$ExpandList.class. please modify your code to use the new namespace
919  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/TridentCollector to org/apache/storm/trident/operation/TridentCollector in storm/starter/trident/TridentReach$ExpandList.class. please modify your code to use the new namespace
920  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/CombinerAggregator to org/apache/storm/trident/operation/CombinerAggregator in storm/starter/trident/TridentReach$One.class. please modify your code to use the new namespace
920  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/state/StateFactory to org/apache/storm/trident/state/StateFactory in storm/starter/trident/TridentReach$StaticSingleKeyMapState$Factory.class. please modify your code to use the new namespace
920  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/task/IMetricsContext to org/apache/storm/task/IMetricsContext in storm/starter/trident/TridentReach$StaticSingleKeyMapState$Factory.class. please modify your code to use the new namespace
920  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/state/State to org/apache/storm/trident/state/State in storm/starter/trident/TridentReach$StaticSingleKeyMapState$Factory.class. please modify your code to use the new namespace
921  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/state/ReadOnlyState to org/apache/storm/trident/state/ReadOnlyState in storm/starter/trident/TridentReach$StaticSingleKeyMapState.class. please modify your code to use the new namespace
921  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/state/map/ReadOnlyMapState to org/apache/storm/trident/state/map/ReadOnlyMapState in storm/starter/trident/TridentReach$StaticSingleKeyMapState.class. please modify your code to use the new namespace
921  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/TridentTopology to org/apache/storm/trident/TridentTopology in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
922  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/TridentState to org/apache/storm/trident/TridentState in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
922  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/Stream to org/apache/storm/trident/Stream in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
922  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/builtin/MapGet to org/apache/storm/trident/operation/builtin/MapGet in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
922  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/state/QueryFunction to org/apache/storm/trident/state/QueryFunction in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
922  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/Function to org/apache/storm/trident/operation/Function in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
922  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/fluent/GroupedStream to org/apache/storm/trident/fluent/GroupedStream in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
922  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/builtin/Sum to org/apache/storm/trident/operation/builtin/Sum in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
924  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/testing/MemoryMapState$Factory to org/apache/storm/trident/testing/MemoryMapState$Factory in storm/starter/trident/TridentWordCount.class. please modify your code to use the new namespace
924  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/testing/MemoryMapState to org/apache/storm/trident/testing/MemoryMapState in storm/starter/trident/TridentWordCount.class. please modify your code to use the new namespace
924  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/testing/FixedBatchSpout to org/apache/storm/trident/testing/FixedBatchSpout in storm/starter/trident/TridentWordCount.class. please modify your code to use the new namespace
924  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/spout/IBatchSpout to org/apache/storm/trident/spout/IBatchSpout in storm/starter/trident/TridentWordCount.class. please modify your code to use the new namespace
925  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/builtin/Count to org/apache/storm/trident/operation/builtin/Count in storm/starter/trident/TridentWordCount.class. please modify your code to use the new namespace
925  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/builtin/FilterNull to org/apache/storm/trident/operation/builtin/FilterNull in storm/starter/trident/TridentWordCount.class. please modify your code to use the new namespace
925  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/Filter to org/apache/storm/trident/operation/Filter in storm/starter/trident/TridentWordCount.class. please modify your code to use the new namespace
927  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/task/ShellBolt to org/apache/storm/task/ShellBolt in storm/starter/WordCountTopology$SplitSentence.class. please modify your code to use the new namespace
Running: /usr/jdk64/jdk1.8.0_112/bin/java -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.6.3.0-235/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /usr/hdp/2.6.3.0-235/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.3.0-235/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.3.0-235/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.3.0-235/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.3.0-235/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.3.0-235/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.3.0-235/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.3.0-235/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.3.0-235/storm/lib/storm-core-1.1.0.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/storm/lib/storm-rename-hack-1.1.0.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/storm/lib/zookeeper.jar:/usr/hdp/2.6.3.0-235/storm/lib/ambari-metrics-storm-sink.jar:/tmp/f931c92ebfb111e78489000d3a92188e.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/2.6.3.0-235/storm/bin -Dstorm.jar=/tmp/f931c92ebfb111e78489000d3a92188e.jar -Dstorm.dependency.jars= -Dstorm.dependency.artifacts={} storm.starter.WordCountTopology WordCountidb034d964_date410217
603  [main] INFO  o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -4726212338922243552:-8562052777460553859
695  [main] WARN  o.a.s.u.NimbusClient - Ignoring exception while trying to get leader nimbus info from realtimevm.ey.com. will retry with a different seed host.
org.apache.storm.thrift.transport.TTransportException: null
	at org.apache.storm.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.generated.Nimbus$Client.recv_getLeader(Nimbus.java:1193) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.generated.Nimbus$Client.getLeader(Nimbus.java:1181) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:85) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:58) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.blobstore.NimbusBlobStore.prepare(NimbusBlobStore.java:268) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.StormSubmitter.getListOfKeysFromBlobStore(StormSubmitter.java:598) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.StormSubmitter.validateConfs(StormSubmitter.java:564) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:210) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:390) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:162) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at storm.starter.WordCountTopology.main(WordCountTopology.java:77) [f931c92ebfb111e78489000d3a92188e.jar:?]
Exception in thread "main" org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts [realtimevm.ey.com]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?
	at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:112)
	at org.apache.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:58)
	at org.apache.storm.blobstore.NimbusBlobStore.prepare(NimbusBlobStore.java:268)
	at org.apache.storm.StormSubmitter.getListOfKeysFromBlobStore(StormSubmitter.java:598)
	at org.apache.storm.StormSubmitter.validateConfs(StormSubmitter.java:564)
	at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:210)
	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:390)
	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:162)
	at storm.starter.WordCountTopology.main(WordCountTopology.java:77)

raghu7711 · ‎11-02-2017

Hi @Jay Kumar SenSharma,

The below log is found in /var/log/storm/nimbus.out

I guess the jar file (jmxetric-1.0.4.jar) is missing under the directory (/usr/hdp/current/storm-nimbus/contrib/storm-jmxetric/lib/). I checked in the path. It does not exists.

[root@realtimevm ~]# cat /var/log/storm/nimbus.out Running: /usr/jdk64/jdk1.8.0_112/bin/java -server -Ddaemon.name=nimbus -Dstorm.options= -Dstorm.home=/usr/hdp/2.6.3.0-235/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /usr/hdp/2.6.3.0-235/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.3.0-235/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.3.0-235/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.3.0-235/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.3.0-235/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.3.0-235/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.3.0-235/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.3.0-235/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.3.0-235/storm/lib/storm-core-1.1.0.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/storm/lib/storm-rename-hack-1.1.0.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/storm/lib/zookeeper.jar:/usr/hdp/2.6.3.0-235/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.6.3.0-235/storm/extlib-daemon/ojdbc6.jar:/usr/hdp/2.6.3.0-235/storm/extlib-daemon/ranger-plugin-classloader-0.7.0.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/storm/extlib-daemon/ranger-storm-plugin-shim-0.7.0.2.6.3.0-235.jar:/usr/hdp/current/storm-nimbus/conf -Xmx1024m -javaagent:/usr/hdp/current/storm-nimbus/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8649,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-nimbus/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Nimbus_JVM -Dlogfile.name=nimbus.log -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dlog4j.configurationFile=/usr/hdp/2.6.3.0-235/storm/log4j2/cluster.xml org.apache.storm.daemon.nimbus Error opening zip file or JAR manifest missing : /usr/hdp/current/storm-nimbus/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar Error occurred during initialization of VM agent library failed to init: instrument

Any help on this would be appreciated.