question Re: Storm Nimbus is not running in HDF cluster in Support Questions

Storm Nimbus is not running in HDF cluster

raghu7711 — Wed, 01 Nov 2017 20:17:05 GMT

Nimbus and Supervisor were stopping immediately after starting. i am getting error message as " org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts [hostname]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?"

Also i am not able to change the nimbus.seed to localhost.

Please help me.

Re: Storm Nimbus is not running in HDF cluster

jsensharma — Wed, 01 Nov 2017 20:31:13 GMT

@Raghu Ramamoorthi

Regarding your query: "i am not able to change the nimbus.seed to localhost"

You can try using the Ambari "config.sh" script to change the values like:

# /var/lib/ambari-server/resources/scripts/configs.sh set ambariserver.example.com test_cluster storm-site "nimbus.seeds" "['localhost']"

Here you will need to replace the following values in the above collamds:

ambariserver.examle.com = > replace with your ambari server hostname

test_cluster => with the name if your cluster.

['localhost'] => with valid nimbus seeds

Also please check if your Storm configuration is correct as per the doc: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.1.1/bk_command-line-installation/content/configure_storm.html

Re: Storm Nimbus is not running in HDF cluster

raghu7711 — Thu, 02 Nov 2017 13:06:00 GMT

Hi @Jay Kumar SenSharma Thanks for your quick look into this issue.

I tried the above said solution. But however i am getting the same issue.

I am using Windows Azure platform and opened 6627, 6700, 6701 ports.

Please help me.

Re: Storm Nimbus is not running in HDF cluster

jsensharma — Thu, 02 Nov 2017 13:20:13 GMT

@Raghu Ramamoorthi

Can you please share the complete stack trace of the error ? May be sharing the log will be more useful.

Also please let us know if it is a secured environment?

Re: Storm Nimbus is not running in HDF cluster

raghu7711 — Thu, 02 Nov 2017 14:07:05 GMT

Hi @Jay Kumar SenSharma,

The cluster is not secured.

Please find the below stack trace while starting.

error.txt

Re: Storm Nimbus is not running in HDF cluster

raghu7711 — Thu, 02 Nov 2017 14:13:35 GMT

Hi @Jay Kumar SenSharma , now i am getting error as below.

org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts ["localhost"]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?
	at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:112)
	at org.apache.storm.ui.core$nimbus_summary.invoke(core.clj:413)
	at org.apache.storm.ui.core$fn__8094.invoke(core.clj:1120)
	at org.apache.storm.shade.compojure.core$make_route$fn__324.invoke(core.clj:100)
	at org.apache.storm.shade.compojure.core$if_route$fn__312.invoke(core.clj:46)
	at org.apache.storm.shade.compojure.core$if_method$fn__305.invoke(core.clj:31)
	at org.apache.storm.shade.compojure.core$routing$fn__330.invoke(core.clj:113)
	at clojure.core$some.invoke(core.clj:2570)
	at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:113)
	at clojure.lang.RestFn.applyTo(RestFn.java:139)
	at clojure.core$apply.invoke(core.clj:632)
	at org.apache.storm.shade.compojure.core$routes$fn__334.invoke(core.clj:118)
	at org.apache.storm.shade.ring.middleware.cors$wrap_cors$fn__7426.invoke(cors.clj:149)
	at org.apache.storm.shade.ring.middleware.json$wrap_json_params$fn__7373.invoke(json.clj:56)
	at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__2506.invoke(multipart_params.clj:118)
	at org.apache.storm.shade.ring.middleware.reload$wrap_reload$fn__6902.invoke(reload.clj:22)
	at org.apache.storm.ui.helpers$requests_middleware$fn__2775.invoke(helpers.clj:50)
	at org.apache.storm.ui.core$catch_errors$fn__8293.invoke(core.clj:1428)
	at org.apache.storm.shade.ring.middleware.keyword_params$wrap_keyword_params$fn__2426.invoke(keyword_params.clj:35)
	at org.apache.storm.shade.ring.middleware.nested_params$wrap_nested_params$fn__2469.invoke(nested_params.clj:84)
	at org.apache.storm.shade.ring.middleware.params$wrap_params$fn__2398.invoke(params.clj:64)
	at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__2506.invoke(multipart_params.clj:118)
	at org.apache.storm.shade.ring.middleware.flash$wrap_flash$fn__2721.invoke(flash.clj:35)
	at org.apache.storm.shade.ring.middleware.session$wrap_session$fn__2707.invoke(session.clj:98)
	at org.apache.storm.shade.ring.util.servlet$make_service_method$fn__2256.invoke(servlet.clj:127)
	at org.apache.storm.shade.ring.util.servlet$servlet$fn__2260.invoke(servlet.clj:136)
	at org.apache.storm.shade.ring.util.servlet.proxy$javax.servlet.http.HttpServlet$ff19274a.service(Unknown Source)
	at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:654)
	at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1320)
	at org.apache.storm.logging.filters.AccessLoggingFilter.handle(AccessLoggingFilter.java:47)
	at org.apache.storm.logging.filters.AccessLoggingFilter.doFilter(AccessLoggingFilter.java:39)
	at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
	at org.apache.storm.ui.helpers$x_frame_options_filter_handler$fn__2868.invoke(helpers.clj:189)
	at org.apache.storm.ui.helpers.proxy$java.lang.Object$Filter$abec9a8f.doFilter(Unknown Source)
	at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
	at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)
	at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)
	at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
	at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
	at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
	at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
	at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
	at org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369)
	at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
	at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933)
	at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995)
	at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
	at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
	at org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
	at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
	at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:745)

Re: Storm Nimbus is not running in HDF cluster

jsensharma — Thu, 02 Nov 2017 14:31:40 GMT

@Raghu Ramamoorthi

In case of impersonation, May be you can check if the folllowing properties are setup correctly:

nimbus.impersonation.authorizer=org.apache.storm.security.auth.authorizer.ImpersonationAuthorizer
nimbus.impersonation.acl={ {{storm_bare_jaas_principal}} : {hosts: ['*'], groups: ['*']}}

Also if the ports of nimbus are accessible (I means there is no firewall issue and the nimbus host resolution is fine)
Like from a remote machine we can check if the Nimbus host & port are accessible?

From Any other cluster host:

# telnet  $Nimbus_Hostname  $Port
# nc -v  $Nimbus_Hostname  $Port

On Nimbus host,

Please verify if the port is accessible (firewall is disabled) , Hostname (FQDN) is correctly set and there are no errors on Nimbus logs.

# netstat -tnlpa | grep 6627
# service iptables status
# hostname -f

Re: Storm Nimbus is not running in HDF cluster

jsensharma — Thu, 02 Nov 2017 14:33:05 GMT

@Raghu Ramamoorthi

You should specify the correct Nimbus hostname in "nimbus.seeds" instead of "localhost"

So please do SSH to your Nimbus host and then get the correct FQDN from the following command and then specify that hostname to your nimbus.seeds.

# hostname -f

The worker nodes need to know which machines are the candidate of master in order to download topology jars and confs.

So the worker nodes also have the correct "/etc/hosts" file entry to resolved the Nimbos hosts.

Re: Storm Nimbus is not running in HDF cluster

raghu7711 — Thu, 02 Nov 2017 14:56:24 GMT

@Jay Kumar SenSharma

The error is not because of impersonation.'

The error is due to " connection refused"

Telnet is passing for hostname 6667. But the nimbus.thrift.port is 6627.

Is there any ways to change the port. ?

Thanks for all your help 🙂

Re: Storm Nimbus is not running in HDF cluster

raghu7711 — Thu, 02 Nov 2017 15:02:11 GMT

Hi @Jay Kumar SenSharma I tried with the hostname also. But no luck .

Port 6627 is failing while trying to telnet with the nimbus host.

Also i am using a single node cluster on Azure VM. I purchased a RHEL 7.2 VM then install hdp and hdf on top of it.

thank you

Re: Storm Nimbus is not running in HDF cluster

jsensharma — Thu, 02 Nov 2017 15:23:47 GMT

@Raghu Ramamoorthi

Regarding your query: "Is there any ways to change the port?"

You can try the same API call to change the port (6627) to something else that is not blocked:

# /var/lib/ambari-server/resources/scripts/configs.sh set ambariserver.example.com test_cluster storm-site "nimbus.thrift.port" "7727"

Re: Storm Nimbus is not running in HDF cluster

raghu7711 — Thu, 02 Nov 2017 16:23:04 GMT

Hi @Jay Kumar SenSharma i am able to change the port to 6667. But still the issue persists.

Not sure where i am getting wrong.

thank you

Re: Storm Nimbus is not running in HDF cluster

raghu7711 — Thu, 02 Nov 2017 16:42:55 GMT

Hi @Jay Kumar SenSharma I am getting the below error log while running the Storm service check

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/STORM/0.9.1/package/scripts/service_check.py", line 79, in <module>
    ServiceCheck().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/STORM/0.9.1/package/scripts/service_check.py", line 70, in service_check
    user=params.storm_user
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'storm jar /tmp/wordCount.jar storm.starter.WordCountTopology WordCountidb034d964_date410217' returned 1. Running: /usr/jdk64/jdk1.8.0_112/bin/java -server -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.6.3.0-235/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/hdp/2.6.3.0-235/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.3.0-235/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.3.0-235/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.3.0-235/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.3.0-235/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.3.0-235/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.3.0-235/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.3.0-235/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.3.0-235/storm/lib/storm-core-1.1.0.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/storm/lib/storm-rename-hack-1.1.0.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/storm/lib/zookeeper.jar:/usr/hdp/2.6.3.0-235/storm/lib/ambari-metrics-storm-sink.jar org.apache.storm.daemon.ClientJarTransformerRunner org.apache.storm.hack.StormShadeTransformer /tmp/wordCount.jar /tmp/f931c92ebfb111e78489000d3a92188e.jar
834  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/base/BaseBasicBolt to org/apache/storm/topology/base/BaseBasicBolt in storm/starter/BasicDRPCTopology$ExclaimBolt.class. please modify your code to use the new namespace
838  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/tuple/Tuple to org/apache/storm/tuple/Tuple in storm/starter/BasicDRPCTopology$ExclaimBolt.class. please modify your code to use the new namespace
838  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/BasicOutputCollector to org/apache/storm/topology/BasicOutputCollector in storm/starter/BasicDRPCTopology$ExclaimBolt.class. please modify your code to use the new namespace
838  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/tuple/Values to org/apache/storm/tuple/Values in storm/starter/BasicDRPCTopology$ExclaimBolt.class. please modify your code to use the new namespace
839  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/OutputFieldsDeclarer to org/apache/storm/topology/OutputFieldsDeclarer in storm/starter/BasicDRPCTopology$ExclaimBolt.class. please modify your code to use the new namespace
839  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/tuple/Fields to org/apache/storm/tuple/Fields in storm/starter/BasicDRPCTopology$ExclaimBolt.class. please modify your code to use the new namespace
839  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/drpc/LinearDRPCTopologyBuilder to org/apache/storm/drpc/LinearDRPCTopologyBuilder in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
840  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/IBasicBolt to org/apache/storm/topology/IBasicBolt in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
840  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/drpc/LinearDRPCInputDeclarer to org/apache/storm/drpc/LinearDRPCInputDeclarer in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
840  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/Config to org/apache/storm/Config in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
841  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/LocalDRPC to org/apache/storm/LocalDRPC in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
841  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/LocalCluster to org/apache/storm/LocalCluster in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
841  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/ILocalDRPC to org/apache/storm/ILocalDRPC in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
841  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/generated/StormTopology to org/apache/storm/generated/StormTopology in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
842  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/StormSubmitter to org/apache/storm/StormSubmitter in storm/starter/BasicDRPCTopology.class. please modify your code to use the new namespace
846  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/base/BaseRichBolt to org/apache/storm/topology/base/BaseRichBolt in storm/starter/bolt/RollingCountBolt.class. please modify your code to use the new namespace
846  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/task/OutputCollector to org/apache/storm/task/OutputCollector in storm/starter/bolt/RollingCountBolt.class. please modify your code to use the new namespace
846  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/task/TopologyContext to org/apache/storm/task/TopologyContext in storm/starter/bolt/RollingCountBolt.class. please modify your code to use the new namespace
848  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/utils/TimeCacheMap$ExpiredCallback to org/apache/storm/utils/TimeCacheMap$ExpiredCallback in storm/starter/bolt/SingleJoinBolt$ExpireCallback.class. please modify your code to use the new namespace
848  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/generated/GlobalStreamId to org/apache/storm/generated/GlobalStreamId in storm/starter/bolt/SingleJoinBolt$ExpireCallback.class. please modify your code to use the new namespace
849  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/utils/TimeCacheMap to org/apache/storm/utils/TimeCacheMap in storm/starter/bolt/SingleJoinBolt$ExpireCallback.class. please modify your code to use the new namespace
855  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/spout/ISpout to org/apache/storm/spout/ISpout in storm/starter/clj/word_count$sentence_spout__$fn$reify__23.class. please modify your code to use the new namespace
858  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/task/IBolt to org/apache/storm/task/IBolt in storm/starter/clj/word_count$split_sentence__$fn$reify__42.class. please modify your code to use the new namespace
894  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/TopologyBuilder to org/apache/storm/topology/TopologyBuilder in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
894  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/testing/TestWordSpout to org/apache/storm/testing/TestWordSpout in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
894  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/IRichSpout to org/apache/storm/topology/IRichSpout in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
894  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/SpoutDeclarer to org/apache/storm/topology/SpoutDeclarer in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
894  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/IRichBolt to org/apache/storm/topology/IRichBolt in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
894  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/BoltDeclarer to org/apache/storm/topology/BoltDeclarer in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
895  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/InputDeclarer to org/apache/storm/topology/InputDeclarer in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
895  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/utils/Utils to org/apache/storm/utils/Utils in storm/starter/ExclamationTopology.class. please modify your code to use the new namespace
896  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/drpc/DRPCSpout to org/apache/storm/drpc/DRPCSpout in storm/starter/ManualDRPC.class. please modify your code to use the new namespace
896  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/drpc/ReturnResults to org/apache/storm/drpc/ReturnResults in storm/starter/ManualDRPC.class. please modify your code to use the new namespace
897  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/base/BaseBatchBolt to org/apache/storm/topology/base/BaseBatchBolt in storm/starter/ReachTopology$CountAggregator.class. please modify your code to use the new namespace
897  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/coordination/BatchOutputCollector to org/apache/storm/coordination/BatchOutputCollector in storm/starter/ReachTopology$CountAggregator.class. please modify your code to use the new namespace
900  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/coordination/IBatchBolt to org/apache/storm/coordination/IBatchBolt in storm/starter/ReachTopology.class. please modify your code to use the new namespace
901  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/testing/FeederSpout to org/apache/storm/testing/FeederSpout in storm/starter/SingleJoinExample.class. please modify your code to use the new namespace
902  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/base/BaseRichSpout to org/apache/storm/topology/base/BaseRichSpout in storm/starter/spout/RandomSentenceSpout.class. please modify your code to use the new namespace
902  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/spout/SpoutOutputCollector to org/apache/storm/spout/SpoutOutputCollector in storm/starter/spout/RandomSentenceSpout.class. please modify your code to use the new namespace
903  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/utils/Time to org/apache/storm/utils/Time in storm/starter/tools/NthLastModifiedTimeTracker.class. please modify your code to use the new namespace
910  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/topology/base/BaseTransactionalBolt to org/apache/storm/topology/base/BaseTransactionalBolt in storm/starter/TransactionalGlobalCount$UpdateGlobalCount.class. please modify your code to use the new namespace
910  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/transactional/ICommitter to org/apache/storm/transactional/ICommitter in storm/starter/TransactionalGlobalCount$UpdateGlobalCount.class. please modify your code to use the new namespace
910  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/transactional/TransactionAttempt to org/apache/storm/transactional/TransactionAttempt in storm/starter/TransactionalGlobalCount$UpdateGlobalCount.class. please modify your code to use the new namespace
911  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/testing/MemoryTransactionalSpout to org/apache/storm/testing/MemoryTransactionalSpout in storm/starter/TransactionalGlobalCount.class. please modify your code to use the new namespace
911  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/transactional/TransactionalTopologyBuilder to org/apache/storm/transactional/TransactionalTopologyBuilder in storm/starter/TransactionalGlobalCount.class. please modify your code to use the new namespace
912  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/transactional/partitioned/IPartitionedTransactionalSpout to org/apache/storm/transactional/partitioned/IPartitionedTransactionalSpout in storm/starter/TransactionalGlobalCount.class. please modify your code to use the new namespace
919  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/BaseFunction to org/apache/storm/trident/operation/BaseFunction in storm/starter/trident/TridentReach$ExpandList.class. please modify your code to use the new namespace
919  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/tuple/TridentTuple to org/apache/storm/trident/tuple/TridentTuple in storm/starter/trident/TridentReach$ExpandList.class. please modify your code to use the new namespace
919  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/TridentCollector to org/apache/storm/trident/operation/TridentCollector in storm/starter/trident/TridentReach$ExpandList.class. please modify your code to use the new namespace
920  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/CombinerAggregator to org/apache/storm/trident/operation/CombinerAggregator in storm/starter/trident/TridentReach$One.class. please modify your code to use the new namespace
920  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/state/StateFactory to org/apache/storm/trident/state/StateFactory in storm/starter/trident/TridentReach$StaticSingleKeyMapState$Factory.class. please modify your code to use the new namespace
920  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/task/IMetricsContext to org/apache/storm/task/IMetricsContext in storm/starter/trident/TridentReach$StaticSingleKeyMapState$Factory.class. please modify your code to use the new namespace
920  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/state/State to org/apache/storm/trident/state/State in storm/starter/trident/TridentReach$StaticSingleKeyMapState$Factory.class. please modify your code to use the new namespace
921  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/state/ReadOnlyState to org/apache/storm/trident/state/ReadOnlyState in storm/starter/trident/TridentReach$StaticSingleKeyMapState.class. please modify your code to use the new namespace
921  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/state/map/ReadOnlyMapState to org/apache/storm/trident/state/map/ReadOnlyMapState in storm/starter/trident/TridentReach$StaticSingleKeyMapState.class. please modify your code to use the new namespace
921  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/TridentTopology to org/apache/storm/trident/TridentTopology in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
922  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/TridentState to org/apache/storm/trident/TridentState in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
922  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/Stream to org/apache/storm/trident/Stream in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
922  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/builtin/MapGet to org/apache/storm/trident/operation/builtin/MapGet in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
922  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/state/QueryFunction to org/apache/storm/trident/state/QueryFunction in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
922  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/Function to org/apache/storm/trident/operation/Function in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
922  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/fluent/GroupedStream to org/apache/storm/trident/fluent/GroupedStream in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
922  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/builtin/Sum to org/apache/storm/trident/operation/builtin/Sum in storm/starter/trident/TridentReach.class. please modify your code to use the new namespace
924  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/testing/MemoryMapState$Factory to org/apache/storm/trident/testing/MemoryMapState$Factory in storm/starter/trident/TridentWordCount.class. please modify your code to use the new namespace
924  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/testing/MemoryMapState to org/apache/storm/trident/testing/MemoryMapState in storm/starter/trident/TridentWordCount.class. please modify your code to use the new namespace
924  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/testing/FixedBatchSpout to org/apache/storm/trident/testing/FixedBatchSpout in storm/starter/trident/TridentWordCount.class. please modify your code to use the new namespace
924  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/spout/IBatchSpout to org/apache/storm/trident/spout/IBatchSpout in storm/starter/trident/TridentWordCount.class. please modify your code to use the new namespace
925  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/builtin/Count to org/apache/storm/trident/operation/builtin/Count in storm/starter/trident/TridentWordCount.class. please modify your code to use the new namespace
925  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/builtin/FilterNull to org/apache/storm/trident/operation/builtin/FilterNull in storm/starter/trident/TridentWordCount.class. please modify your code to use the new namespace
925  [main] WARN  o.a.s.h.DefaultShader - Relocating storm/trident/operation/Filter to org/apache/storm/trident/operation/Filter in storm/starter/trident/TridentWordCount.class. please modify your code to use the new namespace
927  [main] WARN  o.a.s.h.DefaultShader - Relocating backtype/storm/task/ShellBolt to org/apache/storm/task/ShellBolt in storm/starter/WordCountTopology$SplitSentence.class. please modify your code to use the new namespace
Running: /usr/jdk64/jdk1.8.0_112/bin/java -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.6.3.0-235/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /usr/hdp/2.6.3.0-235/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.3.0-235/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.3.0-235/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.3.0-235/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.3.0-235/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.3.0-235/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.3.0-235/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.3.0-235/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.3.0-235/storm/lib/storm-core-1.1.0.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/storm/lib/storm-rename-hack-1.1.0.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/storm/lib/zookeeper.jar:/usr/hdp/2.6.3.0-235/storm/lib/ambari-metrics-storm-sink.jar:/tmp/f931c92ebfb111e78489000d3a92188e.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/2.6.3.0-235/storm/bin -Dstorm.jar=/tmp/f931c92ebfb111e78489000d3a92188e.jar -Dstorm.dependency.jars= -Dstorm.dependency.artifacts={} storm.starter.WordCountTopology WordCountidb034d964_date410217
603  [main] INFO  o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -4726212338922243552:-8562052777460553859
695  [main] WARN  o.a.s.u.NimbusClient - Ignoring exception while trying to get leader nimbus info from realtimevm.ey.com. will retry with a different seed host.
org.apache.storm.thrift.transport.TTransportException: null
	at org.apache.storm.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.generated.Nimbus$Client.recv_getLeader(Nimbus.java:1193) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.generated.Nimbus$Client.getLeader(Nimbus.java:1181) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:85) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:58) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.blobstore.NimbusBlobStore.prepare(NimbusBlobStore.java:268) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.StormSubmitter.getListOfKeysFromBlobStore(StormSubmitter.java:598) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.StormSubmitter.validateConfs(StormSubmitter.java:564) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:210) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:390) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:162) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
	at storm.starter.WordCountTopology.main(WordCountTopology.java:77) [f931c92ebfb111e78489000d3a92188e.jar:?]
Exception in thread "main" org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts [realtimevm.ey.com]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?
	at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:112)
	at org.apache.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:58)
	at org.apache.storm.blobstore.NimbusBlobStore.prepare(NimbusBlobStore.java:268)
	at org.apache.storm.StormSubmitter.getListOfKeysFromBlobStore(StormSubmitter.java:598)
	at org.apache.storm.StormSubmitter.validateConfs(StormSubmitter.java:564)
	at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:210)
	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:390)
	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:162)
	at storm.starter.WordCountTopology.main(WordCountTopology.java:77)

Re: Storm Nimbus is not running in HDF cluster

raghu7711 — Thu, 02 Nov 2017 21:40:34 GMT

Hi @Jay Kumar SenSharma,

The below log is found in /var/log/storm/nimbus.out

I guess the jar file (jmxetric-1.0.4.jar) is missing under the directory (/usr/hdp/current/storm-nimbus/contrib/storm-jmxetric/lib/). I checked in the path. It does not exists.

[root@realtimevm ~]# cat /var/log/storm/nimbus.out Running: /usr/jdk64/jdk1.8.0_112/bin/java -server -Ddaemon.name=nimbus -Dstorm.options= -Dstorm.home=/usr/hdp/2.6.3.0-235/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /usr/hdp/2.6.3.0-235/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.3.0-235/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.3.0-235/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.3.0-235/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.3.0-235/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.3.0-235/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.3.0-235/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.3.0-235/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.3.0-235/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.3.0-235/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.3.0-235/storm/lib/storm-core-1.1.0.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/storm/lib/storm-rename-hack-1.1.0.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/storm/lib/zookeeper.jar:/usr/hdp/2.6.3.0-235/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.6.3.0-235/storm/extlib-daemon/ojdbc6.jar:/usr/hdp/2.6.3.0-235/storm/extlib-daemon/ranger-plugin-classloader-0.7.0.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/storm/extlib-daemon/ranger-storm-plugin-shim-0.7.0.2.6.3.0-235.jar:/usr/hdp/current/storm-nimbus/conf -Xmx1024m -javaagent:/usr/hdp/current/storm-nimbus/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8649,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-nimbus/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Nimbus_JVM -Dlogfile.name=nimbus.log -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dlog4j.configurationFile=/usr/hdp/2.6.3.0-235/storm/log4j2/cluster.xml org.apache.storm.daemon.nimbus Error opening zip file or JAR manifest missing : /usr/hdp/current/storm-nimbus/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar Error occurred during initialization of VM agent library failed to init: instrument

Any help on this would be appreciated.

thank you

Re: Storm Nimbus is not running in HDF cluster

raghu7711 — Fri, 03 Nov 2017 18:04:45 GMT

Hi @Jay Kumar SenSharma the problem was with nimbus.childopts and supervisor.chilopts.

i executed the below commands and restarted the storm services. Storm Nimbus and Supervisor started working.

For Nimbus:

/var/lib/ambari-server/resources/scripts/configs.sh set ambariserver.example.com test_cluster "nimbus.childopts" "'-Xmx1024m'"

For Supervisor:

/var/lib/ambari-server/resources/scripts/configs.sh set ambariserver.example.com test_cluster "supervisor.childopts" "'-Xmx1024m'"

The issue is resolved now.

Thanks for your valuable efforts.

Re: Storm Nimbus is not running in HDF cluster

jsensharma — Fri, 03 Nov 2017 18:17:28 GMT

@Raghu Ramamoorthi

Wonderful finding Raghu, Thank you for sharing the solution.

Re: Storm Nimbus is not running in HDF cluster

yao_atian — Sat, 04 Nov 2017 13:10:02 GMT

Thank you very much!! I resolved this issue using your solution!