Storm Nimbus is not running in HDF cluster

[#1]

Nimbus and Supervisor were stopping immediately after starting. i am getting error message as " org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts [hostname]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?"

Also i am not able to change the nimbus.seed to localhost.

Please help me.

[#2]

@Raghu Ramamoorthi

Regarding your query: "i am not able to change the nimbus.seed to localhost"

You can try using the Ambari "config.sh" script to change the values like:

# /var/lib/ambari-server/resources/scripts/configs.sh set ambariserver.example.com test_cluster storm-site "nimbus.seeds" "['localhost']"

.

Here you will need to replace the following values in the above collamds:

ambariserver.examle.com = > replace with your ambari server hostname

test_cluster => with the name if your cluster.

['localhost'] => with valid nimbus seeds

.

Also please check if your Storm configuration is correct as per the doc: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.1.1/bk_command-line-installation/content/conf...

.

[#3]

Hi @Jay Kumar SenSharma Thanks for your quick look into this issue.

I tried the above said solution. But however i am getting the same issue.

I am using Windows Azure platform and opened 6627, 6700, 6701 ports.

Please help me.

[#4]

@Raghu Ramamoorthi

Can you please share the complete stack trace of the error ? May be sharing the log will be more useful.

Also please let us know if it is a secured environment?

[#5]

Hi @Jay Kumar SenSharma,

The cluster is not secured.

Please find the below stack trace while starting.

error.txt

[#6]

Hi @Jay Kumar SenSharma , now i am getting error as below.

org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts ["localhost"]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?
	at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:112)
	at org.apache.storm.ui.core$nimbus_summary.invoke(core.clj:413)
	at org.apache.storm.ui.core$fn__8094.invoke(core.clj:1120)
	at org.apache.storm.shade.compojure.core$make_route$fn__324.invoke(core.clj:100)
	at org.apache.storm.shade.compojure.core$if_route$fn__312.invoke(core.clj:46)
	at org.apache.storm.shade.compojure.core$if_method$fn__305.invoke(core.clj:31)
	at org.apache.storm.shade.compojure.core$routing$fn__330.invoke(core.clj:113)
	at clojure.core$some.invoke(core.clj:2570)
	at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:113)
	at clojure.lang.RestFn.applyTo(RestFn.java:139)
	at clojure.core$apply.invoke(core.clj:632)
	at org.apache.storm.shade.compojure.core$routes$fn__334.invoke(core.clj:118)
	at org.apache.storm.shade.ring.middleware.cors$wrap_cors$fn__7426.invoke(cors.clj:149)
	at org.apache.storm.shade.ring.middleware.json$wrap_json_params$fn__7373.invoke(json.clj:56)
	at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__2506.invoke(multipart_params.clj:118)
	at org.apache.storm.shade.ring.middleware.reload$wrap_reload$fn__6902.invoke(reload.clj:22)
	at org.apache.storm.ui.helpers$requests_middleware$fn__2775.invoke(helpers.clj:50)
	at org.apache.storm.ui.core$catch_errors$fn__8293.invoke(core.clj:1428)
	at org.apache.storm.shade.ring.middleware.keyword_params$wrap_keyword_params$fn__2426.invoke(keyword_params.clj:35)
	at org.apache.storm.shade.ring.middleware.nested_params$wrap_nested_params$fn__2469.invoke(nested_params.clj:84)
	at org.apache.storm.shade.ring.middleware.params$wrap_params$fn__2398.invoke(params.clj:64)
	at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__2506.invoke(multipart_params.clj:118)
	at org.apache.storm.shade.ring.middleware.flash$wrap_flash$fn__2721.invoke(flash.clj:35)
	at org.apache.storm.shade.ring.middleware.session$wrap_session$fn__2707.invoke(session.clj:98)
	at org.apache.storm.shade.ring.util.servlet$make_service_method$fn__2256.invoke(servlet.clj:127)
	at org.apache.storm.shade.ring.util.servlet$servlet$fn__2260.invoke(servlet.clj:136)
	at org.apache.storm.shade.ring.util.servlet.proxy$javax.servlet.http.HttpServlet$ff19274a.service(Unknown Source)
	at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:654)
	at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1320)
	at org.apache.storm.logging.filters.AccessLoggingFilter.handle(AccessLoggingFilter.java:47)
	at org.apache.storm.logging.filters.AccessLoggingFilter.doFilter(AccessLoggingFilter.java:39)
	at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
	at org.apache.storm.ui.helpers$x_frame_options_filter_handler$fn__2868.invoke(helpers.clj:189)
	at org.apache.storm.ui.helpers.proxy$java.lang.Object$Filter$abec9a8f.doFilter(Unknown Source)
	at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
	at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)
	at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)
	at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
	at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
	at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
	at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
	at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
	at org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369)
	at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
	at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933)
	at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995)
	at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
	at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
	at org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
	at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
	at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:745)

[#7]

@Raghu Ramamoorthi

You should specify the correct Nimbus hostname in "nimbus.seeds" instead of "localhost"

So please do SSH to your Nimbus host and then get the correct FQDN from the following command and then specify that hostname to your nimbus.seeds.

# hostname -f

.

The worker nodes need to know which machines are the candidate of master in order to download topology jars and confs.

So the worker nodes also have the correct "/etc/hosts" file entry to resolved the Nimbos hosts.

[#8]

@Raghu Ramamoorthi

In case of impersonation, May be you can check if the folllowing properties are setup correctly:

nimbus.impersonation.authorizer=org.apache.storm.security.auth.authorizer.ImpersonationAuthorizer
nimbus.impersonation.acl={ {{storm_bare_jaas_principal}} : {hosts: ['*'], groups: ['*']}}

Also if the ports of nimbus are accessible (I means there is no firewall issue and the nimbus host resolution is fine)
Like from a remote machine we can check if the Nimbus host & port are accessible?

From Any other cluster host:

# telnet  $Nimbus_Hostname  $Port
# nc -v  $Nimbus_Hostname  $Port

On Nimbus host,

Please verify if the port is accessible (firewall is disabled) , Hostname (FQDN) is correctly set and there are no errors on Nimbus logs.

# netstat -tnlpa | grep 6627
# service iptables status
# hostname -f

.

[#9]

@Jay Kumar SenSharma

The error is not because of impersonation.'

The error is due to " connection refused"

Telnet is passing for hostname 6667. But the nimbus.thrift.port is 6627.

Is there any ways to change the port. ?

Thanks for all your help 🙂

[#10]

Hi @Jay Kumar SenSharma I tried with the hostname also. But no luck .

Port 6627 is failing while trying to telnet with the nimbus host.

Also i am using a single node cluster on Azure VM. I purchased a RHEL 7.2 VM then install hdp and hdf on top of it.

thank you