Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Storm Nimbus is not running in HDF cluster

avatar
Expert Contributor

Nimbus and Supervisor were stopping immediately after starting. i am getting error message as " org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts [hostname]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?"

Also i am not able to change the nimbus.seed to localhost.

Please help me.

1 ACCEPTED SOLUTION

avatar
Expert Contributor

Hi @Jay Kumar SenSharma the problem was with nimbus.childopts and supervisor.chilopts.

i executed the below commands and restarted the storm services. Storm Nimbus and Supervisor started working.

For Nimbus:

/var/lib/ambari-server/resources/scripts/configs.sh set ambariserver.example.com test_cluster "nimbus.childopts" "'-Xmx1024m'"

For Supervisor:

/var/lib/ambari-server/resources/scripts/configs.sh set ambariserver.example.com test_cluster "supervisor.childopts" "'-Xmx1024m'"

The issue is resolved now.

Thanks for your valuable efforts.

View solution in original post

16 REPLIES 16

avatar
Master Mentor

@Raghu Ramamoorthi

Regarding your query: "i am not able to change the nimbus.seed to localhost"

You can try using the Ambari "config.sh" script to change the values like:

# /var/lib/ambari-server/resources/scripts/configs.sh set ambariserver.example.com test_cluster storm-site "nimbus.seeds" "['localhost']"

.

Here you will need to replace the following values in the above collamds:

ambariserver.examle.com = > replace with your ambari server hostname

test_cluster => with the name if your cluster.

['localhost'] => with valid nimbus seeds

.

Also please check if your Storm configuration is correct as per the doc: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.1.1/bk_command-line-installation/content/conf...

.

avatar
Expert Contributor

Hi @Jay Kumar SenSharma Thanks for your quick look into this issue.

I tried the above said solution. But however i am getting the same issue.

I am using Windows Azure platform and opened 6627, 6700, 6701 ports.

Please help me.

avatar
Master Mentor

@Raghu Ramamoorthi

Can you please share the complete stack trace of the error ? May be sharing the log will be more useful.

Also please let us know if it is a secured environment?

avatar
Expert Contributor

Hi @Jay Kumar SenSharma,

The cluster is not secured.

Please find the below stack trace while starting.

error.txt

avatar
Expert Contributor

Hi @Jay Kumar SenSharma , now i am getting error as below.

org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts ["localhost"]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?
	at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:112)
	at org.apache.storm.ui.core$nimbus_summary.invoke(core.clj:413)
	at org.apache.storm.ui.core$fn__8094.invoke(core.clj:1120)
	at org.apache.storm.shade.compojure.core$make_route$fn__324.invoke(core.clj:100)
	at org.apache.storm.shade.compojure.core$if_route$fn__312.invoke(core.clj:46)
	at org.apache.storm.shade.compojure.core$if_method$fn__305.invoke(core.clj:31)
	at org.apache.storm.shade.compojure.core$routing$fn__330.invoke(core.clj:113)
	at clojure.core$some.invoke(core.clj:2570)
	at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:113)
	at clojure.lang.RestFn.applyTo(RestFn.java:139)
	at clojure.core$apply.invoke(core.clj:632)
	at org.apache.storm.shade.compojure.core$routes$fn__334.invoke(core.clj:118)
	at org.apache.storm.shade.ring.middleware.cors$wrap_cors$fn__7426.invoke(cors.clj:149)
	at org.apache.storm.shade.ring.middleware.json$wrap_json_params$fn__7373.invoke(json.clj:56)
	at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__2506.invoke(multipart_params.clj:118)
	at org.apache.storm.shade.ring.middleware.reload$wrap_reload$fn__6902.invoke(reload.clj:22)
	at org.apache.storm.ui.helpers$requests_middleware$fn__2775.invoke(helpers.clj:50)
	at org.apache.storm.ui.core$catch_errors$fn__8293.invoke(core.clj:1428)
	at org.apache.storm.shade.ring.middleware.keyword_params$wrap_keyword_params$fn__2426.invoke(keyword_params.clj:35)
	at org.apache.storm.shade.ring.middleware.nested_params$wrap_nested_params$fn__2469.invoke(nested_params.clj:84)
	at org.apache.storm.shade.ring.middleware.params$wrap_params$fn__2398.invoke(params.clj:64)
	at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__2506.invoke(multipart_params.clj:118)
	at org.apache.storm.shade.ring.middleware.flash$wrap_flash$fn__2721.invoke(flash.clj:35)
	at org.apache.storm.shade.ring.middleware.session$wrap_session$fn__2707.invoke(session.clj:98)
	at org.apache.storm.shade.ring.util.servlet$make_service_method$fn__2256.invoke(servlet.clj:127)
	at org.apache.storm.shade.ring.util.servlet$servlet$fn__2260.invoke(servlet.clj:136)
	at org.apache.storm.shade.ring.util.servlet.proxy$javax.servlet.http.HttpServlet$ff19274a.service(Unknown Source)
	at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:654)
	at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1320)
	at org.apache.storm.logging.filters.AccessLoggingFilter.handle(AccessLoggingFilter.java:47)
	at org.apache.storm.logging.filters.AccessLoggingFilter.doFilter(AccessLoggingFilter.java:39)
	at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
	at org.apache.storm.ui.helpers$x_frame_options_filter_handler$fn__2868.invoke(helpers.clj:189)
	at org.apache.storm.ui.helpers.proxy$java.lang.Object$Filter$abec9a8f.doFilter(Unknown Source)
	at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
	at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)
	at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)
	at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
	at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
	at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
	at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
	at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
	at org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369)
	at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
	at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933)
	at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995)
	at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
	at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
	at org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
	at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
	at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:745)

avatar
Master Mentor

@Raghu Ramamoorthi

You should specify the correct Nimbus hostname in "nimbus.seeds" instead of "localhost"

So please do SSH to your Nimbus host and then get the correct FQDN from the following command and then specify that hostname to your nimbus.seeds.

# hostname -f

.

The worker nodes need to know which machines are the candidate of master in order to download topology jars and confs.

So the worker nodes also have the correct "/etc/hosts" file entry to resolved the Nimbos hosts.


avatar
Master Mentor

@Raghu Ramamoorthi

In case of impersonation, May be you can check if the folllowing properties are setup correctly:

nimbus.impersonation.authorizer=org.apache.storm.security.auth.authorizer.ImpersonationAuthorizer
nimbus.impersonation.acl={ {{storm_bare_jaas_principal}} : {hosts: ['*'], groups: ['*']}}



Also if the ports of nimbus are accessible (I means there is no firewall issue and the nimbus host resolution is fine)
Like from a remote machine we can check if the Nimbus host & port are accessible?

From Any other cluster host:

# telnet  $Nimbus_Hostname  $Port
# nc -v  $Nimbus_Hostname  $Port


On Nimbus host,

Please verify if the port is accessible (firewall is disabled) , Hostname (FQDN) is correctly set and there are no errors on Nimbus logs.

# netstat -tnlpa | grep 6627
# service iptables status
# hostname -f

.

avatar
Expert Contributor

@Jay Kumar SenSharma

The error is not because of impersonation.'

The error is due to " connection refused"

Telnet is passing for hostname 6667. But the nimbus.thrift.port is 6627.

Is there any ways to change the port. ?

Thanks for all your help 🙂

avatar
Expert Contributor

Hi @Jay Kumar SenSharma I tried with the hostname also. But no luck .

Port 6627 is failing while trying to telnet with the nimbus host.

Also i am using a single node cluster on Azure VM. I purchased a RHEL 7.2 VM then install hdp and hdf on top of it.

thank you