Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Storm UI RuntimeException

Solved Go to solution

Storm UI RuntimeException

Contributor

Am running the full-dev environment single node VM for Metron, after restarting the node and starting the services through Ambari there were some issues with Storm. I cleared out the storm.local.dir and restarted which seems to have allowed all the Storm services to start through Ambari, but when I access the web UI for Storm there is an internal server error.

java.lang.RuntimeException: Could not find leader nimbus from seed hosts ["node1"]. Did you specify a valid list of nimubs hosts for config nimbus.seeds.

I have tried restarting zookeeper but am not sure how to clear out the zookeeper configuration in Metron environment.

Any help would be greatly appreciated.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Storm UI RuntimeException

@Aaron Harris

Couple of things to check.

1 ] Can you please confirm whether nimbus service is up and running

2] Check for error in nimbus log available on nimbus host at path /var/log/storm/nimbus.log

3] Check for error in storm ui log available at storm ui server at path /var/log/storm/ui.log

4 REPLIES 4

Re: Storm UI RuntimeException

@Aaron Harris

Couple of things to check.

1 ] Can you please confirm whether nimbus service is up and running

2] Check for error in nimbus log available on nimbus host at path /var/log/storm/nimbus.log

3] Check for error in storm ui log available at storm ui server at path /var/log/storm/ui.log

Re: Storm UI RuntimeException

Contributor

@Santhosh B Gowda

Thanks for your response.

Nimbus is showing as running without errors in Ambari and the service looks to be up correctly.

I am getting an error in the nimbus.log as below;

2016-10-13T08:00:58.228+0000 o.a.t.s.AbstractNonblockingServer$FrameBuffer [ERROR] Unexpected throwable while invoking!
java.lang.RuntimeException: No nimbus leader participant host found, have you started your nimbus hosts?
        at backtype.storm.zookeeper$to_NimbusInfo.invoke(zookeeper.clj:233) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at backtype.storm.zookeeper$zk_leader_elector$reify__1009.getLeader(zookeeper.clj:305) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_40]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_40]
        at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.6.0.jar:na]
        at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:313) ~[clojure-1.6.0.jar:na]
        at backtype.storm.daemon.nimbus$fn__6231$exec_fn__1296__auto__$reify__6250.getClusterInfo(nimbus.clj:1349) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at backtype.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:1812) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at backtype.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:1796) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:39) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:39) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at backtype.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:159) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.server.Invocation.run(Invocation.java:18) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_40]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_40]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40]

Then the below is occurring in the ui.log

2016-10-13T08:02:08.239+0000 b.s.u.NimbusClient [WARN] Ignoring exception while trying to get leader nimbus info from node1. will retry with a different seed host.
org.apache.thrift7.transport.TTransportException: null
        at org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:69) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:559) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:547) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:68) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at backtype.storm.ui.core$nimbus_summary.invoke(core.clj:580) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at backtype.storm.ui.core$fn__10249.invoke(core.clj:982) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at compojure.core$make_route$fn__1889.invoke(core.clj:93) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at compojure.core$if_route$fn__1877.invoke(core.clj:39) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at compojure.core$if_method$fn__1870.invoke(core.clj:24) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at compojure.core$routing$fn__1895.invoke(core.clj:106) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]



Re: Storm UI RuntimeException

Contributor

Actually I have found a section further up in the log that might be the actual error;

2016-10-13T08:08:54.289+0000 b.s.zookeeper [INFO] node1 gained leadership, checking if it has all the topology code locally.
2016-10-13T08:08:54.297+0000 b.s.zookeeper [INFO] active-topology-ids [yaf-1-1476261289,bro-11-1476195353,squid-15-1476196296,enrichment-17-1476215724,snort-13-1476195443] local-topology-ids [enrichment-10-1476302158,bro-8-1476302148,yaf-7-1476302143,snort-9-1476302153] diff-topology [yaf-1-1476261289,bro-11-1476195353,squid-15-1476196296,enrichment-17-1476215724,snort-13-1476195443]
2016-10-13T08:08:54.299+0000 b.s.zookeeper [INFO] code for all active topologies not available locally, giving up leadership.



Re: Storm UI RuntimeException

Contributor

Fixed now thanks, the issue was that I had backed up my storm local data previously because of an issue with the supervisor starting.

Once I copied the nimbus/stormdist folder back into the storm local data folder and started nimbus again it all came up correctly.

Don't have an account?
Coming from Hortonworks? Activate your account here