Storm UI RuntimeException

[#1]

Am running the full-dev environment single node VM for Metron, after restarting the node and starting the services through Ambari there were some issues with Storm. I cleared out the storm.local.dir and restarted which seems to have allowed all the Storm services to start through Ambari, but when I access the web UI for Storm there is an internal server error.

java.lang.RuntimeException: Could not find leader nimbus from seed hosts ["node1"]. Did you specify a valid list of nimubs hosts for config nimbus.seeds.

I have tried restarting zookeeper but am not sure how to clear out the zookeeper configuration in Metron environment.

Any help would be greatly appreciated.

[#2 · Accepted solution]

@Aaron Harris

Couple of things to check.

1 ] Can you please confirm whether nimbus service is up and running

2] Check for error in nimbus log available on nimbus host at path /var/log/storm/nimbus.log

3] Check for error in storm ui log available at storm ui server at path /var/log/storm/ui.log

[#3]

@Santhosh B Gowda

Thanks for your response.

Nimbus is showing as running without errors in Ambari and the service looks to be up correctly.

I am getting an error in the nimbus.log as below;

2016-10-13T08:00:58.228+0000 o.a.t.s.AbstractNonblockingServer$FrameBuffer [ERROR] Unexpected throwable while invoking!
java.lang.RuntimeException: No nimbus leader participant host found, have you started your nimbus hosts?
        at backtype.storm.zookeeper$to_NimbusInfo.invoke(zookeeper.clj:233) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at backtype.storm.zookeeper$zk_leader_elector$reify__1009.getLeader(zookeeper.clj:305) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_40]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_40]
        at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.6.0.jar:na]
        at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:313) ~[clojure-1.6.0.jar:na]
        at backtype.storm.daemon.nimbus$fn__6231$exec_fn__1296__auto__$reify__6250.getClusterInfo(nimbus.clj:1349) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at backtype.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:1812) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at backtype.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:1796) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:39) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:39) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at backtype.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:159) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.server.Invocation.run(Invocation.java:18) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_40]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_40]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40]

Then the below is occurring in the ui.log

2016-10-13T08:02:08.239+0000 b.s.u.NimbusClient [WARN] Ignoring exception while trying to get leader nimbus info from node1. will retry with a different seed host.
org.apache.thrift7.transport.TTransportException: null
        at org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:69) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:559) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:547) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:68) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at backtype.storm.ui.core$nimbus_summary.invoke(core.clj:580) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at backtype.storm.ui.core$fn__10249.invoke(core.clj:982) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at compojure.core$make_route$fn__1889.invoke(core.clj:93) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at compojure.core$if_route$fn__1877.invoke(core.clj:39) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at compojure.core$if_method$fn__1870.invoke(core.clj:24) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
        at compojure.core$routing$fn__1895.invoke(core.clj:106) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]

[#4]

Actually I have found a section further up in the log that might be the actual error;

2016-10-13T08:08:54.289+0000 b.s.zookeeper [INFO] node1 gained leadership, checking if it has all the topology code locally.
2016-10-13T08:08:54.297+0000 b.s.zookeeper [INFO] active-topology-ids [yaf-1-1476261289,bro-11-1476195353,squid-15-1476196296,enrichment-17-1476215724,snort-13-1476195443] local-topology-ids [enrichment-10-1476302158,bro-8-1476302148,yaf-7-1476302143,snort-9-1476302153] diff-topology [yaf-1-1476261289,bro-11-1476195353,squid-15-1476196296,enrichment-17-1476215724,snort-13-1476195443]
2016-10-13T08:08:54.299+0000 b.s.zookeeper [INFO] code for all active topologies not available locally, giving up leadership.

[#5]

Fixed now thanks, the issue was that I had backed up my storm local data previously because of an issue with the supervisor starting.

Once I copied the nimbus/stormdist folder back into the storm local data folder and started nimbus again it all came up correctly.