Problem: In HDP 2.3.4 and Ambari 2.4.2, Storm nimbus was not coming up: 2017-05-30 14:35:18.756 o.a.t.s.TThreadPoolServer [ERROR] Error occurred during processing of message. java.lang.RuntimeException: No nimbus leader participant host found, have you started your nimbus hosts?? Following is the error from Nimbus log: 2017-05-30 14:35:18.750 b.s.d.nimbus [WARN] principal: storm@EXAMPLE.COM is trying to impersonate principal: ambari-server@EXAMPLE.COM 2017-05-30 14:35:18.751 b.s.d.nimbus [WARN] impersonation attempt but nimbus.impersonation.authorizer has no authorizer configured. potential security risk, please see SECURITY.MD to learn how to configure impersonation authorizer. 2017-05-30 14:35:18.756 o.a.t.s.TThreadPoolServer [ERROR] Error occurred during processing of message. java.lang.RuntimeException: No nimbus leader participant host found, have you started your nimbus hosts? Cause: Property ''nimbus.impersonation.authorizer" was set to 'org.apache.storm.security.auth.authorizer.ImpersonationAuthorizer'. Prior to HDP 2.5, the storm package was backtype instead of org.apache. Solution: Modify the property as follows: nimbus.impersonation.authorizer=backtype.storm.security.auth.authorizer.ImpersonationAuthorizer ... View more

Problem Description: Unable to start storm nimbus from Ambari, throwing following error: raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-sudo.sh ln -s /usr/lib/storm/lib/ambari-metrics-storm-sink-legacy-with-common-*.jar /usr/hdp/current/storm-nimbus/lib/ambari-metrics-storm-sink.jar' returned 1. ln: target `/usr/hdp/current/storm-nimbus/lib/ambari-metrics-storm-sink.jar' is not a directory Cause: Two legacy jars under storm lib: # ls -ld /usr/lib/storm/lib/ambari-metrics-storm-sink-legacy-with-common* -rw-r--r-- 1 root root 1937430 May 11 15:57 /usr/lib/storm/lib/ambari-metrics-storm-sink-legacy-with-common-2.4.1.0.22.jar -rw-r--r-- 1 root root 1933392 Nov 23 2016 /usr/lib/storm/lib/ambari-metrics-storm-sink-legacy-with-common-2.4.2.0.136.jar Solution: As the Ambari version was 2.4.2 so, moved ambari-metrics-storm-sink-legacy-with-common-2.4.1.0.22.jar to a different location and started storm nimbus. ... View more

Following was the error in supervisor log: 2017-06-09 14:40:18.348 o.a.s.d.supervisor [ERROR] Error on initialization of server mk-supervisor java.lang.RuntimeException: java.lang.ClassNotFoundException: backtype.storm.generated.LSSupervisorId at org.apache.storm.utils.LocalState.deserialize(LocalState.java:83) at org.apache.storm.utils.LocalState.get(LocalState.java:130) at org.apache.storm.local_state$ls_supervisor_id.invoke(local_state.clj:61) at org.apache.storm.daemon.supervisor$standalone_supervisor$reify__7977.prepare(supervisor.clj:1216) at org.apache.storm.daemon.supervisor$fn__7833$exec_fn__3537__auto____7834.invoke(supervisor.clj:766) at clojure.lang.AFn.applyToHelper(AFn.java:160) at clojure.lang.AFn.applyTo(AFn.java:144) at clojure.core$apply.invoke(core.clj:630) at org.apache.storm.daemon.supervisor$fn__7833$mk_supervisor__7878.doInvoke(supervisor.clj:764) at clojure.lang.RestFn.invoke(RestFn.java:436) at org.apache.storm.daemon.supervisor$_launch.invoke(supervisor.clj:1204) at org.apache.storm.daemon.supervisor$_main.invoke(supervisor.clj:1237) at clojure.lang.AFn.applyToHelper(AFn.java:152) at clojure.lang.AFn.applyTo(AFn.java:144) at org.apache.storm.daemon.supervisor.main(Unknown Source) Caused by: java.lang.ClassNotFoundException: backtype.storm.generated.LSSupervisorId at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.storm.utils.LocalState.deserialize(LocalState.java:78) ... 14 more 2017-06-09 14:40:18.351 o.a.s.util [ERROR] Halting process: ("Error on initialization") java.lang.RuntimeException: ("Error on initialization") at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) at clojure.lang.RestFn.invoke(RestFn.java:423) at org.apache.storm.daemon.supervisor$fn__7833$mk_supervisor__7878.doInvoke(supervisor.clj:764) at clojure.lang.RestFn.invoke(RestFn.java:436) at org.apache.storm.daemon.supervisor$_launch.invoke(supervisor.clj:1204) at org.apache.storm.daemon.supervisor$_main.invoke(supervisor.clj:1237) at clojure.lang.AFn.applyToHelper(AFn.java:152) at clojure.lang.AFn.applyTo(AFn.java:144) at org.apache.storm.daemon.supervisor.main(Unknown Source) Cause: 1. Few storm configs were still configured with backtype packages 2. Stale states in zookeeper and local data Solution: 1. Filtered in the configs and changed all the backtype packages to org.apache as Storm from HDP 2.5 onwards uses org.apache packages. 2. Follow below steps to clear stale states and local data: -> Deactivate all running topologies. -> Stop Storm service -> Delete all states under zookeeper: $/usr/hdp/current/zookeeper-client/bin/zkCli.sh (optionally in secure environment specify -server zk.server:port) > rmr /storm -> Delete all states under the storm-local directory. Please make sure to run this on all storm hosts: $ rm -rf <value of stormlocal.dir> -> Start storm service. ... View more