Problem: In HDP 2.3.4 and Ambari 2.4.2, Storm nimbus was not coming up: 2017-05-30 14:35:18.756 o.a.t.s.TThreadPoolServer [ERROR] Error occurred during processing of message. java.lang.RuntimeException: No nimbus leader participant host found, have you started your nimbus hosts?? Following is the error from Nimbus log: 2017-05-30 14:35:18.750 b.s.d.nimbus [WARN] principal: storm@EXAMPLE.COM is trying to impersonate principal: ambari-server@EXAMPLE.COM 2017-05-30 14:35:18.751 b.s.d.nimbus [WARN] impersonation attempt but nimbus.impersonation.authorizer has no authorizer configured. potential security risk, please see SECURITY.MD to learn how to configure impersonation authorizer. 2017-05-30 14:35:18.756 o.a.t.s.TThreadPoolServer [ERROR] Error occurred during processing of message. java.lang.RuntimeException: No nimbus leader participant host found, have you started your nimbus hosts? Cause: Property ''nimbus.impersonation.authorizer" was set to 'org.apache.storm.security.auth.authorizer.ImpersonationAuthorizer'. Prior to HDP 2.5, the storm package was backtype instead of org.apache. Solution: Modify the property as follows: nimbus.impersonation.authorizer=backtype.storm.security.auth.authorizer.ImpersonationAuthorizer ... View more

Problem Description: Unable to start storm nimbus from Ambari, throwing following error: raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-sudo.sh ln -s /usr/lib/storm/lib/ambari-metrics-storm-sink-legacy-with-common-*.jar /usr/hdp/current/storm-nimbus/lib/ambari-metrics-storm-sink.jar' returned 1. ln: target `/usr/hdp/current/storm-nimbus/lib/ambari-metrics-storm-sink.jar' is not a directory Cause: Two legacy jars under storm lib: # ls -ld /usr/lib/storm/lib/ambari-metrics-storm-sink-legacy-with-common* -rw-r--r-- 1 root root 1937430 May 11 15:57 /usr/lib/storm/lib/ambari-metrics-storm-sink-legacy-with-common-2.4.1.0.22.jar -rw-r--r-- 1 root root 1933392 Nov 23 2016 /usr/lib/storm/lib/ambari-metrics-storm-sink-legacy-with-common-2.4.2.0.136.jar Solution: As the Ambari version was 2.4.2 so, moved ambari-metrics-storm-sink-legacy-with-common-2.4.1.0.22.jar to a different location and started storm nimbus. ... View more

Following was the error in supervisor log: 2017-06-09 14:40:18.348 o.a.s.d.supervisor [ERROR] Error on initialization of server mk-supervisor java.lang.RuntimeException: java.lang.ClassNotFoundException: backtype.storm.generated.LSSupervisorId at org.apache.storm.utils.LocalState.deserialize(LocalState.java:83) at org.apache.storm.utils.LocalState.get(LocalState.java:130) at org.apache.storm.local_state$ls_supervisor_id.invoke(local_state.clj:61) at org.apache.storm.daemon.supervisor$standalone_supervisor$reify__7977.prepare(supervisor.clj:1216) at org.apache.storm.daemon.supervisor$fn__7833$exec_fn__3537__auto____7834.invoke(supervisor.clj:766) at clojure.lang.AFn.applyToHelper(AFn.java:160) at clojure.lang.AFn.applyTo(AFn.java:144) at clojure.core$apply.invoke(core.clj:630) at org.apache.storm.daemon.supervisor$fn__7833$mk_supervisor__7878.doInvoke(supervisor.clj:764) at clojure.lang.RestFn.invoke(RestFn.java:436) at org.apache.storm.daemon.supervisor$_launch.invoke(supervisor.clj:1204) at org.apache.storm.daemon.supervisor$_main.invoke(supervisor.clj:1237) at clojure.lang.AFn.applyToHelper(AFn.java:152) at clojure.lang.AFn.applyTo(AFn.java:144) at org.apache.storm.daemon.supervisor.main(Unknown Source) Caused by: java.lang.ClassNotFoundException: backtype.storm.generated.LSSupervisorId at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.storm.utils.LocalState.deserialize(LocalState.java:78) ... 14 more 2017-06-09 14:40:18.351 o.a.s.util [ERROR] Halting process: ("Error on initialization") java.lang.RuntimeException: ("Error on initialization") at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) at clojure.lang.RestFn.invoke(RestFn.java:423) at org.apache.storm.daemon.supervisor$fn__7833$mk_supervisor__7878.doInvoke(supervisor.clj:764) at clojure.lang.RestFn.invoke(RestFn.java:436) at org.apache.storm.daemon.supervisor$_launch.invoke(supervisor.clj:1204) at org.apache.storm.daemon.supervisor$_main.invoke(supervisor.clj:1237) at clojure.lang.AFn.applyToHelper(AFn.java:152) at clojure.lang.AFn.applyTo(AFn.java:144) at org.apache.storm.daemon.supervisor.main(Unknown Source) Cause: 1. Few storm configs were still configured with backtype packages 2. Stale states in zookeeper and local data Solution: 1. Filtered in the configs and changed all the backtype packages to org.apache as Storm from HDP 2.5 onwards uses org.apache packages. 2. Follow below steps to clear stale states and local data: -> Deactivate all running topologies. -> Stop Storm service -> Delete all states under zookeeper: $/usr/hdp/current/zookeeper-client/bin/zkCli.sh (optionally in secure environment specify -server zk.server:port) > rmr /storm -> Delete all states under the storm-local directory. Please make sure to run this on all storm hosts: $ rm -rf <value of stormlocal.dir> -> Start storm service. ... View more

Problem Description: On deleting few kafka topics and creating same topics again there Not Leader for this partition exception while Producing/Consuming and was reading old meta data. Getting Below error: 17/03/10 10:45:35 ERROR ApplicationMaster: User class threw exception: org.apache.spark.SparkException: org.apache.spark.SparkException: Couldn't find leaders for Set([akrnohij-wng-fp9,47], [akrnohij-wng-fp4,224], [akrnohij-wng-fp11,172], [akrnohij-wng-fp1,84], [akrnohij-wng-fp10,117], [akrnohij-wng-fp10,168], [akrnohij-wng-fp4,176], [akrnohij-wng-fp1,136], [akrnohij-wng-fp9,167], [akrnohij-wng-fp2,174], [akrnohij-wng-fp11,74], [akrnohij-wng-fp3,189], [akrnohij-wng-fp11,200], [akrnohij-wng-fp11,168], [akrnohij-wng-fp4,149], [akrnohij-wng-fp7,127], [akrnohij-wng-fp6,39], [akrnohij-wng-fp10,133], [akrnohij-wng-fp9,171], [akrnohij-wng-fp5,175], [akrnohij-wng-fp7,181] Cause: The zookeeper ACL for state znode was not correct : getAcl /brokers/topics/akrnohij-wng-fp4/partitions/135/state ‘auth,’ : cdrwa, ‘world,’anyone :r The following property was set in zookeeper env : -Dzookeeper.skipACL=yes Solution: Deleted the topic manually as deletion was stuck Removed the property -Dzookeeper.skipACL=yes Restart zookeeper and Kafka services NOTE- It is dangerous to use -Dzookeeper.skipACL=yes property and instead it is recommended to use kafka service principal if in case you need to delete the znode for kafka topics. ... View more

PROBLEM: While implementing AutoHDFS for storm-hdfs integration, observed following error: 2017-05-19 11:21:44.865 o.a.s.h.c.s.AutoHDFS [ERROR] Could not populate HDFS credentials. java.lang.RuntimeException: Failed to get delegation tokens. at org.apache.storm.hdfs.common.security.AutoHDFS.getHadoopCredentials(AutoHDFS.java:242) at org.apache.storm.hdfs.common.security.AutoHDFS.populateCredentials(AutoHDFS.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) at org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__11226.submitTopologyWithOpts(nimbus.clj:1544) at org.apache.storm.generated.Nimbus$Processor$submitTopologyWithOpts.getResult(Nimbus.java:2940) at org.apache.storm.generated.Nimbus$Processor$submitTopologyWithOpts.getResult(Nimbus.java:2924) at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.storm.security.auth.SaslTransportPlugin$TUGIWrapProcessor.process(SaslTransportPlugin.java:138) at org.apache.storm.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hdfs.DistributedFileSystem not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2214) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2746) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2759) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2795) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2777) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:386) at org.apache.storm.hdfs.common.security.AutoHDFS$1.run(AutoHDFS.java:217) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1704) at org.apache.storm.hdfs.common.security.AutoHDFS.getHadoopCredentials(AutoHDFS.java:213) ... 17 more Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hdfs.DistributedFileSystem not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2120) CAUSE: Missing Hadoop dependencies in pom.xml caused the following exception: Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hdfs.DistributedFileSystem not found SOLUTION: To resolve this issue, added the following dependency in pom.xml: <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>2.7.3.2.5.3.0-37</version> <exclusions> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.7.3.2.5.3.0-37</version> <exclusions> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> </exclusions> </dependency> ... View more

SYMPTOM: While integrating Storm 1.0.1 to Elastic Search 5.0.0, the following error is observed: Exception in thread "main" java.lang.NoClassDefFoundError: org/elasticsearch/common/base/Preconditions at org.apache.storm.elasticsearch.common.EsConfig.<init>(EsConfig.java:62) at org.apache.storm.elasticsearch.common.EsConfig.<init>(EsConfig.java:49) at com.mz.pipeline.StreamToES5_1.main(StreamToES5_1.java:72) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) Caused by: java.lang.ClassNotFoundException: org.elasticsearch.common.base.Preconditions Following were the maven dependencies: <dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-core</artifactId> <version>1.0.2</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>log4j-over-slf4j</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.elasticsearch</groupId> <artifactId>elasticsearch</artifactId> <version>5.0.0</version> </dependency> <dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-elasticsearch</artifactId> <version>1.0.2</version> </dependency> ROOT CAUSE: Apache libraries were used to compile/package the topology instead of using Hortonworks repository RESOLUTION: Please add the Hortonworks repository in the pom.xml: <repositories> <repository> <id>hortonworks</id> <url>http://repo.hortonworks.com/content/groups/public/</url> </repository> </repositories> Then change the storm artifact versions, which should be in this format: <apache_version>.<HDP-version> For example, for 'storm-core' in HDP 2.5.0.0, version would be 1.0.1.2.5.0.0-1245. Similarly, for 'storm-elasticsearch' it would be: 1.0.1.2.5.0.0-1245 (for HDP 2.5.0.0). Please find the version corresponding to your HDP here: http://repo.hortonworks.com/content/groups/public/org/apache/storm/ ... View more

ROOT CAUSE:kafka.metrics.reporters in Advanced Kafka-broker was pointing to Ganglia metrics reporter. kafka.metrics.reporters=kafka.ganglia.KafkaGangliaMetricsReporter RESOLUTION: Kafka.metrics.reporters is pointing to GangliaMetrics. It should be pointing to Ambari metrics like this: kafka.metrics.reporters=org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter You can modify the property via Ambari -> Kafka -> configs -> Advanced kafka-broker-> kafka.metrics.reporters and modify the property. Please save the changes and restart required services ... View more

SYMPTOM: Storm nimbus fails to come up after Ambari was upgraded to 2.4 version while on HDP 2.4 2016-11-22 13:29:43.066 [timer] b.s.d.nimbus [ERROR] Error when processing event java.lang.NullPointerException at clojure.lang.Numbers.ops(Numbers.java:961) ~[clojure-1.6.0.jar:?] at clojure.lang.Numbers.isZero(Numbers.java:90) ~[clojure-1.6.0.jar:?] at backtype.storm.util$partition_fixed.invoke(util.clj:892) ~[storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.6.0.jar:?] at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.6.0.jar:?] at clojure.core$apply.invoke(core.clj:624) ~[clojure-1.6.0.jar:?] at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.6.0.jar:?] at clojure.lang.RestFn.applyTo(RestFn.java:132) ~[clojure-1.6.0.jar:?] at clojure.core$apply.invoke(core.clj:626) ~[clojure-1.6.0.jar:?] at clojure.core$partial$fn__4228.doInvoke(core.clj:2468) ~[clojure-1.6.0.jar:?] at clojure.lang.RestFn.invoke(RestFn.java:408) ~[clojure-1.6.0.jar:?] at backtype.storm.util$map_val$iter__366__370$fn__371.invoke(util.clj:301) ~[storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] ROOT CAUSE: It is a known issue: https://hortonworks.jira.com/browse/BUG-66735 RESOLUTION: It has been fixed in HDP 2.5.3, please refer section 'Upgrade' here: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_release-notes/content/fixed_issues.html WORKAROUND: This can be resolved by following below steps as a workaround: 1. Deactivate all running topologies. 2. Stop Storm service 3. Delete all states under zookeeper: -> /usr/hdp/current/zookeeper-client/bin/zkCli.sh (optionally in secure environment specify -server zk.server:port) -> rmr /storm 4. Delete all states under the storm-local directory. Please make sure to run this on all storm hosts: rm -rf <value of stormlocal.dir> 5. Start storm service. ... View more