Community Articles

Find and share helpful community-sourced technical articles.
Labels (2)
avatar
Expert Contributor

SYMPTOM: Storm nimbus fails to come up after Ambari was upgraded to 2.4 version while on HDP 2.4

2016-11-22 13:29:43.066 [timer] b.s.d.nimbus [ERROR] Error when processing event
java.lang.NullPointerException
	at clojure.lang.Numbers.ops(Numbers.java:961) ~[clojure-1.6.0.jar:?]
	at clojure.lang.Numbers.isZero(Numbers.java:90) ~[clojure-1.6.0.jar:?]
	at backtype.storm.util$partition_fixed.invoke(util.clj:892) ~[storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169]
	at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.6.0.jar:?]
	at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.6.0.jar:?]
	at clojure.core$apply.invoke(core.clj:624) ~[clojure-1.6.0.jar:?]
	at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.6.0.jar:?]
	at clojure.lang.RestFn.applyTo(RestFn.java:132) ~[clojure-1.6.0.jar:?]
	at clojure.core$apply.invoke(core.clj:626) ~[clojure-1.6.0.jar:?]
	at clojure.core$partial$fn__4228.doInvoke(core.clj:2468) ~[clojure-1.6.0.jar:?]
	at clojure.lang.RestFn.invoke(RestFn.java:408) ~[clojure-1.6.0.jar:?]
	at backtype.storm.util$map_val$iter__366__370$fn__371.invoke(util.clj:301) ~[storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169]

ROOT CAUSE: It is a known issue: https://hortonworks.jira.com/browse/BUG-66735

RESOLUTION: It has been fixed in HDP 2.5.3, please refer section 'Upgrade' here: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_release-notes/content/fixed_issues.html

WORKAROUND: This can be resolved by following below steps as a workaround:

1. Deactivate all running topologies.
2. Stop Storm service
3. Delete all states under zookeeper: -> /usr/hdp/current/zookeeper-client/bin/zkCli.sh (optionally in secure environment specify -server zk.server:port) -> rmr /storm 
4. Delete all states under the storm-local directory.  Please make sure to run this on all storm hosts: rm -rf <value of stormlocal.dir> 
5. Start storm service. 
596 Views