Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nimbus server crashing immdediately after start

Nimbus server crashing immdediately after start

New Contributor

Product: Hortonworks Data Platform 2.6.3 Sandbox

This happened right after I solved this issue:

https://community.hortonworks.com/questions/177783/unable-to-log-storm-audit-events-to-hdfs-sandbox-...

I couldn't start storm nimbus server from ambari. Every time I start it, it crashes immediately.

I see this error in nimbus.log:

2018-03-19 16:06:50.222 o.a.s.s.a.k.KerberosSaslTransportPlugin [INFO] Service principal:nimbus/sandbox-hdp.hortonworks.com@HORTONWORKS.COM
2018-03-19 16:06:50.222 o.a.s.s.a.k.KerberosSaslTransportPlugin [INFO] SASL GSSAPI transport factory will be used
2018-03-19 16:06:50.973 o.a.s.d.nimbus [ERROR] Error when processing event
java.lang.RuntimeException: org.apache.storm.shade.org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /workerbeats/WordCount002-4-1520531102/c208e56f-cac3-47db-8a05-992b70912f96-6701
at org.apache.storm.util$wrap_in_runtime.invoke(util.clj:54) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.zookeeper$get_data.invoke(zookeeper.clj:157) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.cluster_state.zookeeper_state_factory$_mkState$reify__11806.get_data(zookeeper_state_factory.clj:117) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.cluster_state.zookeeper_state_factory$_mkState$reify__11806.get_worker_hb(zookeeper_state_factory.clj:129) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_151]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_151]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.7.0.jar:?]
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.7.0.jar:?]
at org.apache.storm.cluster$mk_storm_cluster_state$reify__6597.get_worker_heartbeat(cluster.clj:383) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.cluster$mk_storm_cluster_state$reify__6597$iter__6605__6609$fn__6610.invoke(cluster.clj:397) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
at clojure.lang.RT.seq(RT.java:507) ~[clojure-1.7.0.jar:?]
at clojure.core$seq__4128.invoke(core.clj:137) ~[clojure-1.7.0.jar:?]
at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]
at org.apache.storm.cluster$mk_storm_cluster_state$reify__6597.executor_beats(cluster.clj:400) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_151]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_151]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.7.0.jar:?]
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$update_heartbeats_BANG_.invoke(nimbus.clj:606) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.daemon.nimbus$update_all_heartbeats_BANG_.invoke(nimbus.clj:617) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
....

followed by this:

2018-03-19 16:06:50.978 o.a.s.util [ERROR] Halting process: ("Error when processing an event")
java.lang.RuntimeException: ("Error when processing an event")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$nimbus_data$fn__8110.invoke(nimbus.clj:213) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.timer$mk_timer$fn__4466$fn__4467.invoke(timer.clj:71) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.timer$mk_timer$fn__4466.invoke(timer.clj:42) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at clojure.lang.AFn.run(AFn.java:22) ~[clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
2018-03-19 16:06:50.981 o.a.s.d.nimbus [INFO] Shutting down master

It seems like a authorization problem, but I am not sure how to handle it.

The sandbox is kerberized.

WordCount is the name of a topology that I started before. It seems that nimbus couldn't start this worker because of authorization issue.

Any advice or suggestion would be appreciated. Thanks.

Don't have an account?
Coming from Hortonworks? Activate your account here