Support Questions
Find answers, ask questions, and share your expertise

Nimbus server crashing immdediately after start

New Contributor

Product: Hortonworks Data Platform 2.6.3 Sandbox

This happened right after I solved this issue:

https://community.hortonworks.com/questions/177783/unable-to-log-storm-audit-events-to-hdfs-sandbox-...

I couldn't start storm nimbus server from ambari. Every time I start it, it crashes immediately.

I see this error in nimbus.log:

2018-03-19 16:06:50.222 o.a.s.s.a.k.KerberosSaslTransportPlugin [INFO] Service principal:nimbus/sandbox-hdp.hortonworks.com@HORTONWORKS.COM
2018-03-19 16:06:50.222 o.a.s.s.a.k.KerberosSaslTransportPlugin [INFO] SASL GSSAPI transport factory will be used
2018-03-19 16:06:50.973 o.a.s.d.nimbus [ERROR] Error when processing event
java.lang.RuntimeException: org.apache.storm.shade.org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /workerbeats/WordCount002-4-1520531102/c208e56f-cac3-47db-8a05-992b70912f96-6701
at org.apache.storm.util$wrap_in_runtime.invoke(util.clj:54) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.zookeeper$get_data.invoke(zookeeper.clj:157) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.cluster_state.zookeeper_state_factory$_mkState$reify__11806.get_data(zookeeper_state_factory.clj:117) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.cluster_state.zookeeper_state_factory$_mkState$reify__11806.get_worker_hb(zookeeper_state_factory.clj:129) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_151]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_151]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.7.0.jar:?]
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.7.0.jar:?]
at org.apache.storm.cluster$mk_storm_cluster_state$reify__6597.get_worker_heartbeat(cluster.clj:383) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.cluster$mk_storm_cluster_state$reify__6597$iter__6605__6609$fn__6610.invoke(cluster.clj:397) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
at clojure.lang.RT.seq(RT.java:507) ~[clojure-1.7.0.jar:?]
at clojure.core$seq__4128.invoke(core.clj:137) ~[clojure-1.7.0.jar:?]
at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]
at org.apache.storm.cluster$mk_storm_cluster_state$reify__6597.executor_beats(cluster.clj:400) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_151]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_151]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.7.0.jar:?]
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$update_heartbeats_BANG_.invoke(nimbus.clj:606) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.daemon.nimbus$update_all_heartbeats_BANG_.invoke(nimbus.clj:617) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
....

followed by this:

2018-03-19 16:06:50.978 o.a.s.util [ERROR] Halting process: ("Error when processing an event")
java.lang.RuntimeException: ("Error when processing an event")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$nimbus_data$fn__8110.invoke(nimbus.clj:213) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.timer$mk_timer$fn__4466$fn__4467.invoke(timer.clj:71) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.timer$mk_timer$fn__4466.invoke(timer.clj:42) ~[storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at clojure.lang.AFn.run(AFn.java:22) ~[clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
2018-03-19 16:06:50.981 o.a.s.d.nimbus [INFO] Shutting down master

It seems like a authorization problem, but I am not sure how to handle it.

The sandbox is kerberized.

WordCount is the name of a topology that I started before. It seems that nimbus couldn't start this worker because of authorization issue.

Any advice or suggestion would be appreciated. Thanks.

0 REPLIES 0