Storm Nimbus fail to start with error org.apache.storm.generated.KeyNotFoundException: null

[#1]

Suddenly our Storm Nimbus stopped and now it won's start. Appreciate any help.

Storm Version: Storm 1.1.0

HDP: HDP-2.6.5.1175

No changes made on the config and OS for a very long time.

Here's the error on two of the Nimbus node:

2022-08-03 08:43:13.372 o.a.s.d.nimbus [INFO] not a leader, skipping assignments
2022-08-03 08:43:13.372 o.a.s.d.nimbus [INFO] not a leader, skipping cleanup
2022-08-03 08:43:23.373 o.a.s.d.nimbus [INFO] not a leader, skipping assignments
2022-08-03 08:43:23.373 o.a.s.d.nimbus [INFO] not a leader, skipping cleanup
2022-08-03 08:43:33.374 o.a.s.d.nimbus [INFO] not a leader, skipping assignments
2022-08-03 08:43:33.374 o.a.s.d.nimbus [INFO] not a leader, skipping cleanup
2022-08-03 08:43:43.374 o.a.s.d.nimbus [INFO] not a leader, skipping assignments
2022-08-03 08:43:43.374 o.a.s.d.nimbus [INFO] not a leader, skipping cleanup
2022-08-03 08:43:53.374 o.a.s.d.nimbus [INFO] not a leader, skipping assignments
2022-08-03 08:43:53.375 o.a.s.d.nimbus [INFO] not a leader, skipping cleanup
2022-08-03 08:44:03.376 o.a.s.d.nimbus [INFO] not a leader, skipping assignments
2022-08-03 08:44:03.376 o.a.s.d.nimbus [INFO] not a leader, skipping cleanup
2022-08-03 08:44:12.025 o.a.s.z.Zookeeper [INFO] active-topology-blobs [] local-topology-blobs [] diff-topology-blobs []
2022-08-03 08:44:12.025 o.a.s.z.Zookeeper [INFO] active-topology-dependencies [] local-blobs [] diff-topology-dependencies []
2022-08-03 08:44:12.025 o.a.s.z.Zookeeper [INFO] Accepting leadership, all active topologies and corresponding dependencies found locally.
2022-08-03 08:44:13.468 o.a.s.d.nimbus [ERROR] Error when processing event
org.apache.storm.generated.KeyNotFoundException: null
at org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:147) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.blobstore.LocalFsBlobStore.getBlob(LocalFsBlobStore.java:270) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.blobstore.BlobStore.readBlobTo(BlobStore.java:271) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.blobstore.BlobStore.readBlob(BlobStore.java:300) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_251]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_251]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_251]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_251]
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.7.0.jar:?]
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$read_storm_conf_as_nimbus.invoke(nimbus.clj:542) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.daemon.nimbus$compute_executors.invoke(nimbus.clj:658) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.daemon.nimbus$compute_topology__GT_executors$iter__8280__8284$fn__8285.invoke(nimbus.clj:688) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
at clojure.lang.RT.seq(RT.java:507) ~[clojure-1.7.0.jar:?]
at clojure.core$seq__4128.invoke(core.clj:137) ~[clojure-1.7.0.jar:?]
at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30) ~[clojure-1.7.0.jar:?]
at clojure.core.protocols$fn__6506.invoke(protocols.clj:101) ~[clojure-1.7.0.jar:?]
at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) ~[clojure-1.7.0.jar:?]
at clojure.core$reduce.invoke(core.clj:6519) ~[clojure-1.7.0.jar:?]
at clojure.core$into.invoke(core.clj:6600) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$compute_topology__GT_executors.invoke(nimbus.clj:687) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.daemon.nimbus$compute_new_scheduler_assignments.invoke(nimbus.clj:842) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:1011) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at clojure.lang.RestFn.invoke(RestFn.java:410) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$fn__9323$exec_fn__1717__auto____9324$fn__9335.invoke(nimbus.clj:2479) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.timer$schedule_recurring$this__4483.invoke(timer.clj:105) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.timer$mk_timer$fn__4466$fn__4467.invoke(timer.clj:50) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.timer$mk_timer$fn__4466.invoke(timer.clj:42) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at clojure.lang.AFn.run(AFn.java:22) ~[clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
2022-08-03 08:44:13.475 o.a.s.util [ERROR] Halting process: ("Error when processing an event")
java.lang.RuntimeException: ("Error when processing an event")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$nimbus_data$fn__8110.invoke(nimbus.clj:213) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.timer$mk_timer$fn__4466$fn__4467.invoke(timer.clj:71) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.timer$mk_timer$fn__4466.invoke(timer.clj:42) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at clojure.lang.AFn.run(AFn.java:22) ~[clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
2022-08-03 08:44:13.479 o.a.s.d.nimbus [INFO] Shutting down master

 Here's the error on the other one Nimbus:

2022-08-03 08:43:00.530 o.a.s.d.nimbus [INFO] Starting nimbus server for storm version '1.1.0.2.6.5.1175-1'
2022-08-03 08:43:01.380 o.a.s.d.nimbus [ERROR] Error when processing event
org.apache.storm.generated.KeyNotFoundException: null
at org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:147) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.blobstore.LocalFsBlobStore.getBlob(LocalFsBlobStore.java:270) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.blobstore.BlobStore.readBlobTo(BlobStore.java:271) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.blobstore.BlobStore.readBlob(BlobStore.java:300) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_251]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_251]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_251]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_251]
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.7.0.jar:?]
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$read_storm_conf_as_nimbus.invoke(nimbus.clj:542) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.daemon.nimbus$compute_executors.invoke(nimbus.clj:658) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.daemon.nimbus$compute_topology__GT_executors$iter__8280__8284$fn__8285.invoke(nimbus.clj:688) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
at clojure.lang.RT.seq(RT.java:507) ~[clojure-1.7.0.jar:?]
at clojure.core$seq__4128.invoke(core.clj:137) ~[clojure-1.7.0.jar:?]
at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30) ~[clojure-1.7.0.jar:?]
at clojure.core.protocols$fn__6506.invoke(protocols.clj:101) ~[clojure-1.7.0.jar:?]
at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) ~[clojure-1.7.0.jar:?]
at clojure.core$reduce.invoke(core.clj:6519) ~[clojure-1.7.0.jar:?]
at clojure.core$into.invoke(core.clj:6600) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$compute_topology__GT_executors.invoke(nimbus.clj:687) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.daemon.nimbus$compute_new_scheduler_assignments.invoke(nimbus.clj:842) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:1011) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at clojure.lang.RestFn.invoke(RestFn.java:410) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$fn__9323$exec_fn__1717__auto____9324$fn__9335.invoke(nimbus.clj:2479) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.timer$schedule_recurring$this__4483.invoke(timer.clj:105) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.timer$mk_timer$fn__4466$fn__4467.invoke(timer.clj:50) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.timer$mk_timer$fn__4466.invoke(timer.clj:42) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at clojure.lang.AFn.run(AFn.java:22) ~[clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
2022-08-03 08:43:01.383 o.a.s.util [ERROR] Halting process: ("Error when processing an event")
java.lang.RuntimeException: ("Error when processing an event")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$nimbus_data$fn__8110.invoke(nimbus.clj:213) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.timer$mk_timer$fn__4466$fn__4467.invoke(timer.clj:71) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.timer$mk_timer$fn__4466.invoke(timer.clj:42) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at clojure.lang.AFn.run(AFn.java:22) ~[clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]

[#2]

@cfoo 

I see the following in the Nimbus log:

[ERROR] Error when processing event
org.apache.storm.generated.KeyNotFoundException: null

Could you please try the below action plan and let us know how it goes: 

- Increase the JVM of the nimbus from the existing value to keep the nimbus up for a few additional moments to kill the topologies.

- stop all the topologies

- delete the znode /storm ( From zk cli : rmr /storm )

- take a backup of storm.local.dir and delete the same( From zk cli : rm -rf <value of stormlocal.dir> ). You can check it in the storm.yaml file

- start the supervisors and nimbus, check if the services are up and running

- If there is no Nimbus service running, then the only way to remove the topology processes would be to manually kill the worker processes on all worker nodes.

If you found this response assisted with your query, please take a moment to log in and click on  KUDOS 🙂 & ”Accept as Solution" below this post.

Thank you.