Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search

Storm Nimbus fail to start with error org.apache.storm.generated.KeyNotFoundException: null

avatar
author-rank cfoo
New Contributor

Created ‎08-03-2022 06:06 AM

Suddenly our Storm Nimbus stopped and now it won's start. Appreciate any help.

Storm Version: Storm 1.1.0

HDP: HDP-2.6.5.1175

No changes made on the config and OS for a very long time.

Here's the error on two of the Nimbus node:

2022-08-03 08:43:13.372 o.a.s.d.nimbus [INFO] not a leader, skipping assignments
2022-08-03 08:43:13.372 o.a.s.d.nimbus [INFO] not a leader, skipping cleanup
2022-08-03 08:43:23.373 o.a.s.d.nimbus [INFO] not a leader, skipping assignments
2022-08-03 08:43:23.373 o.a.s.d.nimbus [INFO] not a leader, skipping cleanup
2022-08-03 08:43:33.374 o.a.s.d.nimbus [INFO] not a leader, skipping assignments
2022-08-03 08:43:33.374 o.a.s.d.nimbus [INFO] not a leader, skipping cleanup
2022-08-03 08:43:43.374 o.a.s.d.nimbus [INFO] not a leader, skipping assignments
2022-08-03 08:43:43.374 o.a.s.d.nimbus [INFO] not a leader, skipping cleanup
2022-08-03 08:43:53.374 o.a.s.d.nimbus [INFO] not a leader, skipping assignments
2022-08-03 08:43:53.375 o.a.s.d.nimbus [INFO] not a leader, skipping cleanup
2022-08-03 08:44:03.376 o.a.s.d.nimbus [INFO] not a leader, skipping assignments
2022-08-03 08:44:03.376 o.a.s.d.nimbus [INFO] not a leader, skipping cleanup
2022-08-03 08:44:12.025 o.a.s.z.Zookeeper [INFO] active-topology-blobs [] local-topology-blobs [] diff-topology-blobs []
2022-08-03 08:44:12.025 o.a.s.z.Zookeeper [INFO] active-topology-dependencies [] local-blobs [] diff-topology-dependencies []
2022-08-03 08:44:12.025 o.a.s.z.Zookeeper [INFO] Accepting leadership, all active topologies and corresponding dependencies found locally.
2022-08-03 08:44:13.468 o.a.s.d.nimbus [ERROR] Error when processing event
org.apache.storm.generated.KeyNotFoundException: null
at org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:147) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.blobstore.LocalFsBlobStore.getBlob(LocalFsBlobStore.java:270) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.blobstore.BlobStore.readBlobTo(BlobStore.java:271) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.blobstore.BlobStore.readBlob(BlobStore.java:300) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_251]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_251]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_251]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_251]
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.7.0.jar:?]
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$read_storm_conf_as_nimbus.invoke(nimbus.clj:542) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.daemon.nimbus$compute_executors.invoke(nimbus.clj:658) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.daemon.nimbus$compute_topology__GT_executors$iter__8280__8284$fn__8285.invoke(nimbus.clj:688) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
at clojure.lang.RT.seq(RT.java:507) ~[clojure-1.7.0.jar:?]
at clojure.core$seq__4128.invoke(core.clj:137) ~[clojure-1.7.0.jar:?]
at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30) ~[clojure-1.7.0.jar:?]
at clojure.core.protocols$fn__6506.invoke(protocols.clj:101) ~[clojure-1.7.0.jar:?]
at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) ~[clojure-1.7.0.jar:?]
at clojure.core$reduce.invoke(core.clj:6519) ~[clojure-1.7.0.jar:?]
at clojure.core$into.invoke(core.clj:6600) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$compute_topology__GT_executors.invoke(nimbus.clj:687) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.daemon.nimbus$compute_new_scheduler_assignments.invoke(nimbus.clj:842) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:1011) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at clojure.lang.RestFn.invoke(RestFn.java:410) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$fn__9323$exec_fn__1717__auto____9324$fn__9335.invoke(nimbus.clj:2479) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.timer$schedule_recurring$this__4483.invoke(timer.clj:105) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.timer$mk_timer$fn__4466$fn__4467.invoke(timer.clj:50) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.timer$mk_timer$fn__4466.invoke(timer.clj:42) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at clojure.lang.AFn.run(AFn.java:22) ~[clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
2022-08-03 08:44:13.475 o.a.s.util [ERROR] Halting process: ("Error when processing an event")
java.lang.RuntimeException: ("Error when processing an event")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$nimbus_data$fn__8110.invoke(nimbus.clj:213) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.timer$mk_timer$fn__4466$fn__4467.invoke(timer.clj:71) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.timer$mk_timer$fn__4466.invoke(timer.clj:42) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at clojure.lang.AFn.run(AFn.java:22) ~[clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
2022-08-03 08:44:13.479 o.a.s.d.nimbus [INFO] Shutting down master

 Here's the error on the other one Nimbus:

2022-08-03 08:43:00.530 o.a.s.d.nimbus [INFO] Starting nimbus server for storm version '1.1.0.2.6.5.1175-1'
2022-08-03 08:43:01.380 o.a.s.d.nimbus [ERROR] Error when processing event
org.apache.storm.generated.KeyNotFoundException: null
at org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:147) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.blobstore.LocalFsBlobStore.getBlob(LocalFsBlobStore.java:270) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.blobstore.BlobStore.readBlobTo(BlobStore.java:271) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.blobstore.BlobStore.readBlob(BlobStore.java:300) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_251]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_251]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_251]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_251]
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.7.0.jar:?]
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$read_storm_conf_as_nimbus.invoke(nimbus.clj:542) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.daemon.nimbus$compute_executors.invoke(nimbus.clj:658) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.daemon.nimbus$compute_topology__GT_executors$iter__8280__8284$fn__8285.invoke(nimbus.clj:688) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
at clojure.lang.RT.seq(RT.java:507) ~[clojure-1.7.0.jar:?]
at clojure.core$seq__4128.invoke(core.clj:137) ~[clojure-1.7.0.jar:?]
at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30) ~[clojure-1.7.0.jar:?]
at clojure.core.protocols$fn__6506.invoke(protocols.clj:101) ~[clojure-1.7.0.jar:?]
at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) ~[clojure-1.7.0.jar:?]
at clojure.core$reduce.invoke(core.clj:6519) ~[clojure-1.7.0.jar:?]
at clojure.core$into.invoke(core.clj:6600) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$compute_topology__GT_executors.invoke(nimbus.clj:687) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.daemon.nimbus$compute_new_scheduler_assignments.invoke(nimbus.clj:842) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:1011) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at clojure.lang.RestFn.invoke(RestFn.java:410) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$fn__9323$exec_fn__1717__auto____9324$fn__9335.invoke(nimbus.clj:2479) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.timer$schedule_recurring$this__4483.invoke(timer.clj:105) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.timer$mk_timer$fn__4466$fn__4467.invoke(timer.clj:50) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.timer$mk_timer$fn__4466.invoke(timer.clj:42) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at clojure.lang.AFn.run(AFn.java:22) ~[clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
2022-08-03 08:43:01.383 o.a.s.util [ERROR] Halting process: ("Error when processing an event")
java.lang.RuntimeException: ("Error when processing an event")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$nimbus_data$fn__8110.invoke(nimbus.clj:213) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.timer$mk_timer$fn__4466$fn__4467.invoke(timer.clj:71) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.timer$mk_timer$fn__4466.invoke(timer.clj:42) ~[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at clojure.lang.AFn.run(AFn.java:22) ~[clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]

 

1,336 Views
0 Kudos
1 REPLY 1

avatar
author-rank Babasaheb author-rank
Super Collaborator

Created ‎08-07-2022 05:18 AM

@cfoo 

 

I see the following in the Nimbus log:

 

[ERROR] Error when processing event
org.apache.storm.generated.KeyNotFoundException: null

 

Could you please try the below action plan and let us know how it goes: 

 

- Increase the JVM of the nimbus from the existing value to keep the nimbus up for a few additional moments to kill the topologies.

- stop all the topologies

- delete the znode /storm ( From zk cli : rmr /storm )

- take a backup of storm.local.dir and delete the same( From zk cli : rm -rf <value of stormlocal.dir> ). You can check it in the storm.yaml file

- start the supervisors and nimbus, check if the services are up and running

- If there is no Nimbus service running, then the only way to remove the topology processes would be to manually kill the worker processes on all worker nodes.

 

If you found this response assisted with your query, please take a moment to log in and click on  KUDOS 🙂 & ”Accept as Solution" below this post.

 

Thank you.

1,323 Views
0 Kudos
Announcements
What's New @ Cloudera
[RELEASED] Cloudera Streams Messaging - Kubernetes Operator ...
What's New @ Cloudera
[RELEASED] Cloudera Streaming Analytics 1.14 for Cloudera Pu...
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
View More Announcements