Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

SAM Application Kills Storm Supervisor?

avatar
Expert Contributor

Hi,

I was able to deploy a simple word count application on SAM and integrate with Schema Registry, which I really liked. However, soon after the deploy finished, my Storm supervisors halted with the below error. It looks like an authorization issue, but even I disabled Ranger plugin for Storm, it stills does not work.

2017-06-27 04:25:18.253 o.a.s.d.s.Slot [INFO] STATE WAITING_FOR_BASIC_LOCALIZATION msInState: 2243 -> WAITING_FOR_BLOB_LOCALIZATION msInState: 0
2017-06-27 04:25:18.289 o.a.s.u.NimbusClient [INFO] Found leader nimbus : test.example.com:6627
2017-06-27 04:25:18.439 o.a.s.l.AsyncLocalizer [WARN] Caught Exception While Downloading (rethrowing)...
java.io.IOException: Error getting blobs
        at org.apache.storm.localizer.Localizer.getBlobs(Localizer.java:471) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.call(AsyncLocalizer.java:254) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.call(AsyncLocalizer.java:211) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_77]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_77]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_77]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_77]
        at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_77]
        at org.apache.storm.localizer.Localizer.getBlobs(Localizer.java:460) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        ... 6 more
Caused by: java.lang.NullPointerException
        at org.apache.storm.utils.Utils.canUserReadBlob(Utils.java:1076) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.Localizer.downloadBlob(Localizer.java:534) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.Localizer.access$000(Localizer.java:65) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.Localizer$DownloadBlob.call(Localizer.java:505) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.Localizer$DownloadBlob.call(Localizer.java:481) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        ... 4 more
2017-06-27 04:25:18.441 o.a.s.d.s.Slot [ERROR] Error when processing event
java.util.concurrent.ExecutionException: java.io.IOException: Error getting blobs
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_77]
        at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[?:1.8.0_77]
        at org.apache.storm.localizer.LocalDownloadedResource$NoCancelFuture.get(LocalDownloadedResource.java:63) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.daemon.supervisor.Slot.handleWaitingForBlobLocalization(Slot.java:380) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.daemon.supervisor.Slot.stateMachineStep(Slot.java:275) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:740) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
Caused by: java.io.IOException: Error getting blobs
        at org.apache.storm.localizer.Localizer.getBlobs(Localizer.java:471) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.call(AsyncLocalizer.java:254) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.call(AsyncLocalizer.java:211) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_77]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_77]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_77]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_77]
        at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_77]
        at org.apache.storm.localizer.Localizer.getBlobs(Localizer.java:460) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.call(AsyncLocalizer.java:254) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.call(AsyncLocalizer.java:211) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_77]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_77]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_77]
        at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_77]
Caused by: java.lang.NullPointerException
        at org.apache.storm.utils.Utils.canUserReadBlob(Utils.java:1076) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.Localizer.downloadBlob(Localizer.java:534) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.Localizer.access$000(Localizer.java:65) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.Localizer$DownloadBlob.call(Localizer.java:505) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.Localizer$DownloadBlob.call(Localizer.java:481) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_77]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_77]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_77]
        at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_77]
2017-06-27 04:25:18.441 o.a.s.u.Utils [ERROR] Halting process: Error when processing an event
java.lang.RuntimeException: Halting process: Error when processing an event
        at org.apache.storm.utils.Utils.exitProcess(Utils.java:1774) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:774) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
2017-06-27 04:25:18.444 o.a.s.d.s.Supervisor [INFO] Shutting down supervisor 6c074218-7ac0-45d8-b1b9-52e3404bbf91
2017-06-27 08:32:26.030 o.a.s.z.Zookeeper [INFO] Staring ZK Curator


1 ACCEPTED SOLUTION

avatar
Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
3 REPLIES 3

avatar
Expert Contributor

In streamline.log, I found one error message. Not sure whether or not it relates to the issue.

INFO   [09:23:44.494] [ForkJoinPool-4-worker-11] c.h.s.s.a.s.t.StormTopologyActionsImpl -  Deploying Application WordCount
INFO   [09:23:44.495] [ForkJoinPool-4-worker-11] c.h.s.s.a.s.t.StormTopologyActionsImpl -  /usr/hdf/current/storm-client/bin/storm jar /tmp/storm-artifacts/streamline-8-WordCount/artifacts/streamline-runtime-storm-0.5.0.3.0.0.0-453.jar --jars /tmp/storm-artifacts/streamline-8-WordCount/jars/streamline-functions-282004f0-1cc7-4599-a49b-e6bde71ff3bf.jar --artifacts org.apache.kafka:kafka-clients:0.10.2.1,org.apache.storm:storm-kafka-client:1.1.0.3.0.0.0-453^org.slf4j:slf4j-log4j12^log4j:log4j^org.apache.zookeeper:zookeeper^org.apache.kafka:kafka-clients --artifactRepositories hwx-public^http://repo.hortonworks.com/content/groups/public/,hwx-private^http://nexus-private.hortonworks.com/nexus/content/groups/public/ -c nimbus.host=apac-shared0.field.hortonworks.com -c nimbus.port=6627 -c nimbus.thrift.max_buffer_size1048576 -c storm.thrift.transport=org.apache.storm.security.auth.SimpleTransportPlugin -c storm.principal.tolocal=org.apache.storm.security.auth.DefaultPrincipalToLocal org.apache.storm.flux.Flux --remote /tmp/storm-artifacts/streamline-8-WordCount.yaml
ERROR  [09:24:02.988] [dw-31 - POST /api/v1/catalog/topologies/8/versions/save] c.h.s.s.c.s.StreamCatalogService -  Streams should be specified.

avatar
Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Expert Contributor

Thanks @Jungtaek Lim

Upgrading to HDP2.6.1 and manually deleted all blobs solved the issue.