SAM Application Kills Storm Supervisor?

[#1]

Hi,

I was able to deploy a simple word count application on SAM and integrate with Schema Registry, which I really liked. However, soon after the deploy finished, my Storm supervisors halted with the below error. It looks like an authorization issue, but even I disabled Ranger plugin for Storm, it stills does not work.

2017-06-27 04:25:18.253 o.a.s.d.s.Slot [INFO] STATE WAITING_FOR_BASIC_LOCALIZATION msInState: 2243 -> WAITING_FOR_BLOB_LOCALIZATION msInState: 0
2017-06-27 04:25:18.289 o.a.s.u.NimbusClient [INFO] Found leader nimbus : test.example.com:6627
2017-06-27 04:25:18.439 o.a.s.l.AsyncLocalizer [WARN] Caught Exception While Downloading (rethrowing)...
java.io.IOException: Error getting blobs
        at org.apache.storm.localizer.Localizer.getBlobs(Localizer.java:471) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.call(AsyncLocalizer.java:254) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.call(AsyncLocalizer.java:211) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_77]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_77]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_77]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_77]
        at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_77]
        at org.apache.storm.localizer.Localizer.getBlobs(Localizer.java:460) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        ... 6 more
Caused by: java.lang.NullPointerException
        at org.apache.storm.utils.Utils.canUserReadBlob(Utils.java:1076) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.Localizer.downloadBlob(Localizer.java:534) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.Localizer.access$000(Localizer.java:65) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.Localizer$DownloadBlob.call(Localizer.java:505) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.Localizer$DownloadBlob.call(Localizer.java:481) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        ... 4 more
2017-06-27 04:25:18.441 o.a.s.d.s.Slot [ERROR] Error when processing event
java.util.concurrent.ExecutionException: java.io.IOException: Error getting blobs
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_77]
        at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[?:1.8.0_77]
        at org.apache.storm.localizer.LocalDownloadedResource$NoCancelFuture.get(LocalDownloadedResource.java:63) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.daemon.supervisor.Slot.handleWaitingForBlobLocalization(Slot.java:380) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.daemon.supervisor.Slot.stateMachineStep(Slot.java:275) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:740) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
Caused by: java.io.IOException: Error getting blobs
        at org.apache.storm.localizer.Localizer.getBlobs(Localizer.java:471) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.call(AsyncLocalizer.java:254) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.call(AsyncLocalizer.java:211) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_77]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_77]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_77]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_77]
        at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_77]
        at org.apache.storm.localizer.Localizer.getBlobs(Localizer.java:460) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.call(AsyncLocalizer.java:254) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.call(AsyncLocalizer.java:211) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_77]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_77]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_77]
        at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_77]
Caused by: java.lang.NullPointerException
        at org.apache.storm.utils.Utils.canUserReadBlob(Utils.java:1076) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.Localizer.downloadBlob(Localizer.java:534) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.Localizer.access$000(Localizer.java:65) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.Localizer$DownloadBlob.call(Localizer.java:505) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.localizer.Localizer$DownloadBlob.call(Localizer.java:481) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_77]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_77]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_77]
        at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_77]
2017-06-27 04:25:18.441 o.a.s.u.Utils [ERROR] Halting process: Error when processing an event
java.lang.RuntimeException: Halting process: Error when processing an event
        at org.apache.storm.utils.Utils.exitProcess(Utils.java:1774) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
        at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:774) ~[storm-core-1.1.0.2.6.0.3-8.jar:1.1.0.2.6.0.3-8]
2017-06-27 04:25:18.444 o.a.s.d.s.Supervisor [INFO] Shutting down supervisor 6c074218-7ac0-45d8-b1b9-52e3404bbf91
2017-06-27 08:32:26.030 o.a.s.z.Zookeeper [INFO] Staring ZK Curator

[#2]

In streamline.log, I found one error message. Not sure whether or not it relates to the issue.

INFO   [09:23:44.494] [ForkJoinPool-4-worker-11] c.h.s.s.a.s.t.StormTopologyActionsImpl -  Deploying Application WordCount
INFO   [09:23:44.495] [ForkJoinPool-4-worker-11] c.h.s.s.a.s.t.StormTopologyActionsImpl -  /usr/hdf/current/storm-client/bin/storm jar /tmp/storm-artifacts/streamline-8-WordCount/artifacts/streamline-runtime-storm-0.5.0.3.0.0.0-453.jar --jars /tmp/storm-artifacts/streamline-8-WordCount/jars/streamline-functions-282004f0-1cc7-4599-a49b-e6bde71ff3bf.jar --artifacts org.apache.kafka:kafka-clients:0.10.2.1,org.apache.storm:storm-kafka-client:1.1.0.3.0.0.0-453^org.slf4j:slf4j-log4j12^log4j:log4j^org.apache.zookeeper:zookeeper^org.apache.kafka:kafka-clients --artifactRepositories hwx-public^http://repo.hortonworks.com/content/groups/public/,hwx-private^http://nexus-private.hortonworks.com/nexus/content/groups/public/ -c nimbus.host=apac-shared0.field.hortonworks.com -c nimbus.port=6627 -c nimbus.thrift.max_buffer_size1048576 -c storm.thrift.transport=org.apache.storm.security.auth.SimpleTransportPlugin -c storm.principal.tolocal=org.apache.storm.security.auth.DefaultPrincipalToLocal org.apache.storm.flux.Flux --remote /tmp/storm-artifacts/streamline-8-WordCount.yaml
ERROR  [09:24:02.988] [dw-31 - POST /api/v1/catalog/topologies/8/versions/save] c.h.s.s.c.s.StreamCatalogService -  Streams should be specified.

[#3 · Accepted solution]

That was a bug on Storm side, tracked and fixed in HDP 2.6.1.1 and HDF 3.0.0.0.

Please upgrade your HDP cluster or use HDF Storm cluster and let me know the problem persists.

Unfortunately this is a bug regarding "uploading" artifact which uploaded artifact blobs have bad ACL, so you actually need to delete artifact blobs manually after upgrading.

[#4]

Thanks @Jungtaek Lim

Upgrading to HDP2.6.1 and manually deleted all blobs solved the issue.