In SAM Topology submission failed when storm hook for Atlas is enabled: ClassNotFoundException

[#1]

Hello Guys,

I have setup a standard HDP 2.6.1.0 & HDF 3.0 cluster.

I have enabled the Storm hook for Atlas in Storm config using Ambari. When I disable the hook everything works fine.

But when the hook is enabled, I can not submit a topology anymore failing with the following error message in SAM UI:

Topology submission failed due to: java.lang.Exception: Topology could not be deployed successfully: storm deploy command failed with Exception in thread "main" org.apache.storm.hooks.SubmitterHookException: java.lang.ClassNotFoundException: org.apache.atlas.storm.hook.StormAtlasHook

I have checked on the Storm and all worker nodes that the following jar files are available in /usr/hdp/current/storm-client/extlib as symlink:

lrwxrwxrwx1 root root atlas-plugin-classloader-0.8.0.2.6.1.0-129.jar -> /usr/hdp/current/atlas-server/hook/storm/atlas-plugin-classloader-0.8.0.2.6.1.0-129.jar

lrwxrwxrwx1 root root storm-bridge-shim-0.8.0.2.6.1.0-129.jar -> /usr/hdp/current/atlas-server/hook/storm/storm-bridge-shim-0.8.0.2.6.1.0-129.jar

This is the result of the OOTB Ambari installation and I have installed it fresh - no upgrade.

The java lib path looks like this:

java.library.path /usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib:/usr/hdp/current/storm-client/extlib

So the required jar files are in the build path, right? Could it be a permission issue?

Any ideas? I am deploying the Storm topology using Streaming Analytics Manager running on the same host as Storm.

BR,

Alexander

[#2]

Can you share the stack trace of the error ?

[#3]

The full stack trace from /var/log/streamline/streamline.log:

11675 [main] WARNo.a.s.StormSubmitter - Error occurred in invoking submitter hook:[org.apache.atlas.storm.hook.StormAtlasHook]

java.lang.ClassNotFoundException: org.apache.atlas.storm.hook.StormAtlasHook

at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_131]

at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_131]

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) ~[?:1.8.0_131]

at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_131]

at java.lang.Class.forName0(Native Method) ~[?:1.8.0_131]

at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_131]

at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:361) [storm-core-1.1.0.3.0.0.0-453.jar:1.1.0.3.0.0.0-453]

at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:278) [storm-core-1.1.0.3.0.0.0-453.jar:1.1.0.3.0.0.0-453]

at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:390) [storm-core-1.1.0.3.0.0.0-453.jar:1.1.0.3.0.0.0-453]

at org.apache.storm.flux.Flux.runCli(Flux.java:171) [448dabba7eb011e793c45254006c55a5.jar:?]

at org.apache.storm.flux.Flux.main(Flux.java:98) [448dabba7eb011e793c45254006c55a5.jar:?]

Exception in thread "main" org.apache.storm.hooks.SubmitterHookException: java.lang.ClassNotFoundException: org.apache.atlas.storm.hook.StormAtlasHook

at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:368)

at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:278)

at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:390)

at org.apache.storm.flux.Flux.runCli(Flux.java:171)

at org.apache.storm.flux.Flux.main(Flux.java:98)

Caused by: java.lang.ClassNotFoundException: org.apache.atlas.storm.hook.StormAtlasHook

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:264)

at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:361)

... 4 more

ERROR[18:17:28.844] [ForkJoinPool-4-worker-3] c.h.s.s.a.t.s.TopologyStates -Error while trying to deploy the topology in the streaming engine

java.lang.Exception: Topology could not be deployed successfully: storm deploy command failed with Exception in thread "main" org.apache.storm.hooks.SubmitterHookException: java.lang.ClassNotFoundException: org.apache.atlas.storm.hook.StormAtlasHook

at com.hortonworks.streamline.streams.actions.storm.topology.StormTopologyActionsImpl.deploy(StormTopologyActionsImpl.java:254)

at com.hortonworks.streamline.streams.actions.topology.state.TopologyStates$5.deploy(TopologyStates.java:120)

at com.hortonworks.streamline.streams.actions.topology.state.TopologyContext.deploy(TopologyContext.java:87)

at com.hortonworks.streamline.streams.actions.topology.service.TopologyActionsService.deployTopology(TopologyActionsService.java:116)

at com.hortonworks.streamline.streams.service.TopologyCatalogResource.lambda$deploy$3(TopologyCatalogResource.java:493)

at com.hortonworks.streamline.common.util.ParallelStreamUtil.lambda$runAsync$0(ParallelStreamUtil.java:56)

at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)

at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582)

at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)

at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)

at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)

at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

ERROR[18:17:28.858] [ForkJoinPool-4-worker-3] c.h.s.c.u.ParallelStreamUtil -Got exception while running async task

java.lang.RuntimeException: java.lang.Exception: Topology could not be deployed successfully: storm deploy command failed with Exception in thread "main" org.apache.storm.hooks.SubmitterHookException: java.lang.ClassNotFoundException: org.apache.atlas.storm.hook.StormAtlasHook

at com.hortonworks.streamline.common.util.ParallelStreamUtil.lambda$runAsync$0(ParallelStreamUtil.java:58)

at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)

at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582)

at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)

at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)

at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)

at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

Caused by: java.lang.Exception: Topology could not be deployed successfully: storm deploy command failed with Exception in thread "main" org.apache.storm.hooks.SubmitterHookException: java.lang.ClassNotFoundException: org.apache.atlas.storm.hook.StormAtlasHook

at com.hortonworks.streamline.streams.actions.storm.topology.StormTopologyActionsImpl.deploy(StormTopologyActionsImpl.java:254)

at com.hortonworks.streamline.streams.actions.topology.state.TopologyStates$5.deploy(TopologyStates.java:120)

at com.hortonworks.streamline.streams.actions.topology.state.TopologyContext.deploy(TopologyContext.java:87)

at com.hortonworks.streamline.streams.actions.topology.service.TopologyActionsService.deployTopology(TopologyActionsService.java:116)

at com.hortonworks.streamline.streams.service.TopologyCatalogResource.lambda$deploy$3(TopologyCatalogResource.java:493)

at com.hortonworks.streamline.common.util.ParallelStreamUtil.lambda$runAsync$0(ParallelStreamUtil.java:56)

... 6 common frames omitted

[#4]

I think storm is missing a JAR link to the Atlas Hook Implementation specific to storm. Can you check for atlas-storm-plugin-impl dir and add that to the classpath as well ?

[#5]

Now, I added it, but it doesn't solve the issue. Do you know if this could be a problem with the symbolic links? Interesting thing is that even when the submission fails in SAM with this error, afterwards the storm topology is up and running. You can also view it in SAM. The error you find in streamline log, not storm log.