Storm - Atlas integration error (StormAtlasHook)

[#1]

hdp-version-info.txt

When running Storm - Atlas integration for "Cross Component Lineage with Apache Atlas across Apache Sqoop, Hive, Kafka & Storm" tutorial, at step

	sh 004-run-storm-job.sh

we get a class not found exception:

Exception in thread "main" org.apache.storm.hooks.SubmitterHookException: java.lang.ClassNotFoundException: org.apache.atlas.storm.hook.StormAtlasHook  at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:368)  at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:278)  at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:390)  at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:162)  at com.dsinpractice.storm.samples.WordCountTopology.main(WordCountTopology.java:140)

Any hints here?

storm-error-dump.txt

Regards

[#2]

Storm error check shows the same issue.

Altough this was reported in an older version (2.5.x), it seems to be the same problem. How can it be workarounded for 2.6.x?

[#3]

@Fernando Lopez Bello

Do you see a jar file with some name like "storm-bridge" inside the following location?

/usr/hdp/current/atlas-client/hook/storm/atlas-storm-plugin-impl/storm-bridge-xxx.xxxxxxxxx.jar

Like:

/usr/hdp/current/atlas-client/hook/storm/atlas-storm-plugin-impl/storm-bridge-0.8.0.2.6.0.3-8.jar

Can you try adding that JAR to your command?

OR

May be you can try putting that jar inside the following dir to see if it works.

/usr/hdp/current/storm-client/lib
/usr/hdp/current/storm-client/extlib

.

[#4]

@Fernando Lopez Bello

May be you can try putting that jar inside the following dir to see if it works.

    /usr/hdp/current/storm-client/lib
    /usr/hdp/current/storm-client/extlib

[#5]

@Fernando Lopez Bello

I see the script which says (Although your HDP version seems to be 2.6)

In HDP 2.3, 2.4, and 2.5.0.0, Sqoop and Storm still relied on the following method to setup Atlas hooks because the RPM for Sqoop and Storm did not bring in any dependencies.

https://github.com/apache/ambari/blob/trunk/ambari-common/src/main/python/resource_management/librar...

[#6]

I copied

/usr/hdp/current/atlas-client/hook/storm/atlas-storm-plugin-impl/storm-bridge-xxx.xxxxxxxxx.jar

to

/usr/hdp/current/storm-client/lib
/usr/hdp/current/storm-client/lib/usr/hdp/current/storm-client/extlib

but it didin't work.

[#7]

Yes, there is:

[root@sandbox storm-demo]# ls -l /usr/hdp/current/atlas-client/hook/storm/atlas-storm-plugin-impl/*storm-bridge*

-rw-r--r-- 1 root root 27999 Apr  1  2017 /usr/hdp/current/atlas-client/hook/storm/atlas-storm-plugin-impl/storm-bridge-0.8.0.2.6.0.3-8.jar

[#8 · Accepted solution]

@Fernando Lopez Bello

Do you see the "Enable Atlas Hook" Checkbox is enabled?

.

[#9]

Thanks. I did include the referred jar as you suggested:

storm jar /root/tutoriales/crosscomponent_scripts/storm-demo/lib/storm-samples-1.0-jar-with-dependencies.jar com.dsinpractice.storm.samples.WordCountTopology --cluster true --name storm-demo-topology-03 --path /user/storm/storm-hdfs-test-01 --topic my-topic-01 --jars /usr/hdp/current/atlas-client/hook/storm/atlas-storm-plugin-impl/storm-bridge-0.8.0.2.6.0.3-8.jar

but now there is another missing class "AtlasHook":

4930 [main] INFO  o.a.s.StormSubmitter - Initializing the registered ISubmitterHook [org.apache.atlas.storm.hook.StormAtlasHook]Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/atlas/hook/AtlasHook  at java.lang.ClassLoader.defineClass1(Native Method)  at java.lang.ClassLoader.defineClass(ClassLoader.java:763)

Another jar that needs to be referred?

[#10]

Thanks. That check did the trick.

(It is not explicitly indicated in the above mentioned tutorial, for Storm)