Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Storm - Atlas integration error (StormAtlasHook)

avatar
Expert Contributor
hdp-version-info.txt

When running Storm - Atlas integration for "Cross Component Lineage with Apache Atlas across Apache Sqoop, Hive, Kafka & Storm" tutorial, at step

	sh 004-run-storm-job.sh

we get a class not found exception:

Exception in thread "main" org.apache.storm.hooks.SubmitterHookException: java.lang.ClassNotFoundException: org.apache.atlas.storm.hook.StormAtlasHook  at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:368)  at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:278)  at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:390)  at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:162)  at com.dsinpractice.storm.samples.WordCountTopology.main(WordCountTopology.java:140)


Any hints here?

storm-error-dump.txt

Regards

1 ACCEPTED SOLUTION

avatar
Master Mentor
@Fernando Lopez Bello

Do you see the "Enable Atlas Hook" Checkbox is enabled?

41500-storm-hook.png

.

View solution in original post

9 REPLIES 9

avatar
Expert Contributor


Storm error check shows the same issue.

Altough this was reported in an older version (2.5.x), it seems to be the same problem. How can it be workarounded for 2.6.x?

avatar
Master Mentor

@Fernando Lopez Bello

Do you see a jar file with some name like "storm-bridge" inside the following location?

/usr/hdp/current/atlas-client/hook/storm/atlas-storm-plugin-impl/storm-bridge-xxx.xxxxxxxxx.jar


Like:

/usr/hdp/current/atlas-client/hook/storm/atlas-storm-plugin-impl/storm-bridge-0.8.0.2.6.0.3-8.jar


Can you try adding that JAR to your command?

OR

May be you can try putting that jar inside the following dir to see if it works.

/usr/hdp/current/storm-client/lib
/usr/hdp/current/storm-client/extlib

.

avatar
Master Mentor

@Fernando Lopez Bello

May be you can try putting that jar inside the following dir to see if it works.

    /usr/hdp/current/storm-client/lib
    /usr/hdp/current/storm-client/extlib

avatar
Master Mentor
@Fernando Lopez Bello

I see the script which says (Although your HDP version seems to be 2.6)

In HDP 2.3, 2.4, and 2.5.0.0, Sqoop and Storm still relied on the following method to setup Atlas hooks because the RPM for Sqoop and Storm did not bring in any dependencies.

https://github.com/apache/ambari/blob/trunk/ambari-common/src/main/python/resource_management/librar...

avatar
Expert Contributor

I copied

/usr/hdp/current/atlas-client/hook/storm/atlas-storm-plugin-impl/storm-bridge-xxx.xxxxxxxxx.jar

to

/usr/hdp/current/storm-client/lib
/usr/hdp/current/storm-client/lib/usr/hdp/current/storm-client/extlib

but it didin't work.

avatar
Expert Contributor

Yes, there is:

[root@sandbox storm-demo]# ls -l /usr/hdp/current/atlas-client/hook/storm/atlas-storm-plugin-impl/*storm-bridge*

-rw-r--r-- 1 root root 27999 Apr  1  2017 /usr/hdp/current/atlas-client/hook/storm/atlas-storm-plugin-impl/storm-bridge-0.8.0.2.6.0.3-8.jar

avatar
Master Mentor
@Fernando Lopez Bello

Do you see the "Enable Atlas Hook" Checkbox is enabled?

41500-storm-hook.png

.

avatar
Expert Contributor

Thanks. I did include the referred jar as you suggested:

storm jar /root/tutoriales/crosscomponent_scripts/storm-demo/lib/storm-samples-1.0-jar-with-dependencies.jar com.dsinpractice.storm.samples.WordCountTopology --cluster true --name storm-demo-topology-03 --path /user/storm/storm-hdfs-test-01 --topic my-topic-01 --jars /usr/hdp/current/atlas-client/hook/storm/atlas-storm-plugin-impl/storm-bridge-0.8.0.2.6.0.3-8.jar

but now there is another missing class "AtlasHook":

4930 [main] INFO  o.a.s.StormSubmitter - Initializing the registered ISubmitterHook [org.apache.atlas.storm.hook.StormAtlasHook]Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/atlas/hook/AtlasHook  at java.lang.ClassLoader.defineClass1(Native Method)  at java.lang.ClassLoader.defineClass(ClassLoader.java:763)

Another jar that needs to be referred?

avatar
Expert Contributor

Thanks. That check did the trick.

(It is not explicitly indicated in the above mentioned tutorial, for Storm)