question Re: Storm - Atlas integration error (StormAtlasHook) in Archives of Support Questions (Read Only)

Storm - Atlas integration error (StormAtlasHook)

fernando_lopez — Tue, 24 Oct 2017 02:04:52 GMT

When running Storm - Atlas integration for "Cross Component Lineage with Apache Atlas across Apache Sqoop, Hive, Kafka & Storm" tutorial, at step

	sh 004-run-storm-job.sh

we get a class not found exception:

Exception in thread "main" org.apache.storm.hooks.SubmitterHookException: java.lang.ClassNotFoundException: org.apache.atlas.storm.hook.StormAtlasHook  at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:368)  at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:278)  at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:390)  at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:162)  at com.dsinpractice.storm.samples.WordCountTopology.main(WordCountTopology.java:140)

Any hints here?

storm-error-dump.txt

Regards

Re: Storm - Atlas integration error (StormAtlasHook)

fernando_lopez — Tue, 24 Oct 2017 02:14:41 GMT

Storm error check shows the same issue.

Altough this was reported in an older version (2.5.x), it seems to be the same problem. How can it be workarounded for 2.6.x?

Re: Storm - Atlas integration error (StormAtlasHook)

jsensharma — Tue, 24 Oct 2017 02:28:55 GMT

@Fernando Lopez Bello

Do you see a jar file with some name like "storm-bridge" inside the following location?

/usr/hdp/current/atlas-client/hook/storm/atlas-storm-plugin-impl/storm-bridge-xxx.xxxxxxxxx.jar

Like:

/usr/hdp/current/atlas-client/hook/storm/atlas-storm-plugin-impl/storm-bridge-0.8.0.2.6.0.3-8.jar

Can you try adding that JAR to your command?

May be you can try putting that jar inside the following dir to see if it works.

/usr/hdp/current/storm-client/lib
/usr/hdp/current/storm-client/extlib

Re: Storm - Atlas integration error (StormAtlasHook)

fernando_lopez — Tue, 24 Oct 2017 02:31:50 GMT

Yes, there is:

[root@sandbox storm-demo]# ls -l /usr/hdp/current/atlas-client/hook/storm/atlas-storm-plugin-impl/*storm-bridge*

-rw-r--r-- 1 root root 27999 Apr  1  2017 /usr/hdp/current/atlas-client/hook/storm/atlas-storm-plugin-impl/storm-bridge-0.8.0.2.6.0.3-8.jar

Re: Storm - Atlas integration error (StormAtlasHook)

jsensharma — Tue, 24 Oct 2017 02:34:22 GMT

@Fernando Lopez Bello

May be you can try putting that jar inside the following dir to see if it works.

    /usr/hdp/current/storm-client/lib
    /usr/hdp/current/storm-client/extlib

Re: Storm - Atlas integration error (StormAtlasHook)

jsensharma — Tue, 24 Oct 2017 02:39:29 GMT

@Fernando Lopez Bello

I see the script which says (Although your HDP version seems to be 2.6)

In HDP 2.3, 2.4, and 2.5.0.0, Sqoop and Storm still relied on the following method to setup Atlas hooks because the RPM for Sqoop and Storm did not bring in any dependencies.

https://github.com/apache/ambari/blob/trunk/ambari-common/src/main/python/resource_management/libraries/functions/setup_atlas_hook.py#L151-L152

Re: Storm - Atlas integration error (StormAtlasHook)

jsensharma — Sun, 18 Aug 2019 01:26:44 GMT

@Fernando Lopez Bello

Do you see the "Enable Atlas Hook" Checkbox is enabled?

Re: Storm - Atlas integration error (StormAtlasHook)

fernando_lopez — Tue, 24 Oct 2017 02:58:34 GMT

Thanks. I did include the referred jar as you suggested:

storm jar /root/tutoriales/crosscomponent_scripts/storm-demo/lib/storm-samples-1.0-jar-with-dependencies.jar com.dsinpractice.storm.samples.WordCountTopology --cluster true --name storm-demo-topology-03 --path /user/storm/storm-hdfs-test-01 --topic my-topic-01 --jars /usr/hdp/current/atlas-client/hook/storm/atlas-storm-plugin-impl/storm-bridge-0.8.0.2.6.0.3-8.jar

but now there is another missing class "AtlasHook":

4930 [main] INFO  o.a.s.StormSubmitter - Initializing the registered ISubmitterHook [org.apache.atlas.storm.hook.StormAtlasHook]Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/atlas/hook/AtlasHook  at java.lang.ClassLoader.defineClass1(Native Method)  at java.lang.ClassLoader.defineClass(ClassLoader.java:763)

Another jar that needs to be referred?

Re: Storm - Atlas integration error (StormAtlasHook)

fernando_lopez — Tue, 24 Oct 2017 03:11:49 GMT

Thanks. That check did the trick.

(It is not explicitly indicated in the above mentioned tutorial, for Storm)

Re: Storm - Atlas integration error (StormAtlasHook)

erkansirin78 — Thu, 19 Jul 2018 21:41:12 GMT

I copied

/usr/hdp/current/atlas-client/hook/storm/atlas-storm-plugin-impl/storm-bridge-xxx.xxxxxxxxx.jar

/usr/hdp/current/storm-client/lib
/usr/hdp/current/storm-client/lib/usr/hdp/current/storm-client/extlib

but it didin't work.