Unable to integrate kafka and hbase into atlas 2.2.0

[#1]

Hi Team,

we are trying to integrate kafka and hbase into atlas 2.2.0
we are not using embedded kafka and hbase , seperately installed in same machine.
kafka version : 2.13-2.5.0
atlas version : 2.2.0

/home/ranger/apache-atlas-sources-2.2.0/hook-bin/import-kafka.sh --topic Maestrotopic3
>>>>> /home/ranger/apache-atlas-sources-2.2.0/hook-bin/import-kafka.sh
>>>>> /home/ranger/apache-atlas-sources-2.2.0
Using Kafka configuration directory [/data/kafka_2.13-2.5.0/config]
Log file for import is /var/log/atlas/import-kafka.log
Picked up _JAVA_OPTIONS: -Xms1024m -Xmx1024m -XX:MaxPermSize=512m
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.clas
s]
SLF4J: Found binding in [jar:file:/data/kafka_2.13-2.5.0/libs/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/lang/exception/NestableRuntimeException
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at org.apache.atlas.kafka.bridge.KafkaBridge.main(KafkaBridge.java:95)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.lang.exception.NestableRuntimeException
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 25 more
Failed to import Kafka Data Model!!!

FYI:
kafka-console-producer.sh and kafka-console-consumer.sh are running fine.
if we run the import-kafka.sh the without topic name , received similar error.

Can you please help .

[#2 · Accepted solution]

@Venkatd 

As per the initial description, it looks like you have installed the required components manually and trying integrate them but you are getting the following error:

ERROR:

Exception in thread "main" java.lang.NoClassDefFoundError

Kindly note that the above exception usually occurs when the dependencies in the classpath are missing that are required for the script/component.

First thing please very kafka-client jar (for example kafka-client-2.13-2.5.0.jar) exists in the classpath. 

If yes, then it requires "commons-lang" jar/dependencies in your classpath.

If you found this response assisted with your query, please take a moment to log in and click on  KUDOS 🙂 & ”Accept as Solution" below this post.

Thank you.

[#3 · Accepted solution]

@Venkatd 

As per the initial description, it looks like you have installed the required components manually and trying integrate them but you are getting the following error:

ERROR:

Exception in thread "main" java.lang.NoClassDefFoundError

Kindly note that the above exception usually occurs when the dependencies in the classpath are missing that are required for the script/component.

First thing please very kafka-client jar (for example kafka-client-2.13-2.5.0.jar) exists in the classpath. 

If yes, then it requires "commons-lang" jar/dependencies in your classpath.

If you found this response assisted with your query, please take a moment to log in and click on  KUDOS 🙂 & ”Accept as Solution" below this post.

Thank you.

[#4 · Accepted solution]

Hi  Babasaheb,

Sorry I was in vacation , I could not reply. Thanks for your help , it has resolved the issue with kafka.

I am facing issue with hbase integration , raised request in below link.

Unable-integrate-hbase-with-Atlas/td-p/350587

[#5]

@Venkatd, Has the reply helped resolve your issue? If so, can you please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future?