Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hortonworks Spark Atlas Connector not Capturing DDL Operations

Hortonworks Spark Atlas Connector not Capturing DDL Operations

New Contributor

hi,

I've setup a basic version of atlas and it works perfectly with hive and all the DDL operations and Lineage is being captured by atlas

changes made in hive-site.xml

<property>
<name>hive.exec.post.hooks</name>
<value>org.apache.atlas.hive.hook.HiveHook</value>
</property>

But when I try to create a table using spark-sql or spark-shell the DDL commands and lineage are not captured by atlas.

please help me with what I am doing wrong.

 

I launch spark-sql using the below command

spark-sql --jars /home/hadoop/harsha/spark-atlas-connector/spark-atlas-connector-assembly/target/spark-atlas-connector-assembly-0.1.0-SNAPSHOT.jar --conf spark.extraListeners=com.hortonworks.spark.atlas.SparkAtlasEventTracker --conf spark.sql.queryExecutionListeners=com.hortonworks.spark.atlas.SparkAtlasEventTracker --conf spark.sql.streaming.streamingQueryListeners=com.hortonworks.spark.atlas.SparkAtlasStreamingQueryEventTracker

Don't have an account?
Coming from Hortonworks? Activate your account here