Support Questions
Find answers, ask questions, and share your expertise

Hortonworks Spark Atlas Connector not Capturing DDL Operations

Hortonworks Spark Atlas Connector not Capturing DDL Operations

New Contributor

hi,

I've setup a basic version of atlas and it works perfectly with hive and all the DDL operations and Lineage is being captured by atlas

changes made in hive-site.xml

<property>
<name>hive.exec.post.hooks</name>
<value>org.apache.atlas.hive.hook.HiveHook</value>
</property>

But when I try to create a table using spark-sql or spark-shell the DDL commands and lineage are not captured by atlas.

please help me with what I am doing wrong.

 

I launch spark-sql using the below command

spark-sql --jars /home/hadoop/harsha/spark-atlas-connector/spark-atlas-connector-assembly/target/spark-atlas-connector-assembly-0.1.0-SNAPSHOT.jar --conf spark.extraListeners=com.hortonworks.spark.atlas.SparkAtlasEventTracker --conf spark.sql.queryExecutionListeners=com.hortonworks.spark.atlas.SparkAtlasEventTracker --conf spark.sql.streaming.streamingQueryListeners=com.hortonworks.spark.atlas.SparkAtlasStreamingQueryEventTracker

1 REPLY 1

Re: Hortonworks Spark Atlas Connector not Capturing DDL Operations

New Contributor

Did you find a solution for this Harsha?