Member since
02-10-2020
4
Posts
0
Kudos Received
0
Solutions
03-01-2020
05:45 AM
Hi, did you happen to reach to the end of this? How did you add the tables to atlas once you know what tables are missing?
... View more
03-01-2020
01:50 AM
Hi, I am using atlas for lineage and data governance and all integrations with hive,spark and sqoop are up and running. But I want to know if there is a way to automate the process of adding all the missing tables to atlas at the end of every day in order to make sure that no data is missing. I wrote a script that gets the list of all tables from hive metastore and got the list of missing table from atlas by hitting its api. The plan was to call "import-hive.sh"(Inbuilt hive bridge of atlas) over the missing tables but the script gets stuck at an intermediate phase asking for input of username and password.so i am unable to automate it. I even tried things like these and nothing worked. printf "admin\nadmin" | sh import-hive.sh --database dummydb --table final can someone help me with this?
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Hive
-
Apache Spark
02-10-2020
10:55 PM
hi, I've setup a basic version of atlas and it works perfectly with hive and all the DDL operations and Lineage is being captured by atlas changes made in hive-site.xml <property> <name>hive.exec.post.hooks</name> <value>org.apache.atlas.hive.hook.HiveHook</value> </property> But when I try to create a table using spark-sql or spark-shell the DDL commands and lineage are not captured by atlas. please help me with what I am doing wrong. I launch spark-sql using the below command spark-sql --jars /home/hadoop/harsha/spark-atlas-connector/spark-atlas-connector-assembly/target/spark-atlas-connector-assembly-0.1.0-SNAPSHOT.jar --conf spark.extraListeners=com.hortonworks.spark.atlas.SparkAtlasEventTracker --conf spark.sql.queryExecutionListeners=com.hortonworks.spark.atlas.SparkAtlasEventTracker --conf spark.sql.streaming.streamingQueryListeners=com.hortonworks.spark.atlas.SparkAtlasStreamingQueryEventTracker
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Hive
-
Apache Spark