Hi, I am using atlas for lineage and data governance and all integrations with hive,spark and sqoop are up and running. But I want to know if there is a way to automate the process of adding all the missing tables to atlas at the end of every day in order to make sure that no data is missing.   I wrote a script that gets the list of all tables from hive metastore and got the list of missing table from atlas by hitting its api. The plan was to call "import-hive.sh"(Inbuilt hive bridge of atlas) over the missing tables but the script gets stuck at an intermediate phase asking for input of username and password.so i am unable to automate it.   I even tried things like these and nothing worked. printf "admin\nadmin" | sh import-hive.sh --database dummydb --table final   can someone help me with this? ... View more

Hi,   I am using Hortonworks spark atlas connecter to capture spark lineage but the issue i am facing here is that the connector captures lineage based on application id instead of app name.   This is causing a lot of redundancy and the lineage associated with hive tables becomes non readable. Can someone please suggest a workaround for the same.   sample: https://drive.google.com/open?id=1xZY2XtnxRFHeTlBDq6ljecB3FbD4S-LW   https://drive.google.com/open?id=1g3HMqAm-OWYmbrSZJ58bJ5CEljqxEOyH ... View more

hi, I've setup a basic version of atlas and it works perfectly with hive and all the DDL operations and Lineage is being captured by atlas changes made in hive-site.xml <property> <name>hive.exec.post.hooks</name> <value>org.apache.atlas.hive.hook.HiveHook</value> </property> But when I try to create a table using spark-sql or spark-shell the DDL commands and lineage are not captured by atlas. please help me with what I am doing wrong.   I launch spark-sql using the below command spark-sql --jars /home/hadoop/harsha/spark-atlas-connector/spark-atlas-connector-assembly/target/spark-atlas-connector-assembly-0.1.0-SNAPSHOT.jar --conf spark.extraListeners=com.hortonworks.spark.atlas.SparkAtlasEventTracker --conf spark.sql.queryExecutionListeners=com.hortonworks.spark.atlas.SparkAtlasEventTracker --conf spark.sql.streaming.streamingQueryListeners=com.hortonworks.spark.atlas.SparkAtlasStreamingQueryEventTracker ... View more