Atlas: How to generate lineage between hive external table and hive ordinary table?

[#1]

Hi, Guys:

I am trying to using atlas to capture a lineage between the hive external table and hive ordinary table so as to make a lineage between the hive table and hbase table.

Here is the code I run in hive:

create table dddd(key string ,id bigint ,aname string ,bname string ,cname string)STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'WITH SERDEPROPERTIES("hbase.columns.mapping" = ":key,info:id,info:aname,info:bname,info:cname")TBLPROPERTIES("hbase.table.name" = "dddd");

insert into table datalake_demo.dddd select a.id as key, a.id, a.aname ,b.bname,c.cname from datalake_demo.a,datalake_demo.b,datalake_demo.c where d = b.id and b.id = c.id;

But the hive hook did not create a lineage between hive ordinary tables a, b and c with the hive external table d, I check the code and found there is no "insert into" option in hive hook to generate lineage

So my question is do you guys know how can I generate a lineage between hive external table and ordinary tables?

I am using atlas 0.7-incubating

Thanks so much for your help~

Qinglin

[#2 · Accepted solution]

The following link shows that Insert is an action supported by Hive Atlas Hook (see bottom, Limitations) http://atlas.incubator.apache.org/Bridge-Hive.html

Note that CTAS and Load, Import are also supported. Perhaps you can try these approaches.

[#3]

@Qinglin Xia

What is the version of Atlas and HDP ??

I encountered the same issue on 2.5.0.0 but see this output with 2.5.3 on how to reproduce lineage. Make sure that the Ranger Plugins for Atlas ,hbase,hive,hdfs,kafka have been enable with corresponding Ranger Policies !! see the Additional Requirements for Atlas with Ranger and Kerberos

Have a look at the attached document running Atlas in a kerberized the only difference if your environment is not kerberiszed s not to run the kinit:-)