question Atlas: How to generate lineage between hive external table and hive ordinary table? in Archives of Support Questions (Read Only)

Atlas: How to generate lineage between hive external table and hive ordinary table?

xiaqinglinv5 — Tue, 23 May 2017 16:14:01 GMT

Hi, Guys:

I am trying to using atlas to capture a lineage between the hive external table and hive ordinary table so as to make a lineage between the hive table and hbase table.

Here is the code I run in hive:

create table dddd(key string ,id bigint ,aname string ,bname string ,cname string)STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'WITH SERDEPROPERTIES("hbase.columns.mapping" = ":key,info:id,info:aname,info:bname,info:cname")TBLPROPERTIES("hbase.table.name" = "dddd");

insert into table datalake_demo.dddd select a.id as key, a.id, a.aname ,b.bname,c.cname from datalake_demo.a,datalake_demo.b,datalake_demo.c where d = b.id and b.id = c.id;

But the hive hook did not create a lineage between hive ordinary tables a, b and c with the hive external table d, I check the code and found there is no "insert into" option in hive hook to generate lineage

So my question is do you guys know how can I generate a lineage between hive external table and ordinary tables?

I am using atlas 0.7-incubating

Thanks so much for your help~

Qinglin

Re: Atlas: How to generate lineage between hive external table and hive ordinary table?

gkeys — Tue, 23 May 2017 21:09:44 GMT

The following link shows that Insert is an action supported by Hive Atlas Hook (see bottom, Limitations) http://atlas.incubator.apache.org/Bridge-Hive.html

Note that CTAS and Load, Import are also supported. Perhaps you can try these approaches.

Re: Atlas: How to generate lineage between hive external table and hive ordinary table?

Shelton — Tue, 23 May 2017 21:59:40 GMT

@Qinglin Xia

What is the version of Atlas and HDP ??

I encountered the same issue on 2.5.0.0 but see this output with 2.5.3 on how to reproduce lineage. Make sure that the Ranger Plugins for Atlas ,hbase,hive,hdfs,kafka have been enable with corresponding Ranger Policies !! see the Additional Requirements for Atlas with Ranger and Kerberos

Have a look at the attached document running Atlas in a kerberized the only difference if your environment is not kerberiszed s not to run the kinit:-)