Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Atlas: How to generate lineage between hive external table and hive ordinary table?

Solved Go to solution

Atlas: How to generate lineage between hive external table and hive ordinary table?

New Contributor

Hi, Guys:

I am trying to using atlas to capture a lineage between the hive external table and hive ordinary table so as to make a lineage between the hive table and hbase table.

Here is the code I run in hive:

create table dddd(key string ,id bigint ,aname string ,bname string ,cname string)STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'WITH SERDEPROPERTIES("hbase.columns.mapping" = ":key,info:id,info:aname,info:bname,info:cname")TBLPROPERTIES("hbase.table.name" = "dddd");

insert into table datalake_demo.dddd select a.id as key, a.id, a.aname ,b.bname,c.cname from datalake_demo.a,datalake_demo.b,datalake_demo.c where d = b.id and b.id = c.id;
But the hive hook did not create a lineage between hive ordinary tables a, b and c with the hive external table d, I check the code and found there is no "insert into" option in hive hook to generate lineage
So my question is do you guys know how can I generate a lineage between hive external table and ordinary tables?
I am using atlas 0.7-incubating
Thanks so much for your help~
Qinglin
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Atlas: How to generate lineage between hive external table and hive ordinary table?

Guru

The following link shows that Insert is an action supported by Hive Atlas Hook (see bottom, Limitations) http://atlas.incubator.apache.org/Bridge-Hive.html

Note that CTAS and Load, Import are also supported. Perhaps you can try these approaches.

2 REPLIES 2

Re: Atlas: How to generate lineage between hive external table and hive ordinary table?

Guru

The following link shows that Insert is an action supported by Hive Atlas Hook (see bottom, Limitations) http://atlas.incubator.apache.org/Bridge-Hive.html

Note that CTAS and Load, Import are also supported. Perhaps you can try these approaches.

Re: Atlas: How to generate lineage between hive external table and hive ordinary table?

Mentor

@Qinglin Xia

What is the version of Atlas and HDP ??

I encountered the same issue on 2.5.0.0 but see this output with 2.5.3 on how to reproduce lineage. Make sure that the Ranger Plugins for Atlas ,hbase,hive,hdfs,kafka have been enable with corresponding Ranger Policies !! see the Additional Requirements for Atlas with Ranger and Kerberos

Have a look at the attached document running Atlas in a kerberized the only difference if your environment is not kerberiszed s not to run the kinit:-)