Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

Solved Go to solution

Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

New Contributor

I am trying to see the column level lineage in apache atlas 0.8. To get the lineage i have created a table by using CTAS statement over a subset of columns of a different table. As per this link, Ideally i should get the lineage.but in my Atlas, Lineage is not populating.

Thank you,

Subash

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

New Contributor

@subash sharma This issue is still open UI changes for column level lineage ...Please refer (ATLAS-1181)

8 REPLIES 8

Re: Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

Mentor

@subash sharma

Is Hive-Hook enabled? Check that the hooks are enable in the Ranger config

Highlighted

Re: Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

New Contributor

Hey @Geoffrey Shelton Okot,

Yes Hive Hook enabled, I can see the lineage of hive_table. Only the hive_column_lineage is not showing up.

Re: Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

@subash sharma

Which version of HDP are you using? Even though Column Level Lineage is advertised as available with Atlas 0.8 in the Apache page, it has only been made GA with HDP 2.6.1 rather than HDP 2.6.0. The delay was to ensure it works with the other appropriate HiveQL commands beyond CTAS.

Re: Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

New Contributor

I am using HDP 2.6.1

Re: Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

Expert Contributor

@subash sharma, do you see the lineage graph rendered in UI when you select one of the hive column in CTAS table ?

Alternatively could you issue the below curl command on one of the column guid and see if you can get edges with label "__Process.outputs" or "__Process.inputs".

curl -X GET "http://localhost:21000/api/atlas/discovery/search/gremlin?query=g.V("__guid","<column_guid>").inE.toList()"

Hope this helps.

Re: Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

New Contributor

@Sarath Subramanian, Yes i can see the "__Process.outputs" for the CTAS column. Attaching output below :

{"requestId":"pool-2-thread-4 - 4718810e-6302-48cb-9b75-d6d665242731","query":"g.V(\"__guid\",\"90b6074b-532e-4562-8b37-42a8636926ac\").inE.toList()","queryType":"gremlin","results":[{"__modifiedBy":"admin","__state":"ACTIVE","__createdBy":"admin","inVertex":"163856480","__modificationTimestamp":"1499686956258","id":"2pkg7r-2pk0jc-opp1-2pk0gw","label":"__hive_column.table","outVertex":"163856568","__timestamp":"1499685457722"},{"__modifiedBy":"admin","__state":"ACTIVE","__createdBy":"admin","inVertex":"163856480","__modificationTimestamp":"1499686956258","id":"2pkw1e-2pk0pc-opp1-2pk0gw","label":"__hive_column.table","outVertex":"163856784","__timestamp":"1499685457722"},{"__modifiedBy":"admin","__state":"ACTIVE","__createdBy":"admin","inVertex":"163856480","__modificationTimestamp":"1499686956258","id":"2pl8oi-2pkdcg-osut-2pk0gw","label":"__hive_storagedesc.table","outVertex":"163873168","__timestamp":"1499685457722"},{"__modifiedBy":"admin","__state":"ACTIVE","__createdBy":"admin","inVertex":"163856480","__modificationTimestamp":"1499686956258","id":"2pmiia-2pkpzk-1a80l-2pk0gw","label":"__Process.outputs","outVertex":"163889552","__timestamp":"1499685462156"}],"count":4}

Let me know where i am going wrong.

Re: Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

New Contributor

@subash sharma This issue is still open UI changes for column level lineage ...Please refer (ATLAS-1181)

Re: Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

New Contributor

Thank you @Parag Redij