Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

avatar
Expert Contributor

I am trying to see the column level lineage in apache atlas 0.8. To get the lineage i have created a table by using CTAS statement over a subset of columns of a different table. As per this link, Ideally i should get the lineage.but in my Atlas, Lineage is not populating.

Thank you,

Subash

1 ACCEPTED SOLUTION

avatar
New Contributor

@subash sharma This issue is still open UI changes for column level lineage ...Please refer (ATLAS-1181)

View solution in original post

8 REPLIES 8

avatar
Master Mentor

@subash sharma

Is Hive-Hook enabled? Check that the hooks are enable in the Ranger config

avatar
Expert Contributor

Hey @Geoffrey Shelton Okot,

Yes Hive Hook enabled, I can see the lineage of hive_table. Only the hive_column_lineage is not showing up.

avatar

@subash sharma

Which version of HDP are you using? Even though Column Level Lineage is advertised as available with Atlas 0.8 in the Apache page, it has only been made GA with HDP 2.6.1 rather than HDP 2.6.0. The delay was to ensure it works with the other appropriate HiveQL commands beyond CTAS.

avatar
Expert Contributor

I am using HDP 2.6.1

avatar
Expert Contributor

@subash sharma, do you see the lineage graph rendered in UI when you select one of the hive column in CTAS table ?

Alternatively could you issue the below curl command on one of the column guid and see if you can get edges with label "__Process.outputs" or "__Process.inputs".

curl -X GET "http://localhost:21000/api/atlas/discovery/search/gremlin?query=g.V("__guid","<column_guid>").inE.toList()"

Hope this helps.

avatar
Expert Contributor

@Sarath Subramanian, Yes i can see the "__Process.outputs" for the CTAS column. Attaching output below :

{"requestId":"pool-2-thread-4 - 4718810e-6302-48cb-9b75-d6d665242731","query":"g.V(\"__guid\",\"90b6074b-532e-4562-8b37-42a8636926ac\").inE.toList()","queryType":"gremlin","results":[{"__modifiedBy":"admin","__state":"ACTIVE","__createdBy":"admin","inVertex":"163856480","__modificationTimestamp":"1499686956258","id":"2pkg7r-2pk0jc-opp1-2pk0gw","label":"__hive_column.table","outVertex":"163856568","__timestamp":"1499685457722"},{"__modifiedBy":"admin","__state":"ACTIVE","__createdBy":"admin","inVertex":"163856480","__modificationTimestamp":"1499686956258","id":"2pkw1e-2pk0pc-opp1-2pk0gw","label":"__hive_column.table","outVertex":"163856784","__timestamp":"1499685457722"},{"__modifiedBy":"admin","__state":"ACTIVE","__createdBy":"admin","inVertex":"163856480","__modificationTimestamp":"1499686956258","id":"2pl8oi-2pkdcg-osut-2pk0gw","label":"__hive_storagedesc.table","outVertex":"163873168","__timestamp":"1499685457722"},{"__modifiedBy":"admin","__state":"ACTIVE","__createdBy":"admin","inVertex":"163856480","__modificationTimestamp":"1499686956258","id":"2pmiia-2pkpzk-1a80l-2pk0gw","label":"__Process.outputs","outVertex":"163889552","__timestamp":"1499685462156"}],"count":4}

Let me know where i am going wrong.

avatar
New Contributor

@subash sharma This issue is still open UI changes for column level lineage ...Please refer (ATLAS-1181)

avatar
Expert Contributor

Thank you @Parag Redij