Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

[#1]

I am trying to see the column level lineage in apache atlas 0.8. To get the lineage i have created a table by using CTAS statement over a subset of columns of a different table. As per this link, Ideally i should get the lineage.but in my Atlas, Lineage is not populating.

Thank you,

Subash

[#2]

@subash sharma

Is Hive-Hook enabled? Check that the hooks are enable in the Ranger config

[#3]

Hey @Geoffrey Shelton Okot,

Yes Hive Hook enabled, I can see the lineage of hive_table. Only the hive_column_lineage is not showing up.

[#4]

@subash sharma

Which version of HDP are you using? Even though Column Level Lineage is advertised as available with Atlas 0.8 in the Apache page, it has only been made GA with HDP 2.6.1 rather than HDP 2.6.0. The delay was to ensure it works with the other appropriate HiveQL commands beyond CTAS.

[#5]

I am using HDP 2.6.1

[#6]

@subash sharma, do you see the lineage graph rendered in UI when you select one of the hive column in CTAS table ?

Alternatively could you issue the below curl command on one of the column guid and see if you can get edges with label "__Process.outputs" or "__Process.inputs".

curl -X GET "http://localhost:21000/api/atlas/discovery/search/gremlin?query=g.V("__guid","<column_guid>").inE.toList()"

Hope this helps.

[#7]

@Sarath Subramanian, Yes i can see the "__Process.outputs" for the CTAS column. Attaching output below :

{"requestId":"pool-2-thread-4 - 4718810e-6302-48cb-9b75-d6d665242731","query":"g.V(\"__guid\",\"90b6074b-532e-4562-8b37-42a8636926ac\").inE.toList()","queryType":"gremlin","results":[{"__modifiedBy":"admin","__state":"ACTIVE","__createdBy":"admin","inVertex":"163856480","__modificationTimestamp":"1499686956258","id":"2pkg7r-2pk0jc-opp1-2pk0gw","label":"__hive_column.table","outVertex":"163856568","__timestamp":"1499685457722"},{"__modifiedBy":"admin","__state":"ACTIVE","__createdBy":"admin","inVertex":"163856480","__modificationTimestamp":"1499686956258","id":"2pkw1e-2pk0pc-opp1-2pk0gw","label":"__hive_column.table","outVertex":"163856784","__timestamp":"1499685457722"},{"__modifiedBy":"admin","__state":"ACTIVE","__createdBy":"admin","inVertex":"163856480","__modificationTimestamp":"1499686956258","id":"2pl8oi-2pkdcg-osut-2pk0gw","label":"__hive_storagedesc.table","outVertex":"163873168","__timestamp":"1499685457722"},{"__modifiedBy":"admin","__state":"ACTIVE","__createdBy":"admin","inVertex":"163856480","__modificationTimestamp":"1499686956258","id":"2pmiia-2pkpzk-1a80l-2pk0gw","label":"__Process.outputs","outVertex":"163889552","__timestamp":"1499685462156"}],"count":4}

Let me know where i am going wrong.

[#8 · Accepted solution]

@subash sharma This issue is still open UI changes for column level lineage ...Please refer (ATLAS-1181)

[#9]

Thank you @Parag Redij