question Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement in Archives of Support Questions (Read Only)

Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

subash_sharma — Mon, 10 Jul 2017 18:28:55 GMT

I am trying to see the column level lineage in apache atlas 0.8. To get the lineage i have created a table by using CTAS statement over a subset of columns of a different table. As per this link, Ideally i should get the lineage.but in my Atlas, Lineage is not populating.

Thank you,

Subash

Re: Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

Shelton — Mon, 10 Jul 2017 19:34:54 GMT

@subash sharma

Is Hive-Hook enabled? Check that the hooks are enable in the Ranger config

Re: Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

subash_sharma — Mon, 10 Jul 2017 19:51:58 GMT

Hey @Geoffrey Shelton Okot,

Yes Hive Hook enabled, I can see the lineage of hive_table. Only the hive_column_lineage is not showing up.

Re: Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

egarelnabi — Mon, 10 Jul 2017 21:39:05 GMT

@subash sharma

Which version of HDP are you using? Even though Column Level Lineage is advertised as available with Atlas 0.8 in the Apache page, it has only been made GA with HDP 2.6.1 rather than HDP 2.6.0. The delay was to ensure it works with the other appropriate HiveQL commands beyond CTAS.

Re: Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

ssubramanian — Tue, 11 Jul 2017 15:10:13 GMT

@subash sharma, do you see the lineage graph rendered in UI when you select one of the hive column in CTAS table ?

Alternatively could you issue the below curl command on one of the column guid and see if you can get edges with label "__Process.outputs" or "__Process.inputs".

curl -X GET "http://localhost:21000/api/atlas/discovery/search/gremlin?query=g.V("__guid","<column_guid>").inE.toList()"

Hope this helps.

Re: Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

subash_sharma — Tue, 11 Jul 2017 16:58:45 GMT

@Sarath Subramanian, Yes i can see the "__Process.outputs" for the CTAS column. Attaching output below :

{"requestId":"pool-2-thread-4 - 4718810e-6302-48cb-9b75-d6d665242731","query":"g.V(\"__guid\",\"90b6074b-532e-4562-8b37-42a8636926ac\").inE.toList()","queryType":"gremlin","results":[{"__modifiedBy":"admin","__state":"ACTIVE","__createdBy":"admin","inVertex":"163856480","__modificationTimestamp":"1499686956258","id":"2pkg7r-2pk0jc-opp1-2pk0gw","label":"__hive_column.table","outVertex":"163856568","__timestamp":"1499685457722"},{"__modifiedBy":"admin","__state":"ACTIVE","__createdBy":"admin","inVertex":"163856480","__modificationTimestamp":"1499686956258","id":"2pkw1e-2pk0pc-opp1-2pk0gw","label":"__hive_column.table","outVertex":"163856784","__timestamp":"1499685457722"},{"__modifiedBy":"admin","__state":"ACTIVE","__createdBy":"admin","inVertex":"163856480","__modificationTimestamp":"1499686956258","id":"2pl8oi-2pkdcg-osut-2pk0gw","label":"__hive_storagedesc.table","outVertex":"163873168","__timestamp":"1499685457722"},{"__modifiedBy":"admin","__state":"ACTIVE","__createdBy":"admin","inVertex":"163856480","__modificationTimestamp":"1499686956258","id":"2pmiia-2pkpzk-1a80l-2pk0gw","label":"__Process.outputs","outVertex":"163889552","__timestamp":"1499685462156"}],"count":4}

Let me know where i am going wrong.

Re: Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

subash_sharma — Tue, 11 Jul 2017 17:01:12 GMT

I am using HDP 2.6.1

Re: Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

parag_redij — Tue, 14 Nov 2017 21:41:09 GMT

@subash sharma This issue is still open UI changes for column level lineage ...Please refer (ATLAS-1181)

Re: Column level lineage is not reflecting in Atlas 0.8 after creating a table using CTAS statement

subash_sharma — Tue, 14 Nov 2017 22:07:09 GMT

Thank you @Parag Redij