How can we get hive lineage data using REST API in Apache Atlas?

[#1]

Hello all,

I am exploring on REST API for Apache Atlas and wants to fetch hive lineage data from Apache Atlas repository using REST API.I refer following link

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP2.3.0/bk_data_governance/content/section_atlas_res...

http://atlas.incubator.apache.org/api/resource_HiveLineageResource.html

please provide me an example to get the lineage data using REST API.

Thanks in advance.

[#2 · Accepted solution]

@Manoj Dhake

The following series of REST calls shows how to figure out the name of the entities classified as Tables. You can then call out to the Lineage resource URI for that table to get input/output graphs and schema.

curl -X GET http://localhost:21000/api/atlas/types
{"results":["Fact","Process","storm_topology_reference","event","View","ETL","DB","Dimension","nifi_flow","Infrastructure","JdbcAccess","StorageDesc","Column","DataSet","PII","Table","Metric","LoadProcess"],"count":18,"requestId":"qtp1320338790-13 - dcd2b32a-ad75-43e0-89ee-10b87bdbd1fd"}


curl -X GET http://localhost:21000/api/atlas/entities?type=Table
{"requestId":"qtp1320338790-79 - 4a8eb1d7-a694-4936-819e-07808f8383e2","typeName":"Table","results":["cd71b47d-2616-4494-a3de-ddd04bc569e8","4efdeae5-2a27-4b24-88bf-cf25c0c156d5","582650eb-72ca-4994-a1ec-3c833be0120e","f4e4a676-e19f-4d3b-bc07-af3ce166ea4e","34ac76b3-b38a-4a40-a48b-57a3270177ab","729d43dc-b8be-4cb5-9f9a-9c22de205c54"]

curl -X GET http://localhost:21000/api/atlas/entities/cd71b47d-2616-4494-a3de-ddd04bc569e8
{"requestId":"qtp1320338790-13 - e5f16b73-80c9-4fe9-bf5a-aadba2f2910e","GUID":"cd71b47d-2616-4494-a3de-ddd04bc569e8","definition":"{\n  \"jsonClass\":\"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference\",\n  \"id\":{\n    \"jsonClass\":\"org.apache.atlas.typesystem.json.InstanceSerialization$_Id\",\n    \"id\":\"cd71b47d-2616-4494-a3de-ddd04bc569e8\",\n    \"version\":0,\n    \"typeName\":\"Table\"\n  },\n  \"typeName\":\"Table\",\n  \"values\":{\n    \"tableType\":\"Managed\",\n    \"name\":\"sales_fact_monthly_mv\"

curl -X GET http://localhost:21000/api/atlas/lineage/hive/table/sales_fact_monthly_mv/inputs/graph
{"requestId":"qtp1320338790-84 - 7fd236b2-501b-4155-b94b-6e56b38c72c8","tableName":"sales_fact_monthly_mv","results":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct11","values":{"vertices":{"4efdeae5-2a27-4b24-88bf-cf25c0c156d5":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct10","values":{"vertexId":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__IdType","values":{"guid":"4efdeae5-2a27-4b24-88bf-cf25c0c156d5","typeName":"Table"}},"name":"sales_fact"}},"f4e4a676-e19f-4d3b-bc07-af3ce166ea4e":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct10","values":{"vertexId":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__IdType","values":{"guid":"f4e4a676-e19f-4d3b-bc07-af3ce166ea4e","typeName":"Table"}},"name":"time_dim"}},"729d43dc-b8be-4cb5-9f9a-9c22de205c54":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct10","values":{"vertexId":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__IdType","values":{"guid":"729d43dc-b8be-4cb5-9f9a-9c22de205c54","typeName":"Table"}},"name":"sales_fact_daily_mv"}},"cd71b47d-2616-4494-a3de-ddd04bc569e8":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct10","values":{"vertexId":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__IdType","values":{"guid":"cd71b47d-2616-4494-a3de-ddd04bc569e8","typeName":"Table"}},"name":"sales_fact_monthly_mv"}}},"edges":{"81419739-9952-4c3e-a7ab-b2f47ae7a362":["729d43dc-b8be-4cb5-9f9a-9c22de205c54"],"729d43dc-b8be-4cb5-9f9a-9c22de205c54":["43b242f7-39ed-45b4-8861-ba7c4c50ff0c"],"cd71b47d-2616-4494-a3de-ddd04bc569e8":["81419739-9952-4c3e-a7ab-b2f47ae7a362"],"43b242f7-39ed-45b4-8861-ba7c4c50ff0c":["4efdeae5-2a27-4b24-88bf-cf25c0c156d5","f4e4a676-e19f-4d3b-bc07-af3ce166ea4e"]}}}}

curl -X GET http://localhost:21000/api/atlas/lineage/hive/table/sales_fact_monthly_mv/schema
{"requestId":"qtp1320338790-79 - a316d9c1-638a-4d26-b297-0772cbdde1fd","tableName":"sales_fact_monthly_mv","results":{"query":"Table where (name = \"sales_fact_monthly_mv\") columns","dataType":{"superTypes":[],"hierarchicalMetaTypeName":"org.apache.atlas.typesystem.types.ClassType","typeName":"Column","attributeDefinitions":[{"name":"name","dataTypeName":"string","multiplicity":{"lower":0,"upper":1,"isUnique":false},"isComposite":false,"isUnique":false,"isIndexable":true,"reverseAttributeName":null},{"name":"dataType","dataTypeName":"string","multiplicity":{"lower":0,"upper":1,"isUnique":false},"isComposite":false,"isUnique":false,"isIndexable":true,"reverseAttributeName":null},{"name":"comment","dataTypeName":"string","multiplicity":{"lower":0,"upper":1,"isUnique":false},"isComposite":false,"isUnique":false,"isIndexable":true,"reverseAttributeName":null}]},"rows":[{"$typeName$":"Column","$id$":{"id":"77448e1e-ef27-4ee3-a35c-f0124b613dd8","$typeName$":"Column","version":0},"comment":"product id","name":"product_id","dataType":"int"},{"$typeName$":"Column","$id$":{"id":"3e647f08-a319-4ca2-8595-17b7734e6d15","$typeName$":"Column","version":0},"comment":"customer id","name":"customer_id","dataType":"int","$traits$":{"PII":{"$typeName$":"PII"}}},{"$typeName$":"Column","$id$":{"id":"eb23c523-d450-44d8-b87b-f9adf75512d8","$typeName$":"Column","version":0},"comment":"product id","name":"sales","dataType":"double","$traits$":{"Metric":{"$typeName$":"Metric"}}},{"$typeName$":"Column","$id$":{"id":"66f7eff9-73f1-4944-af16-03099b201b4d","$typeName$":"Column","version":0},"comment":"time id","name":"time_id","dataType":"int"}]}}

[#3]

Thank you Vadim,

The examples you have post are really helped me,but again could you please tell how to use REST API for

1) creating,deleting,modifying entity.

[#4]

@Manoj Dhake

Would you mind asking that as a separate question? It will be easier for others to find. After you submit the question, reply to this thread @ me.

[#5]

The response provided gives me a good head start on how to utilize the REST API - thank you!!!

The problem i am still having is how do I process the result of the call to inputs/graph - speicifically - the JSON has the vertices and the edges.

{"results":{"typeName":"__tempQueryResultStruct401","values":{"edges":{"cb09a6f5-f76d-4208-8a62-17c3ca1e3e9d":["b0092981-6bcf-41a8-b0c9-34f3c5b6efd4"],"5591f85d-054a-4264-a8ee-c5ec250ccccb":["aff53ced-c48a-4b18-a321-4862a6ab84ac"],"6875ac4e-45a3-458a-b586-16e484567f9a":["cb09a6f5-f76d-4208-8a62-17c3ca1e3e9d"],"b0092981-6bcf-41a8-b0c9-34f3c5b6efd4":["5591f85d-054a-4264-a8ee-c5ec250ccccb"]},"vertices":{"aff53ced-c48a-4b18-a321-4862a6ab84ac":{"typeName":"__tempQueryResultStruct400","values":{"vertexId":{"typeName":"__IdType","values":{"guid":"aff53ced-c48a-4b18-a321-4862a6ab84ac","typeName":"sqoop_dbdatastore"},"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct"},"name":"mysql --url jdbc:mysql:\/\/localhost\/test?zeroDateTimeBehavior=convertToNull --table test_table_sqoop"},"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct"},"6875ac4e-45a3-458a-b586-16e484567f9a":{"typeName":"__tempQueryResultStruct400","values":{"vertexId":{"typeName":"__IdType","values":{"guid":"6875ac4e-45a3-458a-b586-16e484567f9a","typeName":"hive_table"},"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct"},"name":"default.russ@erietp"},"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct"},"b0092981-6bcf-41a8-b0c9-34f3c5b6efd4":{"typeName":"__tempQueryResultStruct400","values":{"vertexId":{"typeName":"__IdType","values":{"guid":"b0092981-6bcf-41a8-b0c9-34f3c5b6efd4","typeName":"hive_table"},"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct"},"name":"default.test_hive_table@erietp"},"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct"}}},"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct"},"requestId":"qtp1635546341-128 - c44ffa0a-4451-4661-846d-09a846a7bcfc","tableName":"default.russ@erietp"}

how do i process the verticies and the edges ? what order do i do this ?

This is not clear

[#6]

@Russell Anderson

Could you ask your question as a new question and in more detail? I want to make sure that the question and answer is easily searchable.