question Re: How can we get hive lineage data using REST API in Apache Atlas? in Archives of Support Questions (Read Only)

How can we get hive lineage data using REST API in Apache Atlas?

Manus — Wed, 08 Jun 2016 12:23:41 GMT

Hello all,

I am exploring on REST API for Apache Atlas and wants to fetch hive lineage data from Apache Atlas repository using REST API.I refer following link

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP2.3.0/bk_data_governance/content/section_atlas_restapi_hivelineageresource.html

http://atlas.incubator.apache.org/api/resource_HiveLineageResource.html

please provide me an example to get the lineage data using REST API.

Thanks in advance.

Re: How can we get hive lineage data using REST API in Apache Atlas?

vvaks — Mon, 13 Jun 2016 05:58:32 GMT

@Manoj Dhake

The following series of REST calls shows how to figure out the name of the entities classified as Tables. You can then call out to the Lineage resource URI for that table to get input/output graphs and schema.

curl -X GET http://localhost:21000/api/atlas/types
{"results":["Fact","Process","storm_topology_reference","event","View","ETL","DB","Dimension","nifi_flow","Infrastructure","JdbcAccess","StorageDesc","Column","DataSet","PII","Table","Metric","LoadProcess"],"count":18,"requestId":"qtp1320338790-13 - dcd2b32a-ad75-43e0-89ee-10b87bdbd1fd"}


curl -X GET http://localhost:21000/api/atlas/entities?type=Table
{"requestId":"qtp1320338790-79 - 4a8eb1d7-a694-4936-819e-07808f8383e2","typeName":"Table","results":["cd71b47d-2616-4494-a3de-ddd04bc569e8","4efdeae5-2a27-4b24-88bf-cf25c0c156d5","582650eb-72ca-4994-a1ec-3c833be0120e","f4e4a676-e19f-4d3b-bc07-af3ce166ea4e","34ac76b3-b38a-4a40-a48b-57a3270177ab","729d43dc-b8be-4cb5-9f9a-9c22de205c54"]

curl -X GET http://localhost:21000/api/atlas/entities/cd71b47d-2616-4494-a3de-ddd04bc569e8
{"requestId":"qtp1320338790-13 - e5f16b73-80c9-4fe9-bf5a-aadba2f2910e","GUID":"cd71b47d-2616-4494-a3de-ddd04bc569e8","definition":"{\n  \"jsonClass\":\"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference\",\n  \"id\":{\n    \"jsonClass\":\"org.apache.atlas.typesystem.json.InstanceSerialization$_Id\",\n    \"id\":\"cd71b47d-2616-4494-a3de-ddd04bc569e8\",\n    \"version\":0,\n    \"typeName\":\"Table\"\n  },\n  \"typeName\":\"Table\",\n  \"values\":{\n    \"tableType\":\"Managed\",\n    \"name\":\"sales_fact_monthly_mv\"

curl -X GET http://localhost:21000/api/atlas/lineage/hive/table/sales_fact_monthly_mv/inputs/graph
{"requestId":"qtp1320338790-84 - 7fd236b2-501b-4155-b94b-6e56b38c72c8","tableName":"sales_fact_monthly_mv","results":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct11","values":{"vertices":{"4efdeae5-2a27-4b24-88bf-cf25c0c156d5":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct10","values":{"vertexId":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__IdType","values":{"guid":"4efdeae5-2a27-4b24-88bf-cf25c0c156d5","typeName":"Table"}},"name":"sales_fact"}},"f4e4a676-e19f-4d3b-bc07-af3ce166ea4e":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct10","values":{"vertexId":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__IdType","values":{"guid":"f4e4a676-e19f-4d3b-bc07-af3ce166ea4e","typeName":"Table"}},"name":"time_dim"}},"729d43dc-b8be-4cb5-9f9a-9c22de205c54":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct10","values":{"vertexId":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__IdType","values":{"guid":"729d43dc-b8be-4cb5-9f9a-9c22de205c54","typeName":"Table"}},"name":"sales_fact_daily_mv"}},"cd71b47d-2616-4494-a3de-ddd04bc569e8":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct10","values":{"vertexId":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__IdType","values":{"guid":"cd71b47d-2616-4494-a3de-ddd04bc569e8","typeName":"Table"}},"name":"sales_fact_monthly_mv"}}},"edges":{"81419739-9952-4c3e-a7ab-b2f47ae7a362":["729d43dc-b8be-4cb5-9f9a-9c22de205c54"],"729d43dc-b8be-4cb5-9f9a-9c22de205c54":["43b242f7-39ed-45b4-8861-ba7c4c50ff0c"],"cd71b47d-2616-4494-a3de-ddd04bc569e8":["81419739-9952-4c3e-a7ab-b2f47ae7a362"],"43b242f7-39ed-45b4-8861-ba7c4c50ff0c":["4efdeae5-2a27-4b24-88bf-cf25c0c156d5","f4e4a676-e19f-4d3b-bc07-af3ce166ea4e"]}}}}

curl -X GET http://localhost:21000/api/atlas/lineage/hive/table/sales_fact_monthly_mv/schema
{"requestId":"qtp1320338790-79 - a316d9c1-638a-4d26-b297-0772cbdde1fd","tableName":"sales_fact_monthly_mv","results":{"query":"Table where (name = \"sales_fact_monthly_mv\") columns","dataType":{"superTypes":[],"hierarchicalMetaTypeName":"org.apache.atlas.typesystem.types.ClassType","typeName":"Column","attributeDefinitions":[{"name":"name","dataTypeName":"string","multiplicity":{"lower":0,"upper":1,"isUnique":false},"isComposite":false,"isUnique":false,"isIndexable":true,"reverseAttributeName":null},{"name":"dataType","dataTypeName":"string","multiplicity":{"lower":0,"upper":1,"isUnique":false},"isComposite":false,"isUnique":false,"isIndexable":true,"reverseAttributeName":null},{"name":"comment","dataTypeName":"string","multiplicity":{"lower":0,"upper":1,"isUnique":false},"isComposite":false,"isUnique":false,"isIndexable":true,"reverseAttributeName":null}]},"rows":[{"$typeName$":"Column","$id$":{"id":"77448e1e-ef27-4ee3-a35c-f0124b613dd8","$typeName$":"Column","version":0},"comment":"product id","name":"product_id","dataType":"int"},{"$typeName$":"Column","$id$":{"id":"3e647f08-a319-4ca2-8595-17b7734e6d15","$typeName$":"Column","version":0},"comment":"customer id","name":"customer_id","dataType":"int","$traits$":{"PII":{"$typeName$":"PII"}}},{"$typeName$":"Column","$id$":{"id":"eb23c523-d450-44d8-b87b-f9adf75512d8","$typeName$":"Column","version":0},"comment":"product id","name":"sales","dataType":"double","$traits$":{"Metric":{"$typeName$":"Metric"}}},{"$typeName$":"Column","$id$":{"id":"66f7eff9-73f1-4944-af16-03099b201b4d","$typeName$":"Column","version":0},"comment":"time id","name":"time_id","dataType":"int"}]}}

Re: How can we get hive lineage data using REST API in Apache Atlas?

Manus — Mon, 13 Jun 2016 15:31:14 GMT

Thank you Vadim,

The examples you have post are really helped me,but again could you please tell how to use REST API for

1) creating,deleting,modifying entity.

Re: How can we get hive lineage data using REST API in Apache Atlas?

vvaks — Sat, 18 Jun 2016 00:35:29 GMT

@Manoj Dhake

Would you mind asking that as a separate question? It will be easier for others to find. After you submit the question, reply to this thread @ me.

Re: How can we get hive lineage data using REST API in Apache Atlas?

RussHockey2 — Fri, 01 Jul 2016 00:07:27 GMT

The response provided gives me a good head start on how to utilize the REST API - thank you!!!

The problem i am still having is how do I process the result of the call to inputs/graph - speicifically - the JSON has the vertices and the edges.

{"results":{"typeName":"__tempQueryResultStruct401","values":{"edges":{"cb09a6f5-f76d-4208-8a62-17c3ca1e3e9d":["b0092981-6bcf-41a8-b0c9-34f3c5b6efd4"],"5591f85d-054a-4264-a8ee-c5ec250ccccb":["aff53ced-c48a-4b18-a321-4862a6ab84ac"],"6875ac4e-45a3-458a-b586-16e484567f9a":["cb09a6f5-f76d-4208-8a62-17c3ca1e3e9d"],"b0092981-6bcf-41a8-b0c9-34f3c5b6efd4":["5591f85d-054a-4264-a8ee-c5ec250ccccb"]},"vertices":{"aff53ced-c48a-4b18-a321-4862a6ab84ac":{"typeName":"__tempQueryResultStruct400","values":{"vertexId":{"typeName":"__IdType","values":{"guid":"aff53ced-c48a-4b18-a321-4862a6ab84ac","typeName":"sqoop_dbdatastore"},"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct"},"name":"mysql --url jdbc:mysql:\/\/localhost\/test?zeroDateTimeBehavior=convertToNull --table test_table_sqoop"},"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct"},"6875ac4e-45a3-458a-b586-16e484567f9a":{"typeName":"__tempQueryResultStruct400","values":{"vertexId":{"typeName":"__IdType","values":{"guid":"6875ac4e-45a3-458a-b586-16e484567f9a","typeName":"hive_table"},"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct"},"name":"default.russ@erietp"},"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct"},"b0092981-6bcf-41a8-b0c9-34f3c5b6efd4":{"typeName":"__tempQueryResultStruct400","values":{"vertexId":{"typeName":"__IdType","values":{"guid":"b0092981-6bcf-41a8-b0c9-34f3c5b6efd4","typeName":"hive_table"},"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct"},"name":"default.test_hive_table@erietp"},"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct"}}},"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct"},"requestId":"qtp1635546341-128 - c44ffa0a-4451-4661-846d-09a846a7bcfc","tableName":"default.russ@erietp"}

how do i process the verticies and the edges ? what order do i do this ?

This is not clear

Re: How can we get hive lineage data using REST API in Apache Atlas?

vvaks — Mon, 04 Jul 2016 10:51:56 GMT

@Russell Anderson

Could you ask your question as a new question and in more detail? I want to make sure that the question and answer is easily searchable.