Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How can we get hive lineage data using REST API in Apache Atlas?

Solved Go to solution

How can we get hive lineage data using REST API in Apache Atlas?

Expert Contributor

Hello all,

I am exploring on REST API for Apache Atlas and wants to fetch hive lineage data from Apache Atlas repository using REST API.I refer following link

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP2.3.0/bk_data_governance/content/section_atlas_res...

http://atlas.incubator.apache.org/api/resource_HiveLineageResource.html

please provide me an example to get the lineage data using REST API.

Thanks in advance.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: How can we get hive lineage data using REST API in Apache Atlas?

Guru

@Manoj Dhake

The following series of REST calls shows how to figure out the name of the entities classified as Tables. You can then call out to the Lineage resource URI for that table to get input/output graphs and schema.

curl -X GET http://localhost:21000/api/atlas/types
{"results":["Fact","Process","storm_topology_reference","event","View","ETL","DB","Dimension","nifi_flow","Infrastructure","JdbcAccess","StorageDesc","Column","DataSet","PII","Table","Metric","LoadProcess"],"count":18,"requestId":"qtp1320338790-13 - dcd2b32a-ad75-43e0-89ee-10b87bdbd1fd"}


curl -X GET http://localhost:21000/api/atlas/entities?type=Table
{"requestId":"qtp1320338790-79 - 4a8eb1d7-a694-4936-819e-07808f8383e2","typeName":"Table","results":["cd71b47d-2616-4494-a3de-ddd04bc569e8","4efdeae5-2a27-4b24-88bf-cf25c0c156d5","582650eb-72ca-4994-a1ec-3c833be0120e","f4e4a676-e19f-4d3b-bc07-af3ce166ea4e","34ac76b3-b38a-4a40-a48b-57a3270177ab","729d43dc-b8be-4cb5-9f9a-9c22de205c54"]

curl -X GET http://localhost:21000/api/atlas/entities/cd71b47d-2616-4494-a3de-ddd04bc569e8
{"requestId":"qtp1320338790-13 - e5f16b73-80c9-4fe9-bf5a-aadba2f2910e","GUID":"cd71b47d-2616-4494-a3de-ddd04bc569e8","definition":"{\n  \"jsonClass\":\"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference\",\n  \"id\":{\n    \"jsonClass\":\"org.apache.atlas.typesystem.json.InstanceSerialization$_Id\",\n    \"id\":\"cd71b47d-2616-4494-a3de-ddd04bc569e8\",\n    \"version\":0,\n    \"typeName\":\"Table\"\n  },\n  \"typeName\":\"Table\",\n  \"values\":{\n    \"tableType\":\"Managed\",\n    \"name\":\"sales_fact_monthly_mv\"

curl -X GET http://localhost:21000/api/atlas/lineage/hive/table/sales_fact_monthly_mv/inputs/graph
{"requestId":"qtp1320338790-84 - 7fd236b2-501b-4155-b94b-6e56b38c72c8","tableName":"sales_fact_monthly_mv","results":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct11","values":{"vertices":{"4efdeae5-2a27-4b24-88bf-cf25c0c156d5":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct10","values":{"vertexId":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__IdType","values":{"guid":"4efdeae5-2a27-4b24-88bf-cf25c0c156d5","typeName":"Table"}},"name":"sales_fact"}},"f4e4a676-e19f-4d3b-bc07-af3ce166ea4e":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct10","values":{"vertexId":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__IdType","values":{"guid":"f4e4a676-e19f-4d3b-bc07-af3ce166ea4e","typeName":"Table"}},"name":"time_dim"}},"729d43dc-b8be-4cb5-9f9a-9c22de205c54":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct10","values":{"vertexId":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__IdType","values":{"guid":"729d43dc-b8be-4cb5-9f9a-9c22de205c54","typeName":"Table"}},"name":"sales_fact_daily_mv"}},"cd71b47d-2616-4494-a3de-ddd04bc569e8":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct10","values":{"vertexId":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__IdType","values":{"guid":"cd71b47d-2616-4494-a3de-ddd04bc569e8","typeName":"Table"}},"name":"sales_fact_monthly_mv"}}},"edges":{"81419739-9952-4c3e-a7ab-b2f47ae7a362":["729d43dc-b8be-4cb5-9f9a-9c22de205c54"],"729d43dc-b8be-4cb5-9f9a-9c22de205c54":["43b242f7-39ed-45b4-8861-ba7c4c50ff0c"],"cd71b47d-2616-4494-a3de-ddd04bc569e8":["81419739-9952-4c3e-a7ab-b2f47ae7a362"],"43b242f7-39ed-45b4-8861-ba7c4c50ff0c":["4efdeae5-2a27-4b24-88bf-cf25c0c156d5","f4e4a676-e19f-4d3b-bc07-af3ce166ea4e"]}}}}

curl -X GET http://localhost:21000/api/atlas/lineage/hive/table/sales_fact_monthly_mv/schema
{"requestId":"qtp1320338790-79 - a316d9c1-638a-4d26-b297-0772cbdde1fd","tableName":"sales_fact_monthly_mv","results":{"query":"Table where (name = \"sales_fact_monthly_mv\") columns","dataType":{"superTypes":[],"hierarchicalMetaTypeName":"org.apache.atlas.typesystem.types.ClassType","typeName":"Column","attributeDefinitions":[{"name":"name","dataTypeName":"string","multiplicity":{"lower":0,"upper":1,"isUnique":false},"isComposite":false,"isUnique":false,"isIndexable":true,"reverseAttributeName":null},{"name":"dataType","dataTypeName":"string","multiplicity":{"lower":0,"upper":1,"isUnique":false},"isComposite":false,"isUnique":false,"isIndexable":true,"reverseAttributeName":null},{"name":"comment","dataTypeName":"string","multiplicity":{"lower":0,"upper":1,"isUnique":false},"isComposite":false,"isUnique":false,"isIndexable":true,"reverseAttributeName":null}]},"rows":[{"$typeName$":"Column","$id$":{"id":"77448e1e-ef27-4ee3-a35c-f0124b613dd8","$typeName$":"Column","version":0},"comment":"product id","name":"product_id","dataType":"int"},{"$typeName$":"Column","$id$":{"id":"3e647f08-a319-4ca2-8595-17b7734e6d15","$typeName$":"Column","version":0},"comment":"customer id","name":"customer_id","dataType":"int","$traits$":{"PII":{"$typeName$":"PII"}}},{"$typeName$":"Column","$id$":{"id":"eb23c523-d450-44d8-b87b-f9adf75512d8","$typeName$":"Column","version":0},"comment":"product id","name":"sales","dataType":"double","$traits$":{"Metric":{"$typeName$":"Metric"}}},{"$typeName$":"Column","$id$":{"id":"66f7eff9-73f1-4944-af16-03099b201b4d","$typeName$":"Column","version":0},"comment":"time id","name":"time_id","dataType":"int"}]}}

View solution in original post

5 REPLIES 5
Highlighted

Re: How can we get hive lineage data using REST API in Apache Atlas?

Guru

@Manoj Dhake

The following series of REST calls shows how to figure out the name of the entities classified as Tables. You can then call out to the Lineage resource URI for that table to get input/output graphs and schema.

curl -X GET http://localhost:21000/api/atlas/types
{"results":["Fact","Process","storm_topology_reference","event","View","ETL","DB","Dimension","nifi_flow","Infrastructure","JdbcAccess","StorageDesc","Column","DataSet","PII","Table","Metric","LoadProcess"],"count":18,"requestId":"qtp1320338790-13 - dcd2b32a-ad75-43e0-89ee-10b87bdbd1fd"}


curl -X GET http://localhost:21000/api/atlas/entities?type=Table
{"requestId":"qtp1320338790-79 - 4a8eb1d7-a694-4936-819e-07808f8383e2","typeName":"Table","results":["cd71b47d-2616-4494-a3de-ddd04bc569e8","4efdeae5-2a27-4b24-88bf-cf25c0c156d5","582650eb-72ca-4994-a1ec-3c833be0120e","f4e4a676-e19f-4d3b-bc07-af3ce166ea4e","34ac76b3-b38a-4a40-a48b-57a3270177ab","729d43dc-b8be-4cb5-9f9a-9c22de205c54"]

curl -X GET http://localhost:21000/api/atlas/entities/cd71b47d-2616-4494-a3de-ddd04bc569e8
{"requestId":"qtp1320338790-13 - e5f16b73-80c9-4fe9-bf5a-aadba2f2910e","GUID":"cd71b47d-2616-4494-a3de-ddd04bc569e8","definition":"{\n  \"jsonClass\":\"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference\",\n  \"id\":{\n    \"jsonClass\":\"org.apache.atlas.typesystem.json.InstanceSerialization$_Id\",\n    \"id\":\"cd71b47d-2616-4494-a3de-ddd04bc569e8\",\n    \"version\":0,\n    \"typeName\":\"Table\"\n  },\n  \"typeName\":\"Table\",\n  \"values\":{\n    \"tableType\":\"Managed\",\n    \"name\":\"sales_fact_monthly_mv\"

curl -X GET http://localhost:21000/api/atlas/lineage/hive/table/sales_fact_monthly_mv/inputs/graph
{"requestId":"qtp1320338790-84 - 7fd236b2-501b-4155-b94b-6e56b38c72c8","tableName":"sales_fact_monthly_mv","results":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct11","values":{"vertices":{"4efdeae5-2a27-4b24-88bf-cf25c0c156d5":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct10","values":{"vertexId":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__IdType","values":{"guid":"4efdeae5-2a27-4b24-88bf-cf25c0c156d5","typeName":"Table"}},"name":"sales_fact"}},"f4e4a676-e19f-4d3b-bc07-af3ce166ea4e":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct10","values":{"vertexId":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__IdType","values":{"guid":"f4e4a676-e19f-4d3b-bc07-af3ce166ea4e","typeName":"Table"}},"name":"time_dim"}},"729d43dc-b8be-4cb5-9f9a-9c22de205c54":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct10","values":{"vertexId":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__IdType","values":{"guid":"729d43dc-b8be-4cb5-9f9a-9c22de205c54","typeName":"Table"}},"name":"sales_fact_daily_mv"}},"cd71b47d-2616-4494-a3de-ddd04bc569e8":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct10","values":{"vertexId":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__IdType","values":{"guid":"cd71b47d-2616-4494-a3de-ddd04bc569e8","typeName":"Table"}},"name":"sales_fact_monthly_mv"}}},"edges":{"81419739-9952-4c3e-a7ab-b2f47ae7a362":["729d43dc-b8be-4cb5-9f9a-9c22de205c54"],"729d43dc-b8be-4cb5-9f9a-9c22de205c54":["43b242f7-39ed-45b4-8861-ba7c4c50ff0c"],"cd71b47d-2616-4494-a3de-ddd04bc569e8":["81419739-9952-4c3e-a7ab-b2f47ae7a362"],"43b242f7-39ed-45b4-8861-ba7c4c50ff0c":["4efdeae5-2a27-4b24-88bf-cf25c0c156d5","f4e4a676-e19f-4d3b-bc07-af3ce166ea4e"]}}}}

curl -X GET http://localhost:21000/api/atlas/lineage/hive/table/sales_fact_monthly_mv/schema
{"requestId":"qtp1320338790-79 - a316d9c1-638a-4d26-b297-0772cbdde1fd","tableName":"sales_fact_monthly_mv","results":{"query":"Table where (name = \"sales_fact_monthly_mv\") columns","dataType":{"superTypes":[],"hierarchicalMetaTypeName":"org.apache.atlas.typesystem.types.ClassType","typeName":"Column","attributeDefinitions":[{"name":"name","dataTypeName":"string","multiplicity":{"lower":0,"upper":1,"isUnique":false},"isComposite":false,"isUnique":false,"isIndexable":true,"reverseAttributeName":null},{"name":"dataType","dataTypeName":"string","multiplicity":{"lower":0,"upper":1,"isUnique":false},"isComposite":false,"isUnique":false,"isIndexable":true,"reverseAttributeName":null},{"name":"comment","dataTypeName":"string","multiplicity":{"lower":0,"upper":1,"isUnique":false},"isComposite":false,"isUnique":false,"isIndexable":true,"reverseAttributeName":null}]},"rows":[{"$typeName$":"Column","$id$":{"id":"77448e1e-ef27-4ee3-a35c-f0124b613dd8","$typeName$":"Column","version":0},"comment":"product id","name":"product_id","dataType":"int"},{"$typeName$":"Column","$id$":{"id":"3e647f08-a319-4ca2-8595-17b7734e6d15","$typeName$":"Column","version":0},"comment":"customer id","name":"customer_id","dataType":"int","$traits$":{"PII":{"$typeName$":"PII"}}},{"$typeName$":"Column","$id$":{"id":"eb23c523-d450-44d8-b87b-f9adf75512d8","$typeName$":"Column","version":0},"comment":"product id","name":"sales","dataType":"double","$traits$":{"Metric":{"$typeName$":"Metric"}}},{"$typeName$":"Column","$id$":{"id":"66f7eff9-73f1-4944-af16-03099b201b4d","$typeName$":"Column","version":0},"comment":"time id","name":"time_id","dataType":"int"}]}}

View solution in original post

Highlighted

Re: How can we get hive lineage data using REST API in Apache Atlas?

Expert Contributor

Thank you Vadim,

The examples you have post are really helped me,but again could you please tell how to use REST API for

1) creating,deleting,modifying entity.

Highlighted

Re: How can we get hive lineage data using REST API in Apache Atlas?

Guru

@Manoj Dhake

Would you mind asking that as a separate question? It will be easier for others to find. After you submit the question, reply to this thread @ me.

Highlighted

Re: How can we get hive lineage data using REST API in Apache Atlas?

Explorer

The response provided gives me a good head start on how to utilize the REST API - thank you!!!

The problem i am still having is how do I process the result of the call to inputs/graph - speicifically - the JSON has the vertices and the edges.

  • {"results":{"typeName":"__tempQueryResultStruct401","values":{"edges":{"cb09a6f5-f76d-4208-8a62-17c3ca1e3e9d":["b0092981-6bcf-41a8-b0c9-34f3c5b6efd4"],"5591f85d-054a-4264-a8ee-c5ec250ccccb":["aff53ced-c48a-4b18-a321-4862a6ab84ac"],"6875ac4e-45a3-458a-b586-16e484567f9a":["cb09a6f5-f76d-4208-8a62-17c3ca1e3e9d"],"b0092981-6bcf-41a8-b0c9-34f3c5b6efd4":["5591f85d-054a-4264-a8ee-c5ec250ccccb"]},"vertices":{"aff53ced-c48a-4b18-a321-4862a6ab84ac":{"typeName":"__tempQueryResultStruct400","values":{"vertexId":{"typeName":"__IdType","values":{"guid":"aff53ced-c48a-4b18-a321-4862a6ab84ac","typeName":"sqoop_dbdatastore"},"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct"},"name":"mysql --url jdbc:mysql:\/\/localhost\/test?zeroDateTimeBehavior=convertToNull --table test_table_sqoop"},"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct"},"6875ac4e-45a3-458a-b586-16e484567f9a":{"typeName":"__tempQueryResultStruct400","values":{"vertexId":{"typeName":"__IdType","values":{"guid":"6875ac4e-45a3-458a-b586-16e484567f9a","typeName":"hive_table"},"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct"},"name":"default.russ@erietp"},"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct"},"b0092981-6bcf-41a8-b0c9-34f3c5b6efd4":{"typeName":"__tempQueryResultStruct400","values":{"vertexId":{"typeName":"__IdType","values":{"guid":"b0092981-6bcf-41a8-b0c9-34f3c5b6efd4","typeName":"hive_table"},"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct"},"name":"default.test_hive_table@erietp"},"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct"}}},"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct"},"requestId":"qtp1635546341-128 - c44ffa0a-4451-4661-846d-09a846a7bcfc","tableName":"default.russ@erietp"}
  • how do i process the verticies and the edges ? what order do i do this ?
  • This is not clear
  • Highlighted

    Re: How can we get hive lineage data using REST API in Apache Atlas?

    Guru

    @Russell Anderson

    Could you ask your question as a new question and in more detail? I want to make sure that the question and answer is easily searchable.

    Don't have an account?
    Coming from Hortonworks? Activate your account here