Importing existing metadata and lineage in Atlas

[#1]

Hi, we are in process of creating our Lake environment over HDP and Atlas will not be enabled for phase1. We want to capture metadata and lineage in MySQL currently so it can be imported into Atlas once its available in cluster.

My question is,

1. How to know what attributes are required by Atlas and how to see data model Atlas uses?

2. How can we import data from MySQL (technical metadata + lineage) into Atlas? I know there are REST API available but so far i have not been able to find out good examples of using those.

@Dave Russell: seen many responses from you in this forum on Atlas. Hoping you are someone from your team and provide some insights on above questions.

[#2 · Accepted solution]

@Vivek Dhagat

1. The Atlas data model is located here:

http://atlas.incubator.apache.org/TypeSystem.html

See this thread for examples of how to create entities:

https://community.hortonworks.com/questions/41409/how-to-use-the-function-of-data-classification-of....

See this tread for examples of how to create traits (tags)

https://community.hortonworks.com/questions/33501/how-to-create-attribute-sets-and-collections-using...

2. Try the Cross Component Lineage lab outline here

http://hortonworks.com/hadoop-tutorial/cross-component-lineage-apache-atlas/

This will run a Sqoop job to import data from MySQL into Hive. That should create a series of meta data entities including the entity that represents the MySQL table, the Sqoop process, the Hive Table, and all of the lineage that was involved with the import process. This capability will be available in HDP 2.5.

[#3]

Vadim Vaks

Can you please help to provide the steps for creating the lineage for oracle DB in Atlas.