Before completing this tutorial, it is important to understand data lineage.
What is Data Lineage
Data lineage is defined as a data life cycle that conveys data origin and where data moves over time. In Apache Hive, if I create a table (TableA) and then insert data (from another table TableB), the data lineage will display TableA as the target and Table B as the source/origin. These two tables are linked together by a process "insert into Table..", allowing a user to understand the data life cycle. In a Hadoop ecosystem, Apache Atlas contains the data lineage for various systems like Apache Hive, Apache Falcon and Apache Sqoop.
What is Apache Atlas
Apache Atlas is a centralized governance framework that supports the Hadoop ecosystem as a metastore repository. To add metadata to Atlas, libraries called ‘hooks’ are enabled in various systems which automatically capture metadata events in the respective systems and propagate those events to Atlas. (More on Atlas' Architecture).
Prerequisites
Download Atlas-Ranger preview VM here
Once the Atlas-Ranger VM is running, you can login through an SSH shell with user = root, password = hadoop
Atlas UI: http://localhost:21000 (use: Data Lineage), user = admin, password = admin
Ambari UI: http://localhost:8080 (use: Hive View), user = admin, password = admin
Step 1 - Login to Ambari and access Hive View
Step 2 - Create table brancha (database = default)
create table brancha(full_name string, ssn string, location string);
Step 3 - Create table branchb (database = default)
create table branchb(full_name string, ssn string, location string);
Step 4 - Insert data into both tables
insert into brancha(full_name,ssn,location) values ('ryan', '111-222-333', 'chicago');
insert into brancha(full_name,ssn,location) values ('brad', '444-555-666', 'minneapolis');
insert into brancha(full_name,ssn,location) values ('rupert', '000-000-000', 'chicago');
insert into brancha(full_name,ssn,location) values ('john', '555-111-555', 'boston');
insert into branchb(full_name,ssn,location) values ('jane', '666-777-888', 'dallas');
insert into branchb(full_name,ssn,location) values ('andrew', '999-999-999', 'tampa');
insert into branchb(full_name,ssn,location) values ('ryan', '111-222-333', 'chicago');
insert into branchb(full_name,ssn,location) values ('brad', '444-555-666', 'minneapolis');
(Using Atlas-Ranger preview - execute one insert statement at a time)
Step 5 - In a web browser, access Atlas UI at http://localhost:21000 and search for default.brancha
Step 6 - In the Atlas UI, select the hyperlink under the column name "default.brancha@abc"
Step 7 - In the Atlas UI, there should be no lineage for brancha
Step 8 - Create table branch_intersect (database = default) as a join of brancha and branchb where the ssn is equal
create table branch_intersect as select b1.full_name,b1.ssn,b1.location from brancha b1 inner join branchb b2 ON b1.ssn = b2.ssn;
Step 9 - In the Atlas UI, refresh the browser from Step 7
(orange = current table) You can see source brancha had a process of “create table br...” populating the target branch_intersect table
Step 10 - In the Atlas UI, search for default.branch_intersect
... View more
