Will I get the Lineage for Apache Falcon in Atlas(Version 0.5)?

[#1]

Hello Everyone,

My Data pipeline is running perfectly in Apache Falcon.

Data Pipeline Problem Statement:

I am loading data from HDFS location to Hive table(Partitioned table).

Question:

I didn't seen lineage for this whole process.I mean,Atlas doesn't showed me any type of icon in lineage diagram.

Why Apache Atlas do not showing me lineage for falcon?

Is it supported in Atlas(Version 0.5)?

Thanks in Advance.

[#2 · Accepted solution]

Hi @Manoj Dhake

I've answered a very similar question recently, the short answer is that integration is coming up very soon, but not available in the current 0.5 release of Atlas. Atlas 0.6 or later currently in preview for the next HDP release.

I would strongly recommend reading my answer to the following question here:

https://community.hortonworks.com/questions/32350/atlas-05-current-functionalities.html#answer-32404

... and if you have any further questions, don't hesitate to let me know.

Also check out another answer in that thread from Andrew Ahn who shows you how you can see the current preview sandbox with the latest functionality in it.

Hope that helps!

[#3]

First of All Thank you Dave,

As per my knowledge,Atlas 0.7 provides this feature.But again One Question for you:

I have running Falcon data pipeline which INSERT data from one hive(table1) table to another Hive(table2)table.

The Script is:

INSERT OVERWRITE TABLE falconexample.table2 PARTITION (${falcon_output_partitions_hive}) SELECT id,firstName,designation,department FROM falconexample.table1 WHERE ${falcon_input_filter};

My Question:

I have checked in the Atlas UI and able to see lineage or metadata only for actions which are performed within hive such as(CREATE TABLE,INSERT OVERWRITE etc).

But Nether I found metadata for falcon nor it's lineage.

I know am behaving like fool but this is very important for me.

Again,I am asking you same question as earlier I have asked but it's necessary for me.

Please Help me.

[#4]

@Manoj Dhake

Look at the below link for Atlas 0.7

http://atlas.incubator.apache.org/Bridge-Falcon.html

Hope this helps!