question Re: Will I get the Lineage for Apache Falcon in Atlas(Version 0.5)? in Archives of Support Questions (Read Only)

Will I get the Lineage for Apache Falcon in Atlas(Version 0.5)?

Manus — Mon, 23 May 2016 15:51:35 GMT

Hello Everyone,

My Data pipeline is running perfectly in Apache Falcon.

Data Pipeline Problem Statement:

I am loading data from HDFS location to Hive table(Partitioned table).

Question:

I didn't seen lineage for this whole process.I mean,Atlas doesn't showed me any type of icon in lineage diagram.

Why Apache Atlas do not showing me lineage for falcon?

Is it supported in Atlas(Version 0.5)?

Thanks in Advance.

Re: Will I get the Lineage for Apache Falcon in Atlas(Version 0.5)?

drussell — Mon, 23 May 2016 18:42:59 GMT

Hi @Manoj Dhake

I've answered a very similar question recently, the short answer is that integration is coming up very soon, but not available in the current 0.5 release of Atlas. Atlas 0.6 or later currently in preview for the next HDP release.

I would strongly recommend reading my answer to the following question here:

https://community.hortonworks.com/questions/32350/atlas-05-current-functionalities.html#answer-32404

... and if you have any further questions, don't hesitate to let me know.

Also check out another answer in that thread from Andrew Ahn who shows you how you can see the current preview sandbox with the latest functionality in it.

Hope that helps!

Re: Will I get the Lineage for Apache Falcon in Atlas(Version 0.5)?

hrongali — Mon, 23 May 2016 21:33:56 GMT

@Manoj Dhake

Look at the below link for Atlas 0.7

http://atlas.incubator.apache.org/Bridge-Falcon.html

Hope this helps!

Re: Will I get the Lineage for Apache Falcon in Atlas(Version 0.5)?

Manus — Tue, 24 May 2016 15:46:10 GMT

First of All Thank you Dave,

As per my knowledge,Atlas 0.7 provides this feature.But again One Question for you:

I have running Falcon data pipeline which INSERT data from one hive(table1) table to another Hive(table2)table.

The Script is:

INSERT OVERWRITE TABLE falconexample.table2 PARTITION (${falcon_output_partitions_hive}) SELECT id,firstName,designation,department FROM falconexample.table1 WHERE ${falcon_input_filter};

My Question:

I have checked in the Atlas UI and able to see lineage or metadata only for actions which are performed within hive such as(CREATE TABLE,INSERT OVERWRITE etc).

But Nether I found metadata for falcon nor it's lineage.

I know am behaving like fool but this is very important for me.

Again,I am asking you same question as earlier I have asked but it's necessary for me.

Please Help me.