Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Will I get the Lineage for Apache Falcon in Atlas(Version 0.5)?

Solved Go to solution

Will I get the Lineage for Apache Falcon in Atlas(Version 0.5)?

Expert Contributor

Hello Everyone,

My Data pipeline is running perfectly in Apache Falcon.

Data Pipeline Problem Statement:

I am loading data from HDFS location to Hive table(Partitioned table).

Question:

I didn't seen lineage for this whole process.I mean,Atlas doesn't showed me any type of icon in lineage diagram.

Why Apache Atlas do not showing me lineage for falcon?

Is it supported in Atlas(Version 0.5)?

Thanks in Advance.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Will I get the Lineage for Apache Falcon in Atlas(Version 0.5)?

Hi @Manoj Dhake

I've answered a very similar question recently, the short answer is that integration is coming up very soon, but not available in the current 0.5 release of Atlas. Atlas 0.6 or later currently in preview for the next HDP release.

I would strongly recommend reading my answer to the following question here:

https://community.hortonworks.com/questions/32350/atlas-05-current-functionalities.html#answer-32404

... and if you have any further questions, don't hesitate to let me know.

Also check out another answer in that thread from Andrew Ahn who shows you how you can see the current preview sandbox with the latest functionality in it.

Hope that helps!

View solution in original post

3 REPLIES 3
Highlighted

Re: Will I get the Lineage for Apache Falcon in Atlas(Version 0.5)?

Hi @Manoj Dhake

I've answered a very similar question recently, the short answer is that integration is coming up very soon, but not available in the current 0.5 release of Atlas. Atlas 0.6 or later currently in preview for the next HDP release.

I would strongly recommend reading my answer to the following question here:

https://community.hortonworks.com/questions/32350/atlas-05-current-functionalities.html#answer-32404

... and if you have any further questions, don't hesitate to let me know.

Also check out another answer in that thread from Andrew Ahn who shows you how you can see the current preview sandbox with the latest functionality in it.

Hope that helps!

View solution in original post

Highlighted

Re: Will I get the Lineage for Apache Falcon in Atlas(Version 0.5)?

Expert Contributor

First of All Thank you Dave,

As per my knowledge,Atlas 0.7 provides this feature.But again One Question for you:

I have running Falcon data pipeline which INSERT data from one hive(table1) table to another Hive(table2)table.

The Script is:

INSERT OVERWRITE TABLE falconexample.table2 PARTITION (${falcon_output_partitions_hive}) SELECT id,firstName,designation,department FROM falconexample.table1 WHERE ${falcon_input_filter};

My Question:

I have checked in the Atlas UI and able to see lineage or metadata only for actions which are performed within hive such as(CREATE TABLE,INSERT OVERWRITE etc).

But Nether I found metadata for falcon nor it's lineage.

I know am behaving like fool but this is very important for me.

Again,I am asking you same question as earlier I have asked but it's necessary for me.

Please Help me.

Highlighted

Re: Will I get the Lineage for Apache Falcon in Atlas(Version 0.5)?

Rising Star
@Manoj Dhake

Look at the below link for Atlas 0.7

http://atlas.incubator.apache.org/Bridge-Falcon.html

Hope this helps!

Don't have an account?
Coming from Hortonworks? Activate your account here