Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Capture Spark Job Lineage in Apache Atlas

Capture Spark Job Lineage in Apache Atlas

New Contributor

Hi,

Can anyone suggest what is the best way to capture Spark Job Lineage in Apache Atlas? I am currently working with Spark2 (2.3.1) and Atlas (1.0.0).

Spline 0.4 has stopped supporting spark lineage capture into Atlas. Any help is highly appreciated.

 

Thanks

 

2 REPLIES 2
Highlighted

Re: Capture Spark Job Lineage in Apache Atlas

Contributor

I will recommend to check hortonworks project in github:

https://github.com/hortonworks-spark/spark-atlas-connector

 

Highlighted

Re: Capture Spark Job Lineage in Apache Atlas

Cloudera Employee

Hi Avjit, unfortunately, Atlas 1.0 does not support Spark lineage. In the Apache Atlas document, I found that no Atlas version supports Spark lineage.

Don't have an account?
Coming from Hortonworks? Activate your account here