Support Questions
Find answers, ask questions, and share your expertise

Capture Spark Job Lineage in Apache Atlas

Capture Spark Job Lineage in Apache Atlas

New Contributor

Hi,

Can anyone suggest what is the best way to capture Spark Job Lineage in Apache Atlas? I am currently working with Spark2 (2.3.1) and Atlas (1.0.0).

Spline 0.4 has stopped supporting spark lineage capture into Atlas. Any help is highly appreciated.

 

Thanks

 

2 REPLIES 2

Re: Capture Spark Job Lineage in Apache Atlas

Contributor

I will recommend to check hortonworks project in github:

https://github.com/hortonworks-spark/spark-atlas-connector

 

Re: Capture Spark Job Lineage in Apache Atlas

Cloudera Employee

Hi Avjit, unfortunately, Atlas 1.0 does not support Spark lineage. In the Apache Atlas document, I found that no Atlas version supports Spark lineage.