Capture Spark Job Lineage in Apache Atlas

[#1]

Hi,

Can anyone suggest what is the best way to capture Spark Job Lineage in Apache Atlas? I am currently working with Spark2 (2.3.1) and Atlas (1.0.0).

Spline 0.4 has stopped supporting spark lineage capture into Atlas. Any help is highly appreciated.

Thanks

[#2]

I will recommend to check hortonworks project in github:

https://github.com/hortonworks-spark/spark-atlas-connector

[#3]

Hi Avjit, unfortunately, Atlas 1.0 does not support Spark lineage. In the Apache Atlas document, I found that no Atlas version supports Spark lineage.