Apache Atlas Spark Data lineage

[#1]

Hi,

Am trying to implement data lineage for my spark application. I Have kafka topic, spark streaming read data from kafka and place in data source. when I checked apache atlas it does n't provide any hooks for spark. I guess we have to use rest api for this implementation. can someone point to some documentation or example for this?

[#2 · Accepted solution]

@vnandigam

You are correct, Atlas does not currently provide lineage for Spark. This is something engineering/community is working on.

You can, however, create your own entities and use the REST API to populate them. Here is some documentation and examples:

http://atlas.apache.org/0.7.0-incubating/AtlasTechnicalUserGuide.pdf

Please note that while this documentation also applies to Atlas 0.7-0.8 (in HDP 2.5-2.6), it does use APIs that have been deprecated in that version and will be removed n future ones. Still, it's good to get you started with your implementation.

As always, if you find any responses here useful, don't forget to "accept" an answer.

[#3]

@Eyad Garelnabi

Thanks for the answer. So I created metadata for my custom object in using rest api, then once I retrieved my event from spark streaming added as entity using rest api. So atlas will take care about lineage or do I need to add event modifications manually each and everytime?

[#4]

Take a look at the "Create Lineage amongst data sets" section (p. 46) in the document link I shared above. It also has a detailed example.

[#5]

yes. Got it @Eyad Garelnabi. Thanks

[#6]

Hi @vnandigam 

Good news. Now Spark Atlas integration is supported using CDP cluster.

References:

1. https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/atlas-reference/topics/atlas-spark-metadata-c...

2. https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade-hdp/topics/amb-enable-spark-cm.ht...