question Re: Apache Atlas Spark Data lineage in Support Questions

Apache Atlas Spark Data lineage

vnandigam — Mon, 22 May 2017 17:00:18 GMT

Hi,

Am trying to implement data lineage for my spark application. I Have kafka topic, spark streaming read data from kafka and place in data source. when I checked apache atlas it does n't provide any hooks for spark. I guess we have to use rest api for this implementation. can someone point to some documentation or example for this?

Re: Apache Atlas Spark Data lineage

egarelnabi — Mon, 22 May 2017 20:38:56 GMT

@vnandigam

You are correct, Atlas does not currently provide lineage for Spark. This is something engineering/community is working on.

You can, however, create your own entities and use the REST API to populate them. Here is some documentation and examples:

http://atlas.apache.org/0.7.0-incubating/AtlasTechnicalUserGuide.pdf

Please note that while this documentation also applies to Atlas 0.7-0.8 (in HDP 2.5-2.6), it does use APIs that have been deprecated in that version and will be removed n future ones. Still, it's good to get you started with your implementation.

As always, if you find any responses here useful, don't forget to "accept" an answer.

Re: Apache Atlas Spark Data lineage

vnandigam — Tue, 23 May 2017 12:16:35 GMT

@Eyad Garelnabi

Thanks for the answer. So I created metadata for my custom object in using rest api, then once I retrieved my event from spark streaming added as entity using rest api. So atlas will take care about lineage or do I need to add event modifications manually each and everytime?

Re: Apache Atlas Spark Data lineage

egarelnabi — Tue, 23 May 2017 21:03:41 GMT

Take a look at the "Create Lineage amongst data sets" section (p. 46) in the document link I shared above. It also has a detailed example.

Re: Apache Atlas Spark Data lineage

vnandigam — Wed, 24 May 2017 12:55:23 GMT

yes. Got it @Eyad Garelnabi. Thanks

Re: Apache Atlas Spark Data lineage

RangaReddy — Fri, 06 Aug 2021 04:17:01 GMT

Hi @vnandigam

Good news. Now Spark Atlas integration is supported using CDP cluster.

References:

1. https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/atlas-reference/topics/atlas-spark-metadata-collection.html

2. https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade-hdp/topics/amb-enable-spark-cm.html