question Re: Atlas : Lineage only with REST API not with Kafka ? in Archives of Support Questions (Read Only)

Atlas : Lineage only with REST API not with Kafka ?

smartdatabundle — Wed, 28 Jun 2017 15:40:13 GMT

Hello,

I used the Kafka topic (ATLAS_HOOK) to create a new entity (for hdfs_path). It seems there is no operations to create a lineage between entities and processes (At the current version). Should I use Kafka to create only entities & REST API to create lineage ?

Any suggestion ?

Thanks.

Re: Atlas : Lineage only with REST API not with Kafka ?

egarelnabi — Wed, 28 Jun 2017 20:10:19 GMT

@Smart Data

I am assuming that you are trying to create entities and lineage for HDFS files. If so, then yes, you would need to use the REST API to create the lineage. You can use the API to create the entities themselves rather than going through Kafka.

If you're using HDP 2.6.1, you can also create your entities through the Atlas UI as per the link below.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_data-governance/content/ch_atlas_searching_and_viewing_entities.html#atlas_manually_creating_entities

Finally, below is a step-by-step example of creating entities and lineage for an HDFS file that is picked and processed by Spark and the results written back to HDFS. It will give you a good idea of how the APIs may be leveraged.

https://community.hortonworks.com/content/kbentry/91237/creating-custom-types-and-entities-in-atlas.html

Re: Atlas : Lineage only with REST API not with Kafka ?

smartdatabundle — Fri, 28 Jul 2017 12:41:29 GMT

Please look at:

https://community.hortonworks.com/questions/115749/messaging-api-with-kafka-lineage.html