Atlas : Lineage only with REST API not with Kafka ?

[#1]

Hello,

I used the Kafka topic (ATLAS_HOOK) to create a new entity (for hdfs_path). It seems there is no operations to create a lineage between entities and processes (At the current version). Should I use Kafka to create only entities & REST API to create lineage ?

Any suggestion ?

Thanks.

[#2]

@Smart Data

I am assuming that you are trying to create entities and lineage for HDFS files. If so, then yes, you would need to use the REST API to create the lineage. You can use the API to create the entities themselves rather than going through Kafka.

If you're using HDP 2.6.1, you can also create your entities through the Atlas UI as per the link below.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_data-governance/content/ch_atlas_searchi...

Finally, below is a step-by-step example of creating entities and lineage for an HDFS file that is picked and processed by Spark and the results written back to HDFS. It will give you a good idea of how the APIs may be leveraged.

https://community.hortonworks.com/content/kbentry/91237/creating-custom-types-and-entities-in-atlas....

[#3 · Accepted solution]

Please look at:

https://community.hortonworks.com/questions/115749/messaging-api-with-kafka-lineage.html