How to create a New Entity in Apache Atlas 0.7 using Rest API ?

[#1]

To bring HDF lineage into Atlas. I would like to create a new entity on HDF, Can we achieve it in Atlas 0.7. If we can how to create a new entity on an existing HDF to bring the lineage.

thank you,

Subash

[#2]

Hello,

You could create a simpe hdfs_path entity :

curl -u admin:admin  -ik -H "Content-Type: application/json" -X POST -d '{"entity": {"typeName" : "hdfs_path", "attributes" : {"name" : "hdfs_file_test", "qualifiedName" : "testfile@hdpcluster", "path" : "/user/data/test.csv", "clusterName":"hdpcluster"}}}' http://YOUR_HOST:21000/api/atlas/v2/entity

[#3]

Hey @Smart Data, Ty can you please let me know how to create type and entity for Nifi Data flow

[#4]

I never did it but I could suggest :

Create a new simple type :

curl -u admin:admin  -ik -H "Content-Type: application/json" -X POST -d '{"enumDefs" : [], "structDefs" : [], "classificationDefs": [], "entityDefs" : [{"name": "nifi_process_test", "superTypes": ["DataSet"], "typeVersion": "1.0", "attributeDefs": [{"name": "processor_name", "typeName": "string", "cardinality": "SINGLE", "isIndexable": false, "isOptional": false, "isUnique": false } ] }] }' http://YOUR_HOST:21000/api/atlas/v2/types/typedefs

I know that Nifi could connect to KAFKA via a processor.

So, you just have to publish into Kafka Topic like this :

 1- Open Producer console:

$KAFKA_HOME/kafka-console-producer.sh --broker-list YOUR_HOST:6667 --topic ATLAS_HOOK

2- Send ENTITY_CREATE message :

{"version": {"version": "1.0.0"}, "message": {"entities": [{"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "id": {"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id": "-1467290565135246000", "version": 0, "typeName": "nifi_process_test", "state": "ACTIVE"}, "typeName": "nifi_process_test", "values": {"qualifiedName": "testNifi", "owner": "admin", "description": "Test NIFI", "name": "Test Nifi"}, "traitNames": [], "traits": {} }], "type": "ENTITY_CREATE", "user": "admin"} }

I hope it could help you.

[#5 · Accepted solution]

@subash sharma

As you are already aware, currently there is no out-of-box integration between HDF/Nifi and Atlas. This is a roadmap item and has bee n documented in the below:

https://issues.apache.org/jira/browse/NIFI-3709

However, you can use the Atlas REST API to create the HDF entities and lineage. Below are a couple of examples that show how this may be done:

https://community.hortonworks.com/repos/66014/nifi-atlas-bridge.html

https://community.hortonworks.com/repos/39432/nifi-atlas-lineage-reporter.html

As always, if you find this post helpful, don't forget to "accept" answer.

[#6]

Hey Eyad,

Ty. I will try to implement the same.