Auto tagging Atlas REST API

[#1]

Hi,

Can anyone post a working example of adding a trait to an entity with the REST API?

So suppose I want to attach the tag 'CONF' to GUID 46c471f1-66ba-453c-9816-08ff144efde4 what is the syntax to do so?

Tags / Traits also confuse me:

If I understand it correctly Atlas traits are what tag based policies in Ranger are build upon.

Way down in the Json response below it shows that what I added as a 'tag' in the Atlas WebUI is actually a 'trait_name' here. But the thing 'tag' is also there, but apparently another thing altogether. So what is the standing of a trait vs. a tag in Atlas?

curl -u admin:admin http://sandbox.hortonworks.com:21000/api/atlas/v1/entities/46c471f1-66ba-453c-9816-08ff144efde4

gives me:

[
    {
        "href": "http://sandbox.hortonworks.com:21000/api/atlas/v1/entities/46c471f1-66ba-453c-9816-08ff144efde4",
        "name": "ssn",
        "id": "46c471f1-66ba-453c-9816-08ff144efde4",
        "type": "hive_column",
        "creation_time": "2016-10-12:21:06:11",
        "entityText": "hive_column  type string table hive_table  createTime Wed Oct 12 21:06:09 UTC 2016 lastAccessTime Wed Oct 12 21:06:09 UTC 2016 retention 0 parameters  numFiles 0 COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"} transient_lastDdlTime 1476306369 numRows 0 totalSize 0 rawDataSize 0 tableType MANAGED_TABLE temporary false qualifiedName default.employee@Sandbox name employee owner admin qualifiedName default.employee.ssn@Sandbox name ssn owner admin",
        "modified_time": "2016-10-17:19:26:21",
        "owner": "admin",
        "qualifiedName": "default.employee.ssn@Sandbox",
        "state": "ACTIVE",
        "version": 0,
        "super_types": [
            "Asset",
            "DataSet",
            "Referenceable"
        ],
        "table": [
            {
                "href": "http://sandbox.hortonworks.com:21000/api/atlas/v1/entities/170a5db7-7b32-4888-a361-448be4daa958",
                "name": "employee",
                "id": "170a5db7-7b32-4888-a361-448be4daa958",
                "type": "hive_table"
            }
        ],
        "tags": [],
        "trait_names": [
            "PERS"
        ],
        "traits": []
    }
]

[#2]

@Jasper

See https://community.hortonworks.com/articles/124/atlas-api-tips-create-trait-type-example.html

Hope it helps.

[#3]

@Greg Keys

That article does not cover attaching traits to GUIDs, only creating new types. Good reading though!

[#4 · Accepted solution]

@Jasper

Below is an example for associating a trait to an entity. Basically it a HTTP POST request with body containing the trait information.

curl -v 'http://172.22.73.216:21000/api/atlas/entities/f9432735-5972-4525-b3fd-17544babd5ee/traits' -H 'Cookie: JSESSIONID=159cepww1tnzaczw2f4wbhhdu' -H 'Origin: http://172.22.73.216:21000' -H 'Accept-Encoding: gzip, deflate' -H 'X-XSRF-HEADER: ""' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36' -H 'Content-Type: application/json' -H 'Accept-Language: en-US,en;q=0.8' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: http://172.22.73.216:21000/index.html' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' --data-binary '{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"addMultipleTraits-3-mho2upf7jh","values":{}}' --compressed



*   Trying 172.22.73.216...
* Connected to 172.22.73.216 (172.22.73.216) port 21000 (#0)
> POST /api/atlas/entities/f9432735-5972-4525-b3fd-17544babd5ee/traits HTTP/1.1
> Host: 172.22.73.216:21000
> Cookie: JSESSIONID=159cepww1tnzaczw2f4wbhhdu
> Origin: http://172.22.73.216:21000
> Accept-Encoding: gzip, deflate
> X-XSRF-HEADER: ""
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36
> Content-Type: application/json
> Accept-Language: en-US,en;q=0.8
> Accept: application/json, text/javascript, */*; q=0.01
> Referer: http://172.22.73.216:21000/index.html
> X-Requested-With: XMLHttpRequest
> Connection: keep-alive
> Content-Length: 134
>
* upload completely sent off: 134 out of 134 bytes
< HTTP/1.1 201 Created
< Date: Tue, 18 Oct 2016 03:47:09 GMT
< Location: http://172.22.73.216:21000/api/atlas/entities/f9432735-5972-4525-b3fd-17544babd5ee/traits/f9432735-5...
< Content-Type: application/json; charset=UTF-8
< Transfer-Encoding: chunked
< Server: Jetty(9.2.12.v20150709)
<
* Connection #0 to host 172.22.73.216 left intact
{"requestId":"qtp297811323-15 - 302927ef-c011-4c22-8cee-352dd4b18c2d"}

In the above response, you see "201 created", this confirms that the trait association is successful. To verify if the trait is attached to the entity, make GET API request for that entity. For example:

Request looks like this:

curl 'http://172.22.73.216:21000/api/atlas/entities/f9432735-5972-4525-b3fd-17544babd5ee' -H 'Cookie: JSESSIONID=159cepww1tnzaczw2f4wbhhdu' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: http://172.22.73.216:21000/index.html' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' --compressed | python -m json.tool

Response should contain the trait you associated:

{
    "definition": {
        "id": {
            "id": "f9432735-5972-4525-b3fd-17544babd5ee",
            "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
            "state": "ACTIVE",
            "typeName": "storm_topology",
            "version": 0
        },
        "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
        "traitNames": [
            "addMultipleTraits-1-mho2upf7jh",
            "addMultipleTraits-10-mho2upf7jh",
            "addMultipleTraits-3-mho2upf7jh"
        ],
        "traits": {
            "addMultipleTraits-1-mho2upf7jh": {
                "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Struct",
                "typeName": "addMultipleTraits-1-mho2upf7jh",
                "values": {}
            },
            "addMultipleTraits-10-mho2upf7jh": {
                "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Struct",
                "typeName": "addMultipleTraits-10-mho2upf7jh",
                "values": {}
            },
            "addMultipleTraits-3-mho2upf7jh": {
                "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Struct",
                "typeName": "addMultipleTraits-3-mho2upf7jh",
                "values": {}
            }
        },
        "typeName": "storm_topology",
        "values": {
            "clusterName": "cl1",
            "conf": null,
            "description": null,
            "endTime": null,
            "id": "kafka_mongo1-2-1476723908",
            "inputs": [
                {
                    "id": "dc50e010-af69-490a-b7b6-8f98bba776da",
                    "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
                    "state": "ACTIVE",
                    "typeName": "DataSet",
                    "version": 0
                }
            ],
            "name": "kafka_mongo1",
            "nodes": [
                {
                    "id": {
                        "id": "12086a26-89b9-4aba-bd66-e44067c0396b",
                        "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
                        "state": "ACTIVE",
                        "typeName": "storm_bolt",
                        "version": 0
                    },
                    "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
                    "traitNames": [],
                    "traits": {},
                    "typeName": "storm_bolt",
                    "values": {
                        "conf": {
                            "HiveBolt.currentBatchSize": "0",
                            "HiveBolt.kerberosEnabled": "false",
                            "HiveBolt.options.autoCreatePartitions": "true",
                            "HiveBolt.options.batchSize": "1",
                            "HiveBolt.options.callTimeout": "10000",
                            "HiveBolt.options.databaseName": "default",
                            "HiveBolt.options.heartBeatInterval": "240",
                            "HiveBolt.options.idleTimeout": "3600",
                            "HiveBolt.options.mapper.columnFields._fields": "id,msg",
                            "HiveBolt.options.mapper.columnFields._index.id": "0",
                            "HiveBolt.options.mapper.columnFields._index.msg": "1",
                            "HiveBolt.options.mapper.fieldDelimiter": ",",
                            "HiveBolt.options.mapper.partitionFields._fields": "dt",
                            "HiveBolt.options.mapper.partitionFields._index.dt": "0",
                            "HiveBolt.options.maxOpenConnections": "100",
                            "HiveBolt.options.metaStoreURI": "thrift://atlas-testing-unsecure-4.openstacklocal:9083",
                            "HiveBolt.options.tableName": "test",
                            "HiveBolt.options.txnsPerBatch": "2",
                            "HiveBolt.timeToSendHeartBeat.value": "0"
                        },
                        "description": null,
                        "driverClass": "org.apache.storm.hive.bolt.HiveBolt",
                        "inputs": [
                            "kafkaspout_test"
                        ],
                        "name": "hivebolt_test",
                        "outputs": null
                    }
                },
                {
                    "id": {
                        "id": "42d963e7-64aa-451e-848f-9b8d87886a69",
                        "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
                        "state": "ACTIVE",
                        "typeName": "storm_spout",
                        "version": 0
                    },
                    "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
                    "traitNames": [],
                    "traits": {},
                    "typeName": "storm_spout",
                    "values": {
                        "conf": {
                            "KafkaSpout._currPartitionIndex": "0",
                            "KafkaSpout._lastUpdateMs": "0",
                            "KafkaSpout._spoutConfig.bufferSizeBytes": "1048576",
                            "KafkaSpout._spoutConfig.fetchMaxWait": "10000",
                            "KafkaSpout._spoutConfig.fetchSizeBytes": "1048576",
                            "KafkaSpout._spoutConfig.forceFromStart": "true",
                            "KafkaSpout._spoutConfig.hosts.brokerZkPath": "/brokers",
                            "KafkaSpout._spoutConfig.hosts.brokerZkStr": "atlas-testing-unsecure-2.openstacklocal:2181,atlas-testing-unsecure-4.openstacklocal:2181,atlas-testing-unsecure-1.openstacklocal:2181,atlas-testing-unsecure-5.openstacklocal:2181",
                            "KafkaSpout._spoutConfig.hosts.refreshFreqSecs": "60",
                            "KafkaSpout._spoutConfig.id": "a3fd0b73-b2d4-4fd9-a97f-909ce6a9b336",
                            "KafkaSpout._spoutConfig.maxOffsetBehind": "9223372036854775807",
                            "KafkaSpout._spoutConfig.metricsTimeBucketSizeInSecs": "60",
                            "KafkaSpout._spoutConfig.socketTimeoutMs": "10000",
                            "KafkaSpout._spoutConfig.startOffsetTime": "-2",
                            "KafkaSpout._spoutConfig.stateUpdateIntervalMs": "2000",
                            "KafkaSpout._spoutConfig.topic": "test_topic",
                            "KafkaSpout._spoutConfig.useStartOffsetTimeIfOffsetOutOfRange": "true",
                            "KafkaSpout._spoutConfig.zkRoot": "/test_topic",
                            "KafkaSpout._uuid": "1c08a736-c039-497c-8160-673fdda7e129"
                        },
                        "description": null,
                        "driverClass": "storm.kafka.KafkaSpout",
                        "name": "kafkaspout_test",
                        "outputs": [
                            "hivebolt_test"
                        ]
                    }
                }
            ],
            "outputs": [
                {
                    "id": "e5d69223-f2aa-44b2-a5a9-c3f3743e6a63",
                    "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
                    "state": "ACTIVE",
                    "typeName": "DataSet",
                    "version": 0
                }
            ],
            "owner": "storm",
            "qualifiedName": "kafka_mongo1",
            "startTime": "2016-10-17T17:05:07.365Z"
        }
    },
    "requestId": "qtp297811323-15 - fd83329c-fd4f-4889-b050-5e02c405a886"
}

For more information on these APIs, please refer to these links..

http://atlas.incubator.apache.org/AtlasTechnicalUserGuide.pdf

http://atlas.incubator.apache.org/api/resource_EntityResource.html#path__entities_-guid-_traits.html

[#5]

@Jasper

Did you check if this works for you?

[#6]

@Ayub Khan

Does not work:

curl -u admin:admin -v -X POST -H 'Content-Type: application/json' -H 'Accept-Language: en-US,en;q=0.8' -H 'Accept: application/json, text/javascript, */*; q=0.01' --data-binary '{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_ Struct", "typeName":"CONF", "values":{ } }' --compressed "http://sandbox.hortonworks.com:21000/api/atlas/entities/74dc66ee-1bdb-43da-8f6d-12580877a700/traits/"

Gives:

{"error":"Do not know how to deserialize 'org.apache.atlas.typesystem.json.InstanceSerialization$_ Struct'","stackTrace":"org.json4s.package$MappingException: Do not know how to deserialize 'org.apache.atlas.typesystem.json.InstanceSerialization$_ Struct'\n\tat org.json4s.Extraction$ClassInstanceBuilder.org$json4s$Extraction$ClassInstanceBuilder$$mkWithTypeHint(Extraction.scala:506)

[#7]

@Jasper

I guess there is a typo in the body of the request, you have an extra space before Struct, please modify it and try. Let us know if it works after this change.

"org.apache.atlas.typesystem.json.InstanceSerialization$_ Struct"

[#8]

@Ayub Khan It works now. Thanks for pointing that out.