question Re: Auto tagging Atlas REST API in Archives of Support Questions (Read Only)

Auto tagging Atlas REST API

jknulst — Tue, 18 Oct 2016 02:33:21 GMT

Hi,

Can anyone post a working example of adding a trait to an entity with the REST API?

So suppose I want to attach the tag 'CONF' to GUID 46c471f1-66ba-453c-9816-08ff144efde4 what is the syntax to do so?

Tags / Traits also confuse me:

If I understand it correctly Atlas traits are what tag based policies in Ranger are build upon.

Way down in the Json response below it shows that what I added as a 'tag' in the Atlas WebUI is actually a 'trait_name' here. But the thing 'tag' is also there, but apparently another thing altogether. So what is the standing of a trait vs. a tag in Atlas?

curl -u admin:admin http://sandbox.hortonworks.com:21000/api/atlas/v1/entities/46c471f1-66ba-453c-9816-08ff144efde4

gives me:

[
    {
        "href": "http://sandbox.hortonworks.com:21000/api/atlas/v1/entities/46c471f1-66ba-453c-9816-08ff144efde4",
        "name": "ssn",
        "id": "46c471f1-66ba-453c-9816-08ff144efde4",
        "type": "hive_column",
        "creation_time": "2016-10-12:21:06:11",
        "entityText": "hive_column  type string table hive_table  createTime Wed Oct 12 21:06:09 UTC 2016 lastAccessTime Wed Oct 12 21:06:09 UTC 2016 retention 0 parameters  numFiles 0 COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"} transient_lastDdlTime 1476306369 numRows 0 totalSize 0 rawDataSize 0 tableType MANAGED_TABLE temporary false qualifiedName default.employee@Sandbox name employee owner admin qualifiedName default.employee.ssn@Sandbox name ssn owner admin",
        "modified_time": "2016-10-17:19:26:21",
        "owner": "admin",
        "qualifiedName": "default.employee.ssn@Sandbox",
        "state": "ACTIVE",
        "version": 0,
        "super_types": [
            "Asset",
            "DataSet",
            "Referenceable"
        ],
        "table": [
            {
                "href": "http://sandbox.hortonworks.com:21000/api/atlas/v1/entities/170a5db7-7b32-4888-a361-448be4daa958",
                "name": "employee",
                "id": "170a5db7-7b32-4888-a361-448be4daa958",
                "type": "hive_table"
            }
        ],
        "tags": [],
        "trait_names": [
            "PERS"
        ],
        "traits": []
    }
]

Re: Auto tagging Atlas REST API

gkeys — Tue, 18 Oct 2016 02:39:20 GMT

@Jasper

See https://community.hortonworks.com/articles/124/atlas-api-tips-create-trait-type-example.html

Hope it helps.

Re: Auto tagging Atlas REST API

jknulst — Tue, 18 Oct 2016 02:48:42 GMT

@Greg Keys

That article does not cover attaching traits to GUIDs, only creating new types. Good reading though!

Re: Auto tagging Atlas REST API

apathan — Tue, 18 Oct 2016 10:56:11 GMT

@Jasper

Below is an example for associating a trait to an entity. Basically it a HTTP POST request with body containing the trait information.

curl -v 'http://172.22.73.216:21000/api/atlas/entities/f9432735-5972-4525-b3fd-17544babd5ee/traits' -H 'Cookie: JSESSIONID=159cepww1tnzaczw2f4wbhhdu' -H 'Origin: http://172.22.73.216:21000' -H 'Accept-Encoding: gzip, deflate' -H 'X-XSRF-HEADER: ""' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36' -H 'Content-Type: application/json' -H 'Accept-Language: en-US,en;q=0.8' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: http://172.22.73.216:21000/index.html' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' --data-binary '{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"addMultipleTraits-3-mho2upf7jh","values":{}}' --compressed



*   Trying 172.22.73.216...
* Connected to 172.22.73.216 (172.22.73.216) port 21000 (#0)
> POST /api/atlas/entities/f9432735-5972-4525-b3fd-17544babd5ee/traits HTTP/1.1
> Host: 172.22.73.216:21000
> Cookie: JSESSIONID=159cepww1tnzaczw2f4wbhhdu
> Origin: http://172.22.73.216:21000
> Accept-Encoding: gzip, deflate
> X-XSRF-HEADER: ""
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36
> Content-Type: application/json
> Accept-Language: en-US,en;q=0.8
> Accept: application/json, text/javascript, */*; q=0.01
> Referer: http://172.22.73.216:21000/index.html
> X-Requested-With: XMLHttpRequest
> Connection: keep-alive
> Content-Length: 134
>
* upload completely sent off: 134 out of 134 bytes
< HTTP/1.1 201 Created
< Date: Tue, 18 Oct 2016 03:47:09 GMT
< Location: http://172.22.73.216:21000/api/atlas/entities/f9432735-5972-4525-b3fd-17544babd5ee/traits/f9432735-5972-4525-b3fd-17544babd5ee
< Content-Type: application/json; charset=UTF-8
< Transfer-Encoding: chunked
< Server: Jetty(9.2.12.v20150709)
<
* Connection #0 to host 172.22.73.216 left intact
{"requestId":"qtp297811323-15 - 302927ef-c011-4c22-8cee-352dd4b18c2d"}

In the above response, you see "201 created", this confirms that the trait association is successful. To verify if the trait is attached to the entity, make GET API request for that entity. For example:

Request looks like this:

curl 'http://172.22.73.216:21000/api/atlas/entities/f9432735-5972-4525-b3fd-17544babd5ee' -H 'Cookie: JSESSIONID=159cepww1tnzaczw2f4wbhhdu' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: http://172.22.73.216:21000/index.html' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' --compressed | python -m json.tool

Response should contain the trait you associated:

{
    "definition": {
        "id": {
            "id": "f9432735-5972-4525-b3fd-17544babd5ee",
            "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
            "state": "ACTIVE",
            "typeName": "storm_topology",
            "version": 0
        },
        "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
        "traitNames": [
            "addMultipleTraits-1-mho2upf7jh",
            "addMultipleTraits-10-mho2upf7jh",
            "addMultipleTraits-3-mho2upf7jh"
        ],
        "traits": {
            "addMultipleTraits-1-mho2upf7jh": {
                "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Struct",
                "typeName": "addMultipleTraits-1-mho2upf7jh",
                "values": {}
            },
            "addMultipleTraits-10-mho2upf7jh": {
                "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Struct",
                "typeName": "addMultipleTraits-10-mho2upf7jh",
                "values": {}
            },
            "addMultipleTraits-3-mho2upf7jh": {
                "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Struct",
                "typeName": "addMultipleTraits-3-mho2upf7jh",
                "values": {}
            }
        },
        "typeName": "storm_topology",
        "values": {
            "clusterName": "cl1",
            "conf": null,
            "description": null,
            "endTime": null,
            "id": "kafka_mongo1-2-1476723908",
            "inputs": [
                {
                    "id": "dc50e010-af69-490a-b7b6-8f98bba776da",
                    "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
                    "state": "ACTIVE",
                    "typeName": "DataSet",
                    "version": 0
                }
            ],
            "name": "kafka_mongo1",
            "nodes": [
                {
                    "id": {
                        "id": "12086a26-89b9-4aba-bd66-e44067c0396b",
                        "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
                        "state": "ACTIVE",
                        "typeName": "storm_bolt",
                        "version": 0
                    },
                    "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
                    "traitNames": [],
                    "traits": {},
                    "typeName": "storm_bolt",
                    "values": {
                        "conf": {
                            "HiveBolt.currentBatchSize": "0",
                            "HiveBolt.kerberosEnabled": "false",
                            "HiveBolt.options.autoCreatePartitions": "true",
                            "HiveBolt.options.batchSize": "1",
                            "HiveBolt.options.callTimeout": "10000",
                            "HiveBolt.options.databaseName": "default",
                            "HiveBolt.options.heartBeatInterval": "240",
                            "HiveBolt.options.idleTimeout": "3600",
                            "HiveBolt.options.mapper.columnFields._fields": "id,msg",
                            "HiveBolt.options.mapper.columnFields._index.id": "0",
                            "HiveBolt.options.mapper.columnFields._index.msg": "1",
                            "HiveBolt.options.mapper.fieldDelimiter": ",",
                            "HiveBolt.options.mapper.partitionFields._fields": "dt",
                            "HiveBolt.options.mapper.partitionFields._index.dt": "0",
                            "HiveBolt.options.maxOpenConnections": "100",
                            "HiveBolt.options.metaStoreURI": "thrift://atlas-testing-unsecure-4.openstacklocal:9083",
                            "HiveBolt.options.tableName": "test",
                            "HiveBolt.options.txnsPerBatch": "2",
                            "HiveBolt.timeToSendHeartBeat.value": "0"
                        },
                        "description": null,
                        "driverClass": "org.apache.storm.hive.bolt.HiveBolt",
                        "inputs": [
                            "kafkaspout_test"
                        ],
                        "name": "hivebolt_test",
                        "outputs": null
                    }
                },
                {
                    "id": {
                        "id": "42d963e7-64aa-451e-848f-9b8d87886a69",
                        "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
                        "state": "ACTIVE",
                        "typeName": "storm_spout",
                        "version": 0
                    },
                    "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
                    "traitNames": [],
                    "traits": {},
                    "typeName": "storm_spout",
                    "values": {
                        "conf": {
                            "KafkaSpout._currPartitionIndex": "0",
                            "KafkaSpout._lastUpdateMs": "0",
                            "KafkaSpout._spoutConfig.bufferSizeBytes": "1048576",
                            "KafkaSpout._spoutConfig.fetchMaxWait": "10000",
                            "KafkaSpout._spoutConfig.fetchSizeBytes": "1048576",
                            "KafkaSpout._spoutConfig.forceFromStart": "true",
                            "KafkaSpout._spoutConfig.hosts.brokerZkPath": "/brokers",
                            "KafkaSpout._spoutConfig.hosts.brokerZkStr": "atlas-testing-unsecure-2.openstacklocal:2181,atlas-testing-unsecure-4.openstacklocal:2181,atlas-testing-unsecure-1.openstacklocal:2181,atlas-testing-unsecure-5.openstacklocal:2181",
                            "KafkaSpout._spoutConfig.hosts.refreshFreqSecs": "60",
                            "KafkaSpout._spoutConfig.id": "a3fd0b73-b2d4-4fd9-a97f-909ce6a9b336",
                            "KafkaSpout._spoutConfig.maxOffsetBehind": "9223372036854775807",
                            "KafkaSpout._spoutConfig.metricsTimeBucketSizeInSecs": "60",
                            "KafkaSpout._spoutConfig.socketTimeoutMs": "10000",
                            "KafkaSpout._spoutConfig.startOffsetTime": "-2",
                            "KafkaSpout._spoutConfig.stateUpdateIntervalMs": "2000",
                            "KafkaSpout._spoutConfig.topic": "test_topic",
                            "KafkaSpout._spoutConfig.useStartOffsetTimeIfOffsetOutOfRange": "true",
                            "KafkaSpout._spoutConfig.zkRoot": "/test_topic",
                            "KafkaSpout._uuid": "1c08a736-c039-497c-8160-673fdda7e129"
                        },
                        "description": null,
                        "driverClass": "storm.kafka.KafkaSpout",
                        "name": "kafkaspout_test",
                        "outputs": [
                            "hivebolt_test"
                        ]
                    }
                }
            ],
            "outputs": [
                {
                    "id": "e5d69223-f2aa-44b2-a5a9-c3f3743e6a63",
                    "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
                    "state": "ACTIVE",
                    "typeName": "DataSet",
                    "version": 0
                }
            ],
            "owner": "storm",
            "qualifiedName": "kafka_mongo1",
            "startTime": "2016-10-17T17:05:07.365Z"
        }
    },
    "requestId": "qtp297811323-15 - fd83329c-fd4f-4889-b050-5e02c405a886"
}

For more information on these APIs, please refer to these links..

http://atlas.incubator.apache.org/AtlasTechnicalUserGuide.pdf

http://atlas.incubator.apache.org/api/resource_EntityResource.html#path__entities_-guid-_traits.html

Re: Auto tagging Atlas REST API

apathan — Wed, 19 Oct 2016 00:38:16 GMT

@Jasper

Did you check if this works for you?

Re: Auto tagging Atlas REST API

jknulst — Wed, 19 Oct 2016 04:05:27 GMT

@Ayub Khan

Does not work:

curl -u admin:admin -v -X POST -H 'Content-Type: application/json' -H 'Accept-Language: en-US,en;q=0.8' -H 'Accept: application/json, text/javascript, */*; q=0.01' --data-binary '{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_ Struct", "typeName":"CONF", "values":{ } }' --compressed "http://sandbox.hortonworks.com:21000/api/atlas/entities/74dc66ee-1bdb-43da-8f6d-12580877a700/traits/"

Gives:

{"error":"Do not know how to deserialize 'org.apache.atlas.typesystem.json.InstanceSerialization$_ Struct'","stackTrace":"org.json4s.package$MappingException: Do not know how to deserialize 'org.apache.atlas.typesystem.json.InstanceSerialization$_ Struct'\n\tat org.json4s.Extraction$ClassInstanceBuilder.org$json4s$Extraction$ClassInstanceBuilder$$mkWithTypeHint(Extraction.scala:506)

Re: Auto tagging Atlas REST API

apathan — Thu, 20 Oct 2016 22:35:12 GMT

@Jasper

I guess there is a typo in the body of the request, you have an extra space before Struct, please modify it and try. Let us know if it works after this change.

"org.apache.atlas.typesystem.json.InstanceSerialization$_ Struct"

Re: Auto tagging Atlas REST API

jknulst — Fri, 21 Oct 2016 03:34:20 GMT

@Ayub Khan It works now. Thanks for pointing that out.