question Re: Apache Atlas Rest API post and get command taking a long time to execute in Archives of Support Questions (Read Only)

Apache Atlas Rest API post and get command taking a long time to execute

subash_sharma — Wed, 08 Mar 2017 02:40:29 GMT

I am running a python script to associate an Attribute to a tag (one at a time because of GUID Constraint) and i have to tag around 3k attributes on daily basis. My script functions in below manner :

Run get command on a table and fetch GUID's of all columns available in the table, (i am using DSL Search here)

get_tables= requests.get(hostname +'/api/atlas/discovery/search/dsl?query=hive_table+where+name=%27'+tab+'%27and+__state=%27ACTIVE%27', headers={"Content-Type": "application/json","Accept": "application/json"}, auth=('username','password'))

Check whether the columns are tagged or not, If a column is not tagged, Fetch the GUID of that column and run a post command to associate it with a tag.

post_tag=requests.post((hostname+'/api/atlas/entities/'+guid+'/traits'), auth=(username ,password),json={ "typeName": "tag_name", "values": {}, "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Struct" },headers = {"Content-Type": "application/json","Accept": "application/json"})

Once the tag is associated, I am closing the HTTP connection.

get_tables.close()

post_tag.close()

Now each of the API post call is taking more than 30 seconds to execute (which is not good) :(. Can someone please let me know an efficient method to tag attributes.

Thank you in advance,

Subash

Re: Apache Atlas Rest API post and get command taking a long time to execute

VR46 — Thu, 09 Mar 2017 02:03:04 GMT

A2A @Ayub Khan and @sshivaprasad. Thanks guys.

Re: Apache Atlas Rest API post and get command taking a long time to execute

apathan — Wed, 15 Mar 2017 14:12:32 GMT

@subash sharma Do you see any debug logging in the application log while this POST commands are executed? could you please share the debug logs?

Also, are you executing this POST command from the same machine as the atlas is running? This might also be happening due to network delay. I would recommend to do such bulk operations from the same machine as Atlas to avoid any network delays.

Re: Apache Atlas Rest API post and get command taking a long time to execute

subash_sharma — Wed, 15 Mar 2017 14:33:52 GMT

Sure @Ayub Khan, We were able to resolve the issue by Restarting Apache Atlas (To close All Web Connections) and Multi-Processing to execute REST API command

Re: Apache Atlas Rest API post and get command taking a long time to execute

apathan — Wed, 15 Mar 2017 14:35:57 GMT

@subash sharma

Glad that the issue is resolved. Please close the loop by accepting the answer.