Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Apache Atlas Rest API post and get command taking a long time to execute

avatar
Expert Contributor

I am running a python script to associate an Attribute to a tag (one at a time because of GUID Constraint) and i have to tag around 3k attributes on daily basis. My script functions in below manner :

Run get command on a table and fetch GUID's of all columns available in the table, (i am using DSL Search here)

get_tables= requests.get(hostname +'/api/atlas/discovery/search/dsl?query=hive_table+where+name=%27'+tab+'%27and+__state=%27ACTIVE%27', headers={"Content-Type": "application/json","Accept": "application/json"}, auth=('username','password'))

Check whether the columns are tagged or not, If a column is not tagged, Fetch the GUID of that column and run a post command to associate it with a tag.

post_tag=requests.post((hostname+'/api/atlas/entities/'+guid+'/traits'), auth=(username ,password),json={ "typeName": "tag_name", "values": {}, "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Struct" },headers = {"Content-Type": "application/json","Accept": "application/json"})

Once the tag is associated, I am closing the HTTP connection.

get_tables.close()

post_tag.close()

Now each of the API post call is taking more than 30 seconds to execute (which is not good) :(. Can someone please let me know an efficient method to tag attributes.

Thank you in advance,

Subash

2 ACCEPTED SOLUTIONS

avatar

@subash sharma Do you see any debug logging in the application log while this POST commands are executed? could you please share the debug logs?

Also, are you executing this POST command from the same machine as the atlas is running? This might also be happening due to network delay. I would recommend to do such bulk operations from the same machine as Atlas to avoid any network delays.

View solution in original post

avatar
Expert Contributor

Sure @Ayub Khan, We were able to resolve the issue by Restarting Apache Atlas (To close All Web Connections) and Multi-Processing to execute REST API command

View solution in original post

4 REPLIES 4

avatar
Guru

A2A @Ayub Khan and @sshivaprasad. Thanks guys.

avatar

@subash sharma Do you see any debug logging in the application log while this POST commands are executed? could you please share the debug logs?

Also, are you executing this POST command from the same machine as the atlas is running? This might also be happening due to network delay. I would recommend to do such bulk operations from the same machine as Atlas to avoid any network delays.

avatar

@subash sharma

Glad that the issue is resolved. Please close the loop by accepting the answer.

avatar
Expert Contributor

Sure @Ayub Khan, We were able to resolve the issue by Restarting Apache Atlas (To close All Web Connections) and Multi-Processing to execute REST API command