Get hive query from resource manager using yarn REST api

[#1]

Hi All,

I am exploring YARN REST api's and looking to get hive query from the configuration. I know how to get the query from the UI but can anyone suggest how to get the query from yarn REST api.

I am using http://<rm-host>:8088/ws/v1/cluster/apps?states=running&user=hive

Thank you.

[#2 · Accepted solution]

Update: I built a custom solution for this by using YARN RM API and by parsing job_tracker & hdfs config properties.

Here is the solution for ref: RM-Parser

Feel free to fork it or provide any suggestions.

[#3]

Which URL need to be passed to get_user_query(job_tracker_url) function.

[#4]

Hi, Subash,

The code will handle passing the job tracker URL. ( it is the job tracker URL ).

In resource-manager -> Click on job -> Click on Application Master(Tracking Url).

However, after some testing, I found the above code is working MR code only as Tez and Spark jobs redirects to another URL.

I need to research a bit to make it work for all kind of apps. I hope this helps!

--

Shashant

[#5]

Update: In my recent changes, you will find the "get_user_query(job_tracker_url)" commented out.

Instead, I am fetching the Tracking URL for the job which is taking more time. But, the code is still there. Feel free to play with it.

[#6]

Hey @Shashant ,

You can try out ranger audit logs especially for Hive. Ranger captures all the column and table name. Few customization in code will help to get the full hive query.

Subash

[#7]

Thank you @subash sharma . I will check that as well..