Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Get hive query from resource manager using yarn REST api

avatar
Rising Star

Hi All,

I am exploring YARN REST api's and looking to get hive query from the configuration. I know how to get the query from the UI but can anyone suggest how to get the query from yarn REST api.

I am using http://<rm-host>:8088/ws/v1/cluster/apps?states=running&user=hive

Thank you.

1 ACCEPTED SOLUTION

avatar
Rising Star

Update: I built a custom solution for this by using YARN RM API and by parsing job_tracker & hdfs config properties.

Here is the solution for ref: RM-Parser

Feel free to fork it or provide any suggestions.

View solution in original post

6 REPLIES 6

avatar
Rising Star

Update: I built a custom solution for this by using YARN RM API and by parsing job_tracker & hdfs config properties.

Here is the solution for ref: RM-Parser

Feel free to fork it or provide any suggestions.

avatar
Expert Contributor

Which URL need to be passed to get_user_query(job_tracker_url) function.

avatar
Rising Star

Hi, Subash,

The code will handle passing the job tracker URL. ( it is the job tracker URL ).

In resource-manager -> Click on job -> Click on Application Master(Tracking Url).

However, after some testing, I found the above code is working MR code only as Tez and Spark jobs redirects to another URL.

I need to research a bit to make it work for all kind of apps. I hope this helps!

--

Shashant

avatar
Rising Star

Update: In my recent changes, you will find the "get_user_query(job_tracker_url)" commented out.

Instead, I am fetching the Tracking URL for the job which is taking more time. But, the code is still there. Feel free to play with it.

avatar
Expert Contributor

Hey @Shashant ,

You can try out ranger audit logs especially for Hive. Ranger captures all the column and table name. Few customization in code will help to get the full hive query.

Subash

avatar
Rising Star

Thank you @subash sharma . I will check that as well..