Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Get hive query from resource manager using yarn REST api

Solved Go to solution

Get hive query from resource manager using yarn REST api

Rising Star

Hi All,

I am exploring YARN REST api's and looking to get hive query from the configuration. I know how to get the query from the UI but can anyone suggest how to get the query from yarn REST api.

I am using http://<rm-host>:8088/ws/v1/cluster/apps?states=running&user=hive

Thank you.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Get hive query from resource manager using yarn REST api

Rising Star

Update: I built a custom solution for this by using YARN RM API and by parsing job_tracker & hdfs config properties.

Here is the solution for ref: RM-Parser

Feel free to fork it or provide any suggestions.

6 REPLIES 6

Re: Get hive query from resource manager using yarn REST api

Rising Star

Update: I built a custom solution for this by using YARN RM API and by parsing job_tracker & hdfs config properties.

Here is the solution for ref: RM-Parser

Feel free to fork it or provide any suggestions.

Re: Get hive query from resource manager using yarn REST api

New Contributor

Which URL need to be passed to get_user_query(job_tracker_url) function.

Re: Get hive query from resource manager using yarn REST api

Rising Star

Hi, Subash,

The code will handle passing the job tracker URL. ( it is the job tracker URL ).

In resource-manager -> Click on job -> Click on Application Master(Tracking Url).

However, after some testing, I found the above code is working MR code only as Tez and Spark jobs redirects to another URL.

I need to research a bit to make it work for all kind of apps. I hope this helps!

--

Shashant

Re: Get hive query from resource manager using yarn REST api

Rising Star

Update: In my recent changes, you will find the "get_user_query(job_tracker_url)" commented out.

Instead, I am fetching the Tracking URL for the job which is taking more time. But, the code is still there. Feel free to play with it.

Highlighted

Re: Get hive query from resource manager using yarn REST api

New Contributor

Hey @Shashant ,

You can try out ranger audit logs especially for Hive. Ranger captures all the column and table name. Few customization in code will help to get the full hive query.

Subash

Re: Get hive query from resource manager using yarn REST api

Rising Star

Thank you @subash sharma . I will check that as well..