question Get hive query from resource manager using yarn REST api in Archives of Support Questions (Read Only)

Get hive query from resource manager using yarn REST api

shashant_panwar — Fri, 08 Sep 2017 08:19:27 GMT

Hi All,

I am exploring YARN REST api's and looking to get hive query from the configuration. I know how to get the query from the UI but can anyone suggest how to get the query from yarn REST api.

I am using http://<rm-host>:8088/ws/v1/cluster/apps?states=running&user=hive

Thank you.

Re: Get hive query from resource manager using yarn REST api

shashant_panwar — Fri, 22 Sep 2017 00:43:21 GMT

Update: I built a custom solution for this by using YARN RM API and by parsing job_tracker & hdfs config properties.

Here is the solution for ref: RM-Parser

Feel free to fork it or provide any suggestions.

Re: Get hive query from resource manager using yarn REST api

subash_sharma — Thu, 05 Oct 2017 20:17:55 GMT

Which URL need to be passed to get_user_query(job_tracker_url) function.

Re: Get hive query from resource manager using yarn REST api

shashant_panwar — Thu, 05 Oct 2017 21:19:27 GMT

Hi, Subash,

The code will handle passing the job tracker URL. ( it is the job tracker URL ).

In resource-manager -> Click on job -> Click on Application Master(Tracking Url).

However, after some testing, I found the above code is working MR code only as Tez and Spark jobs redirects to another URL.

I need to research a bit to make it work for all kind of apps. I hope this helps!

Shashant

Re: Get hive query from resource manager using yarn REST api

shashant_panwar — Thu, 05 Oct 2017 21:23:07 GMT

Update: In my recent changes, you will find the "get_user_query(job_tracker_url)" commented out.

Instead, I am fetching the Tracking URL for the job which is taking more time. But, the code is still there. Feel free to play with it.

Re: Get hive query from resource manager using yarn REST api

subash_sharma — Fri, 06 Oct 2017 16:05:02 GMT

Hey @Shashant ,

You can try out ranger audit logs especially for Hive. Ranger captures all the column and table name. Few customization in code will help to get the full hive query.

Subash

Re: Get hive query from resource manager using yarn REST api

shashant_panwar — Tue, 17 Oct 2017 01:11:21 GMT

Thank you @subash sharma . I will check that as well..