Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to find complete Hive Query for application_id?

avatar
New Contributor

Hello ,

As as Hadoop admin , where we can find the complete Hive query for any application in yarn?

We can see initial 50 character of query only on RM UI.

Thanks..

5 REPLIES 5

avatar
Master Guru

@Ashish Wadnerkar

To get full query running for the applicationid goto TEZ ui from ambari(there you can see query history)

Steps go to tez view via ambari:-

1) From the Ambari home page, hover over the top right corner, and select "Tez View"

2) Next, you can either search by application ID or the hive query itself to find your application.

3) Select your application - the entire hive query should be displayed here and also you can see the status of the query.

(or) from hiveserver2.log

avatar
New Contributor

Thanks Shu..

One thing I forgot to mentioned that ..My job in is MR mode.

Another thing is that I tried to find the complete query in hiveserver2.log , but I didn't get it.

Could you please let me know , where I can get the info.

avatar
Expert Contributor
@Ashish Wadnerkar

You can grep hiveserver2.log with "Parsing command" string and you will get complete hive query.

avatar
New Contributor

We are using cloudera version CDH-5.12.0-1.cdh5.12.0.p0.29. And we are able to parse the query in hiveServer2.logs with ": Compiling command" or ": Executing command"
e.g.

2020-02-20 09:37:23,024 INFO org.apache.hadoop.hive.ql.Driver: [HiveServer2-Handler-Pool: Thread-17501]: Executing command(queryId=${ACTUAL_HIVE_QUERY_ID}): ${ACTUAL_QUERY}

2020-02-20 09:37:32,590 INFO org.apache.hadoop.hive.ql.Driver: [HiveServer2-Handler-Pool: Thread-17332]: Compiling command(queryId=${ACTUAL_HIVE_QUERY_ID}):${ACTUAL_QUERY}

But what you need to take care of is, you need to aggregate all following lines to above line untill you find out time stamp at starting of logs. So basically it may happen that query contain multiline and in hiveServer2.log also it printed in multiline. So If you want full query then you might need to aggregate lines of logs.

 

We are actually parsing hiveServer2.log to parse all information about, query submitted by which user, what is query, what are the yarnjobs spawned to execute query, how much time each job took and Status of each yarnJob and query.

avatar
Explorer

This is not working for spark on hive. 

 

no applicaiton id found when grep query id.