Hive Queries started failing

[#1]

Starting yesterday, I have faced this problem multiple times with difference queries:

What seems wrong here? I am able to figure out nothing here but it is coming up in different queries that used to work just fine. I have checked and temp directory has the required space and my nodes are healthy too.

In stderr, I get:

INFO  : Hadoop job information for Stage-11: number of mappers: 1; number of reducers: 1
INFO  : 2017-07-21 15:52:16,549 Stage-11 map = 0%,  reduce = 0%
INFO  : 2017-07-21 15:52:23,783 Stage-11 map = 100%,  reduce = 0%, Cumulative CPU 4.23 sec
INFO  : 2017-07-21 15:52:33,181 Stage-11 map = 100%,  reduce = 100%, Cumulative CPU 6.89 sec
INFO  : MapReduce Total cumulative CPU time: 6 seconds 890 msec
INFO  : Ended Job = <a href="IP:8888/jobbrowser/jobs/job_1499692338187_45811">job_1499692338187_45811</a>
INFO  : Starting task [Stage-20:MAPREDLOCAL] in serial mode
ERROR : Execution failed with exit status: 1
ERROR : Obtaining error information
ERROR : 
Task failed!
Task ID:
  Stage-20

Logs:

ERROR : <a href="http://ec2-35-154-150-43.ap-south-1.compute.amazonaws.com:8888/filebrowser/view=/var/log/hive/hadoop-cmf-CD-HIVE-XCVXskZf-HIVESERVER2-ip-172-31-4-192.ap-south-1.compute.internal.log.out">/var/log/hive/hadoop-cmf-CD-HIVE-XCVXskZf-HIVESERVER2-ip-172-31-4-192.ap-south-1.compute.internal.log.out</a>
ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
INFO  : MapReduce Jobs Launched: 
INFO  : Stage-Stage-1: Map: 1  Reduce: 1   Cumulative CPU: 5.59 sec   HDFS Read: 3487128 HDFS Write: 192 SUCCESS
INFO  : Stage-Stage-7: Map: 1  Reduce: 1   Cumulative CPU: 7.18 sec   HDFS Read: 3484698 HDFS Write: 1572 SUCCESS
INFO  : Stage-Stage-11: Map: 1  Reduce: 1   Cumulative CPU: 6.89 sec   HDFS Read: 3484445 HDFS Write: 1568 SUCCESS
INFO  : Total MapReduce CPU Time Spent: 19 seconds 660 msec
INFO  : Completed executing command(queryId=hive_20170721155151_da599ee2-d378-46ac-a2b3-a9dfd9824850); Time taken: 70.153 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask (state=08S01,code=1)
Closing: 0: jdbc:hive2://ip-172-31-4-192.ap-south-1.compute.internal:10000/default
Intercepting System.exit(2)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.Hive2Main], exit code [2]

[#2]

@Simran Kaur

Check the application log for job_1499692338187_45811 for details on the error.

[#3]

@Simran Kaur

I see in stdout as Oozie launcher failed.

Are you trying to run Hive action in Oozie.

If that's the case please use command (yarn logs -applicationId application_1499692338187_45811)to get logs, or follow the below KB article to trace for logs to debug further

https://community.hortonworks.com/articles/9148/troubleshooting-an-oozie-flow.html

[#4]

@Viswa: I get " Log aggregation has not completed or is not enabled.

[#5]

Checked through oozie dashboard. That's the same and only error I get there too.