Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Could not find status of

[#1]

Hi 

I have some issue with my queries in Hive, they are very simple queries. 

One day start to fail and this is the error :

Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Could not find status of job:job_1639152705224_0018

INFO  : Kill Command = /opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib/hadoop/bin/hadoop job  -kill job_1639152705224_0018
INFO  : Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
INFO  : 2021-12-10 17:35:35,913 Stage-1 map = 0%,  reduce = 0%
INFO  : 2021-12-10 17:35:43,176 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 7.34 sec
ERROR : Ended Job = job_1639152705224_0018 with exception 'java.io.IOException(Could not find status of job:job_1639152705224_0018)'
java.io.IOException: Could not find status of job:job_1639152705224_0018
	at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:300)
	at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:574)
	at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:454)
	at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:151)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2200)
	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1843)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1563)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1339)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1334)
	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:256)
	at org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:92)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:357)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Could not find status of job:job_1639152705224_0018
INFO  : Completed executing command(queryId=hive_20211210173528_ff76c3df-a33b-41d0-b328-460c9b65deda); Time taken: 21.669 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Could not find status of job:job_1639152705224_0018 (state=08S01,code=1)
Closing: 0: jdbc:hive2://datap01.agbar.ga.local:10000/default

 Thanks 

[#2]

@Gcima009 It will be difficult to say what's the issue is from this error stack.

Could you generate yarn application log for this job, and review it or attach it here, or only paste the detailed error message?

yarn logs -applicationId application_1639152705224_0018 > app_log.out

[#3]

hi @smruti . 

this is the response.

[root@datap01 ~]# yarn logs -applicationId application_1639152705224_0018 > app_log.out
WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
 INFO client.RMProxy: Connecting to ResourceManager at srvbigdatap01.agbar.ga.local/10.200.14.72:8032
File /tmp/logs/hdfs/logs/application_1639152705224_0018 does not exist.

Can not find any log file matching the pattern: [ALL] for the application: application_1639152705224_0018
Can not find the logs for the application: application_1639152705224_0018 with the appOwner: hdfs

😕 

[#4]

@Gcima009 are you trying to collect the logs with the same user that you submitted the job with?

This query completed the map phase, and failed in reducer phase.

If you are not able to collect the app logs, do check the HS2 log with the query ID hive_20211210173528_ff76c3df-a33b-41d0-b328-460c9b65deda if you get more information what caused the job to fail.