Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

mapreduce job stuck in pending

mapreduce job stuck in pending

Contributor

Hi,

 

I am running a simple hive query (insert select), also trying with pig (store), the mapreduce job always is stuck in pending,  cannot find any error in the log. Right now i just have one node, will this cause the issue?

 

 

2016-11-02 11:27:13,311 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new applicationId: 3
2016-11-02 11:27:15,005 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application with id 3 submitted by user hdfs
2016-11-02 11:27:15,005 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Storing application with id application_1478097662713_0003
2016-11-02 11:27:15,005 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1478097662713_0003 State change from NEW to NEW_SAVING
2016-11-02 11:27:15,005 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Storing info for app: application_1478097662713_0003
2016-11-02 11:27:15,005 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hdfs IP=10.200.5.162 OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1478097662713_0003
2016-11-02 11:27:15,005 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1478097662713_0003 State change from NEW_SAVING to SUBMITTED
2016-11-02 11:27:15,019 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Accepted application application_1478097662713_0003 from user: hdfs, in queue: default, currently num of applications: 1
2016-11-02 11:27:15,020 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1478097662713_0003 State change from SUBMITTED to ACCEPTED
2016-11-02 11:27:15,020 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1478097662713_0003_000001
2016-11-02 11:27:15,020 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1478097662713_0003_000001 State change from NEW to SUBMITTED
2016-11-02 11:27:15,020 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Added Application Attempt appattempt_1478097662713_0003_000001 to scheduler from user: hdfs
2016-11-02 11:27:15,021 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1478097662713_0003_000001 State change from SUBMITTED to SCHEDULED

 

 

 

Thanks

Shannon

11 REPLIES 11

Re: mapreduce job stuck in pending

Guru

Hi,

 

You mentioned that you only have 1 node, and the symptom makes me thinking that you do not have enough resources. The logs you provided are from the RM log, can you check what is the message in the YARN application logs?  If you can see ongoing "Heart beat..." messages, then that would indicate the YARN is waiting for resources to be available before job can be launched. I would suggest you add more nodes for testing or give more resources for YARN.

 

Cheers

Re: mapreduce job stuck in pending

Rising Star

Alternatively, you can go to Resource Manager Web UI, see the resource usage for each queue in the cluster.

Re: mapreduce job stuck in pending

Contributor

I dont see the heartbeat in yarn log, in the web ui, i have memory total 2G, VCores total 2, and used/reserved are 0.

 

Thanks

Shannon

Re: mapreduce job stuck in pending

Rising Star

Which version of CDH are you running? We have seen presumably similar issues in CDH 5.5.2. 

Re: mapreduce job stuck in pending

Contributor
5.8.2

Re: mapreduce job stuck in pending

Rising Star

Can you upload the job configuration file of the MR job being stuck and a screenshot of the cluster state in RM web UI? 

Re: mapreduce job stuck in pending

Contributor

Where do i find the job config file?

Highlighted

Re: mapreduce job stuck in pending

Rising Star

You will need to find the jod Id of the job being stuck in Hive logs, and then

locate its job configuration file in HDFS under ${staging_dir}/${userName}/${jobId}

The staging directory is by default /tmp/hadoop-yarn/staging, and the userName should be hive

Re: mapreduce job stuck in pending

Contributor

Hi,

 

i dont see /tmp/hadoop-yarm. i see other hadoop related directories but not this one, what config file defines the staging dir?

 

Thanks

Shannon

Don't have an account?
Coming from Hortonworks? Activate your account here