Reply
New Contributor
Posts: 3
Registered: ‎07-27-2015

mapred streaming job stays on status prep and does not run

Hi together

 

I installed a single-node CDH environment with parcels. When I run a mapred job with streaming the job stays on prep status and does not run. But there is no error message.

 

My environment:

Ubuntu 14.04 on VMware ESXi Host

CDH 5.4 installed with parcels

Run jobs with user root

 

What I already did for troubleshooting but with no success:

I installed another single-node CDH to have default settings.

I tested the python mapper an reducer on another cluster where it ran though correctly.

I set the rights on the data directories to 777 and changed owners to user root an group supergroup

 

Does anyone have an idea why this does not work. Is this a known issue? In which logfile I could find a hint?

 

I would be very thankful for a hint to solve the problem because I need to run mapred, hive and impala jobs on this environment for my diploma thesis.

 

Best regards,

Chris

 

 

Cloudera Employee
Posts: 322
Registered: ‎01-16-2014

Re: mapred streaming job stays on status prep and does not run

Check the RM logs to see whatthe scheduler is doing. You might be asking for more resources than you have in the node.

 

Wilfred

New Contributor
Posts: 3
Registered: ‎07-27-2015

Re: mapred streaming job stays on status prep and does not run

Hi Wilfred

 

Thanks for your respones. On this VM I have 12 GB RAM and 8vCPU. I checked the logs again but without success. To me it looks ok. The job is submitted, accepted and scheduled. But it does not do anything.

 

I started the job from command line as follows:

 

root@cloudera20:~# hadoop jar /opt/cloudera/parcels/CDH-5.4.4-1.cdh5.4.4.p0.4/lib/hadoop-mapreduce/hadoop-streaming.jar -file /root/cas-mapper4.py -mapper /root/cas-mapper4.py -file /root/cas-reducer4.py -reducer /root/cas-reducer4.py -input /tmp/wlan/*/ -output /user/root/job001


15/07/31 20:56:52 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.
packageJobJar: [/root/cas-mapper4.py, /root/cas-reducer4.py] [/opt/cloudera/parcels/CDH-5.4.4-1.cdh5.4.4.p0.4/jars/hadoop-streaming-2.6.0-cdh5.4.4.jar] /tmp/streamjob3106949066792837337.jar tmpDir=null
15/07/31 20:56:53 INFO client.RMProxy: Connecting to ResourceManager at cloudera20.hadoop.entiiso.com/10.36.0.20:8032
15/07/31 20:56:54 INFO client.RMProxy: Connecting to ResourceManager at cloudera20.hadoop.entiiso.com/10.36.0.20:8032
15/07/31 20:56:55 INFO mapred.FileInputFormat: Total input paths to process : 512
15/07/31 20:56:55 INFO mapreduce.JobSubmitter: number of splits:512
15/07/31 20:56:55 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1438368709632_0001
15/07/31 20:56:56 INFO impl.YarnClientImpl: Submitted application application_1438368709632_0001
15/07/31 20:56:56 INFO mapreduce.Job: The url to track the job: http://cloudera20.hadoop.entiiso.com:8088/proxy/application_1438368709632_0001/
15/07/31 20:56:56 INFO mapreduce.Job: Running job: job_1438368709632_0001

 

 

 

Following messages appear in log:

 

2015-07-31 20:56:54,274 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new applicationId: 1
2015-07-31 20:56:56,287 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application with id 1 submitted by user root
2015-07-31 20:56:56,288 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Storing application with id application_1438368709632_0001
2015-07-31 20:56:56,289 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root     IP=10.36.0.20   OPERATION=Submit Application Request    TARGET=ClientRMService  RESULT=SUCCESS       APPID=application_1438368709632_0001
2015-07-31 20:56:56,290 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1438368709632_0001 State change from NEW to NEW_SAVING
2015-07-31 20:56:56,298 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Storing info for app: application_1438368709632_0001
2015-07-31 20:56:56,298 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1438368709632_0001 State change from NEW_SAVING to SUBMITTED
2015-07-31 20:56:56,331 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Accepted application application_1438368709632_0001 from user: root, in queue: default, currently num of applications: 1
2015-07-31 20:56:56,337 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1438368709632_0001 State change from SUBMITTED to ACCEPTED
2015-07-31 20:56:56,367 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1438368709632_0001_000001
2015-07-31 20:56:56,368 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1438368709632_0001_000001 State change from NEW to SUBMITTED
2015-07-31 20:56:56,384 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Added Application Attempt appattempt_1438368709632_0001_000001 to scheduler from user: root
2015-07-31 20:56:56,387 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1438368709632_0001_000001 State change from SUBMITTED to SCHEDULED

 

 

I really do not have any idea what I could do else. If there is any hint I will follow it.

 

Chris

Highlighted
Cloudera Employee
Posts: 322
Registered: ‎01-16-2014

Re: mapred streaming job stays on status prep and does not run

If an app stays in the scheduled state than it has not started. The application gets a container and the container needs to move into a running state.

That does not seem to happen for you. In the RM web UI it should show you the capacity of the cluster for containers (memory and vcores) to run an application you need at least space for two containers: the Application Master and one for a task.

Check the 6 gotchas blog. There are a number of good resources referenced in that blog. BTW: this looks like a scheduler issue.

 

Wilfred

New Contributor
Posts: 3
Registered: ‎07-27-2015

Re: mapred streaming job stays on status prep and does not run

Hi Wilfred

 

Thanks for your response. Meanwhile I tested a lot but still with no success. I think you are right, it is a scheduler problem. I tried a lot of parameters, then I installed CDH sometimes with parcels and once unmanaged manually from scratch. I even tested the cloudera quick installation guide with pseudo distributed mode. I still could not run a simple mapred job. It really seems to be a YARN problem with a single node installation. So I installed again from scratch with MR1 and the jobs ran through like a charm.

 

I can do my tests with this environment but it would be very interesting if it is possible to install a single node cluster with yarn maybe on another environment (e.g. other OS) and if it works.

 

I think it is not a big problem in YARN. Maybe only a setting of some parameters. If you have news to this problem or the cloudera support team encounters this problem as well in another case I would be very interested in the solution.

 

Thanks a lot and best regards,

Chris 

Cloudera Employee
Posts: 322
Registered: ‎01-16-2014

Re: mapred streaming job stays on status prep and does not run

We have a tuning guide and a blog entry that should cover all the points of the setup of the scheduler.

Based on your comments those two should help. Setting up the scheduler in the cluster is one of the more complex actions and a lot of people run into issues with that.

 

Wilfred