07-27-2015 09:52 AM
I installed a single-node CDH environment with parcels. When I run a mapred job with streaming the job stays on prep status and does not run. But there is no error message.
Ubuntu 14.04 on VMware ESXi Host
CDH 5.4 installed with parcels
Run jobs with user root
What I already did for troubleshooting but with no success:
I installed another single-node CDH to have default settings.
I tested the python mapper an reducer on another cluster where it ran though correctly.
I set the rights on the data directories to 777 and changed owners to user root an group supergroup
Does anyone have an idea why this does not work. Is this a known issue? In which logfile I could find a hint?
I would be very thankful for a hint to solve the problem because I need to run mapred, hive and impala jobs on this environment for my diploma thesis.
07-31-2015 12:09 PM
Thanks for your respones. On this VM I have 12 GB RAM and 8vCPU. I checked the logs again but without success. To me it looks ok. The job is submitted, accepted and scheduled. But it does not do anything.
I started the job from command line as follows:
root@cloudera20:~# hadoop jar /opt/cloudera/parcels/CDH-5.4.4-1.cdh5.4.4.p0.4/lib/hadoop-mapreduce/hadoop-streaming.jar -file /root/cas-mapper4.py -mapper /root/cas-mapper4.py -file /root/cas-reducer4.py -reducer /root/cas-reducer4.py -input /tmp/wlan/*/ -output /user/root/job001
15/07/31 20:56:52 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.
packageJobJar: [/root/cas-mapper4.py, /root/cas-reducer4.py] [/opt/cloudera/parcels/CDH-5.4.4-1.cdh5.4.4.p0.4/jars/hadoop-streaming-2.6.0-cdh5.4.4.jar] /tmp/streamjob3106949066792837337.jar tmpDir=null
15/07/31 20:56:53 INFO client.RMProxy: Connecting to ResourceManager at cloudera20.hadoop.entiiso.com/10.36.0.20:8032
15/07/31 20:56:54 INFO client.RMProxy: Connecting to ResourceManager at cloudera20.hadoop.entiiso.com/10.36.0.20:8032
15/07/31 20:56:55 INFO mapred.FileInputFormat: Total input paths to process : 512
15/07/31 20:56:55 INFO mapreduce.JobSubmitter: number of splits:512
15/07/31 20:56:55 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1438368709632_0001
15/07/31 20:56:56 INFO impl.YarnClientImpl: Submitted application application_1438368709632_0001
15/07/31 20:56:56 INFO mapreduce.Job: The url to track the job: http://cloudera20.hadoop.entiiso.com:8088/proxy/application_1438368709632_0001/
15/07/31 20:56:56 INFO mapreduce.Job: Running job: job_1438368709632_0001
Following messages appear in log:
2015-07-31 20:56:54,274 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new applicationId: 1
2015-07-31 20:56:56,287 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application with id 1 submitted by user root
2015-07-31 20:56:56,288 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Storing application with id application_1438368709632_0001
2015-07-31 20:56:56,289 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root IP=10.36.0.20 OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1438368709632_0001
2015-07-31 20:56:56,290 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1438368709632_0001 State change from NEW to NEW_SAVING
2015-07-31 20:56:56,298 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Storing info for app: application_1438368709632_0001
2015-07-31 20:56:56,298 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1438368709632_0001 State change from NEW_SAVING to SUBMITTED
2015-07-31 20:56:56,331 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Accepted application application_1438368709632_0001 from user: root, in queue: default, currently num of applications: 1
2015-07-31 20:56:56,337 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1438368709632_0001 State change from SUBMITTED to ACCEPTED
2015-07-31 20:56:56,367 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1438368709632_0001_000001
2015-07-31 20:56:56,368 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1438368709632_0001_000001 State change from NEW to SUBMITTED
2015-07-31 20:56:56,384 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Added Application Attempt appattempt_1438368709632_0001_000001 to scheduler from user: root
2015-07-31 20:56:56,387 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1438368709632_0001_000001 State change from SUBMITTED to SCHEDULED
I really do not have any idea what I could do else. If there is any hint I will follow it.
08-02-2015 05:21 PM
If an app stays in the scheduled state than it has not started. The application gets a container and the container needs to move into a running state.
That does not seem to happen for you. In the RM web UI it should show you the capacity of the cluster for containers (memory and vcores) to run an application you need at least space for two containers: the Application Master and one for a task.
Check the 6 gotchas blog. There are a number of good resources referenced in that blog. BTW: this looks like a scheduler issue.
08-04-2015 10:36 AM
Thanks for your response. Meanwhile I tested a lot but still with no success. I think you are right, it is a scheduler problem. I tried a lot of parameters, then I installed CDH sometimes with parcels and once unmanaged manually from scratch. I even tested the cloudera quick installation guide with pseudo distributed mode. I still could not run a simple mapred job. It really seems to be a YARN problem with a single node installation. So I installed again from scratch with MR1 and the jobs ran through like a charm.
I can do my tests with this environment but it would be very interesting if it is possible to install a single node cluster with yarn maybe on another environment (e.g. other OS) and if it works.
I think it is not a big problem in YARN. Maybe only a setting of some parameters. If you have news to this problem or the cloudera support team encounters this problem as well in another case I would be very interested in the solution.
Thanks a lot and best regards,
08-04-2015 05:16 PM