Created on 07-14-2017 10:33 AM - edited 09-16-2022 04:55 AM
I am ingesting data in CDH5 hdfs using SQOOP using mysql. The job is submitted to mapreduce, but there is no activity after I get mapreduce job id:
INFO mapreduce.JobSubmitter: number of splits:1 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1500040023027_0002 INFO impl.YarnClientImpl: Submitted application application_1500040023027_0002 INFO mapreduce.Job: The url to track the job: http://pc1.localdomain.com:8088/proxy/application_1500040023027_0002/ INFO mapreduce.Job: Running job: job_1500040023027_0002
I have set up CDH5 on RHEL using cluster setup, but I have only one pc in cluster. I do see warnings to have atleast 3 datanodes, but I think it should not be an issue if I am not runninng huge activity.
I have also set the namenode and secondary namenode memory size to be 4GB. The block memory size is set to 64Mb. The log file size is also taken care of by setting them to 2GB minimum.
In Yarn settings, I have set root, and default min and max cores to be 1 and 4, and min /max memory to be 1 and 4 Gb
mapreduce screenshot shows that 0 VC and memory has been assigned to it.
Can somebody point me how to make it working.
Created 07-21-2017 11:51 AM
Following the follwing link worked for me.
Created on 07-18-2017 07:33 AM - edited 07-18-2017 07:39 AM
Why is you cluster has "red" I am suspecting some disk space I am only guessing just run the host health check also
whats the parameter you had put in
yarn.nodemanager.resource.memory-mb
yarn.scheduler.minimum-allocation-mb<
mapreduce.map.memory.mb
mapreduce.reduce.memory.mb
Created 07-18-2017 04:44 PM
<property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>1024</value> </property>
in /hadoop/ conf.cloudera.yarn/map-redsite.xml it the following are set to 0, so I changed them to what is in /haddop/conf/map-redsite.xm
<property> <name>mapreduce.map.memory.mb</name> <value>1024</value> </property> <property> <name>mapreduce.map.cpu.vcores</name> <value>1</value> </property> <property> <name>mapreduce.reduce.memory.mb</name> <value>1024</value> </property>
I could not find the parameter
yarn.nodemanager.resource.memory-mb
It is still at same point, no progress.
Created 07-21-2017 11:51 AM
Following the follwing link worked for me.
Created 07-21-2017 09:25 PM
As pointed mostly like it will be the resource allocation on those parameter good that you found it in that thread .I was close to narrow it down . Guess what sometimes it could be some socket configuration on the OS too . The logs will clearly guide us to it .