11-19-2014 01:47 AM
Hello dear reader,
I am configuring an hadoop cluster CDH5 on 3 nodes.
1 NameNode + ResourceManager 1 DataNode + nodemanager + historyserver 1 DataNode + nodemanager
I configured YARN using advices from Cloudera and other websites
I try to use the example calcultating Pi with this command on my namenode :
asimon@namenode:/etc/hadoop/conf$ sudo hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.2.0.jar pi 2 2
which gives me :
Number of Maps = 2 Samples per Map = 2 Wrote input for Map #0 Wrote input for Map #1 Starting Job 14/11/18 17:03:04 INFO client.RMProxy: Connecting to ResourceManager at namenode.beijaflore.com/192.168.54.111:8032 14/11/18 17:03:04 INFO input.FileInputFormat: Total input paths to process : 2 14/11/18 17:03:04 INFO mapreduce.JobSubmitter: number of splits:2 14/11/18 17:03:05 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1416326563510_0001 14/11/18 17:03:05 INFO impl.YarnClientImpl: Submitted application application_1416326563510_0001 14/11/18 17:03:05 INFO mapreduce.Job: The url to track the job: http://namenode.beijaflore.com:8088/proxy/application_1416326563510_0001/ 14/11/18 17:03:05 INFO mapreduce.Job: Running job: job_1416326563510_0001
But the job is stuck here and can't show the actual job running... My 3 VMs are server and i don't have any UI to check the job is really pending.
For the actuals logs i get in mapred-mapred-historyserver-datanode1.out
14/11/18 17:32:34 INFO hs.JobHistory: History Cleaner started 14/11/18 17:32:34 INFO hs.JobHistory: History Cleaner complete 14/11/18 17:35:04 INFO hs.JobHistory: Starting scan to move intermediate done files 14/11/18 17:38:04 INFO hs.JobHistory: Starting scan to move intermediate done files 14/11/18 17:41:04 INFO hs.JobHistory: Starting scan to move intermediate done files 14/11/18 17:44:04 INFO hs.JobHistory: Starting scan to move intermediate done files
(I interupted the first Job and tried to run it a second time that's why time isn't matching)
Every 3 minutes it does this...(and i tried it for one night, it just keep going)
I don't know what is wrong. I thought i had an hint in configuring minimum and maximum memory size used (because my total config is only 6CPU 8GoRAM)... But it actualy change nothing
If you have any idea of what is wrong or where can I look to find the problem, I would be grateful to any information.