Brand new to Cloudera I have used Cloudera manager to install on a single node for trial purposes. After install there is a helth warning on HDFS -
283 under replicated blocks in the cluster. 283 total blocks in the cluster. Percentage under replicated blocks: 100.00%. Critical threshold: 40.00%.
Not sure what could cause this I have Googled around and not really found anything.
I had a 3 node cluster, in that node#1 has both DN + NN and node#2 & node#3 has only DN.
Now Node#2 & Node#3 were down. And i had free RAM and free disk space available in Node#1.
Ideally Hadoop should replicate the blocks from dead nodes i.e. Node#2&3 and should be recreated in Node#1 itself to maintain replication factor 3. But it is not happening. I had waited for hours, but not seeing any progress and showing the same health issues in cloudera manager.
Note: The same behaviour is also observed when only Node#3 is down and Node#1 & Node#2 are up and running.
Can someone please respond why this is happening?
The replication data just happens on other dataNodes, hadoop does not replicate in the same host.
If you only have one node Up, you only will hava one copy of data.
Can you post your nameNode log?? Thanks.
Indeed, if you have a replication factor of 3 and only one DataNode is alive, then there is nowhere to replicate. 3 nodes with a replication factor of 3 means the blocks are already on that one node and there is nothing to replicate/move.
Please use below command to chang ethe replication factor for existing data on Hadoop FS
$ hadoop fs -setrep -R -w 2 /
Note: You may mention number according to your requirement to have the replication factor.
$ hdfs fsck / -delete