We have 3 nodes in our cluster that almost were out of disk space.
I added 3 volumes to each node and mount the volume to be under: /hadoop/hdfs/data2
in each node I have those folders:
in both, hdfs user is the owner.
In ambari, I added the new directory in the hdfs config under DataNode.
DataNode directories looks like: /hadoop/hdfs/data,/hadoop/hdfs/data2
after this change, I have only 2/3 nodes that live, the other one failed with:
Connection failed: [Errno 111] Connection refused to node1-hortonworks:50010
@Moti Ben Ivgi, Please look at below thread, you may be hitting issue with host setup.
Few things to check
1) Are the datanodes actually running
2) Are those datanodes setup with a different port for some reason?
3) Double check your dfs.exclude and dfs.include files too
Are you sure that other 1/3 of datanodes are running? You may also want to make sure the NN isn't rejecting those datanodes from joining the cluster. Check the dfs.include and dfs.exclude files
Thank you guys, I resolved it.
There is a VERSION file under each data directory, in my case:
in this file, there is a layoutVersion property.
I checked this property for data2 and data directories and it was not the same.
Since I didn't have any data under data2, I remove the content of data2, restart the hdfs and it works.