Created 12-23-2016 09:50 AM
Hi Expert,
Currently, our cluster's hosts has 12 data disks(900G) installed besides the OS disk. and I created files systems /data01~data12 for every data disk. and set dfs.datanode.data.dir
/data01/hadoop/hdfs/data,/data02/hadoop/hdfs/data,/data03/hadoop/hdfs/data,/data04/hadoop/hdfs/data,/data05/hadoop/hdfs/data,/data06/hadoop/hdfs/data,/data07/hadoop/hdfs/data,/data08/hadoop/hdfs/data,/data09/hadoop/hdfs/data,/data10/hadoop/hdfs/data,/data11/hadoop/hdfs/data,/data12/hadoop/hdfs/data
Now, I'd like add several hosts with only 5 data disks(1.8T) installed to the cluster as datanodes. can I do it by creating 5 files systems /data01~data05 and adding new config groups with the dfs.datanode.data.dir in Ambari?
/data01/hadoop/hdfs/data,/data02/hadoop/hdfs/data,/data03/hadoop/hdfs/data,/data04/hadoop/hdfs/data,/data05/hadoop/hdfs/data
For such exending capablity scenario, What is the best practice?
Created 12-23-2016 09:53 AM
I guess you should use the Config Group concept via Ambari to achieve the same. You can add those new hosts to your cluster using ambari and then create a new HDFS config group and choose those hosts where you want different value for the mentioned property.
Created 12-23-2016 09:57 AM
This url seems to manage the existing hosts, how about adding new hosts?
Created 12-23-2016 10:00 AM
You can add new hosts to ambari cluster via Ambari itself as mentioned in the following link.
http://hortonworks.com/hadoop-tutorial/using-apache-ambari-add-new-nodes-existing-cluster/
- Once those hosts are added to your cluster then you can deploy your desired services/components to them like DataNode via ambari. The create the config groups.