Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

datanode config groups?

Highlighted

datanode config groups?

Rising Star

Hi Expert,

Currently, our cluster's hosts has 12 data disks(900G) installed besides the OS disk. and I created files systems /data01~data12 for every data disk. and set dfs.datanode.data.dir

/data01/hadoop/hdfs/data,/data02/hadoop/hdfs/data,/data03/hadoop/hdfs/data,/data04/hadoop/hdfs/data,/data05/hadoop/hdfs/data,/data06/hadoop/hdfs/data,/data07/hadoop/hdfs/data,/data08/hadoop/hdfs/data,/data09/hadoop/hdfs/data,/data10/hadoop/hdfs/data,/data11/hadoop/hdfs/data,/data12/hadoop/hdfs/data

Now, I'd like add several hosts with only 5 data disks(1.8T) installed to the cluster as datanodes. can I do it by creating 5 files systems /data01~data05 and adding new config groups with the dfs.datanode.data.dir in Ambari?

/data01/hadoop/hdfs/data,/data02/hadoop/hdfs/data,/data03/hadoop/hdfs/data,/data04/hadoop/hdfs/data,/data05/hadoop/hdfs/data

For such exending capablity scenario, What is the best practice?

3 REPLIES 3

Re: datanode config groups?

Super Mentor

@Huahua Wei

I guess you should use the Config Group concept via Ambari to achieve the same. You can add those new hosts to your cluster using ambari and then create a new HDFS config group and choose those hosts where you want different value for the mentioned property.

http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-user-guide/content/using_host_conf...

Re: datanode config groups?

Rising Star

This url seems to manage the existing hosts, how about adding new hosts?

Re: datanode config groups?

Super Mentor

@Huahua Wei

You can add new hosts to ambari cluster via Ambari itself as mentioned in the following link.

http://hortonworks.com/hadoop-tutorial/using-apache-ambari-add-new-nodes-existing-cluster/

- Once those hosts are added to your cluster then you can deploy your desired services/components to them like DataNode via ambari. The create the config groups.

Don't have an account?
Coming from Hortonworks? Activate your account here