Adding New Hosts with Extra Disks

Question

Hi, I need to add new hosts to an existing cluster using Ambari but the new hosts have more disks than the old nodes that I want to add to the HDFS (old nodes have /data/disk1& /data/disk2 while new nodes have/data/disk1, /data/disk2, /data/disk3 & /data/disk4). How can I add those disks after adding the nodes? can I just update  dfs.datanode.data.dir?

aen0180 · Accepted Answer

I finally found the correct way to do that. I used Ambari to create a new configuration group that includes the new hosts only, and then I added the extra disks paths to the dfs.datanode.data.dir parameter in the new configuration group only. That will integrate the extra disk on the new nodes only into the HDFS. Older nodes will not be impacted by the change in the parameter.

Reference: https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/managing-and-monitoring-ambari/content/amb_managing_host_configuration_groups.html

SagarKanani · Answer

I am not a 100% sure but i dont think you can add more disks for new machines. HDFS does a round robin writes on all disk, hence you have to either have the same no. of disks or increase the disks on the existing data nodes. Then you update dfs.datanode.data.dir accordingly.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.