Add Disks to hadoop cluster

[#1]

we have Hdp cluster running on 2.5 and ambari 2.4 . we setup through ambari i dont know data directory is taken by default and I have disk space in linux. My root folder is 50GB in linux and i think it was taken by default. IN /mnt/sda I have mounted 500GB disk. Please let me know how do i create mount to hdfs directory

[#2 · Accepted solution]

Hi @Sam Red

You can modify the dfs.datanode.data.dir property within Ambari to add the new mountpoint you are creating.

https://community.hortonworks.com/questions/21687/how-to-increase-the-capacity-of-hdfs.html

[#3]

Hi @Sonu Sahi

Thank you. Yes I created a directory in root folder like these and updated in ambari still it is showing same DFS and I am not sure if I followed right steps on it. I Have disk space in /mnt/sda. do i need to add this mount to that dir ?

mkdir /hadoop/hdfs/data1

chown -R hdfs:hadoop /hadoop/hdfs/data1

chmod -R 777 /hadoop/hdfs/data1

[#4]

Hi @Sonu Sahi

i tried to Mount to that dir it keep on throwing error

mount: /mnt/sdb is not a block device

[#5]

@Sam Red If you are adding 500GB of new disk to the host, then in Linux, you would mount/format the new disk. In Hadoop, via Ambari, you would add the new mount point to the list of dirs in the dfs.datanote.data.dir property. It will need to be mounted/formatted prior to adding the mount point in Ambari.

[#6]

@Sonu Sahi

Thank you. Yup perfect I mounted on data1 and now i am able to see in lsblk.I restarted the node now again it is node is there any anything to need in fstab. Please let us know how to add in fstab

[#7]

@Sam Red Awesome. In /etc/fstab a best practice is to use 'noatime,nodiratime' to help improve performance.

[#8]

@Sonu Sahi

I added directories and mounted in data nodes only. If i attach to master nodes does it act like datanode ?

[#9]

Not just by adding the dirs, you would need to add the datanode service in Ambari to the master hosts also, but this is not recommended. Instead, it would be better to just add another slave node to the cluster vs taking resources from the master nodes