Support Questions
Find answers, ask questions, and share your expertise

Add Disks to hadoop cluster

we have Hdp cluster running on 2.5 and ambari 2.4 . we setup through ambari i dont know data directory is taken by default and I have disk space in linux. My root folder is 50GB in linux and i think it was taken by default. IN /mnt/sda I have mounted 500GB disk. Please let me know how do i create mount to hdfs directory

1 ACCEPTED SOLUTION

Guru

Hi @Sam Red

You can modify the dfs.datanode.data.dir property within Ambari to add the new mountpoint you are creating.

https://community.hortonworks.com/questions/21687/how-to-increase-the-capacity-of-hdfs.html

View solution in original post

8 REPLIES 8

Guru

Hi @Sam Red

You can modify the dfs.datanode.data.dir property within Ambari to add the new mountpoint you are creating.

https://community.hortonworks.com/questions/21687/how-to-increase-the-capacity-of-hdfs.html

Hi @Sonu Sahi

Thank you. Yes I created a directory in root folder like these and updated in ambari still it is showing same DFS and I am not sure if I followed right steps on it. I Have disk space in /mnt/sda. do i need to add this mount to that dir ?

mkdir /hadoop/hdfs/data1

chown -R hdfs:hadoop /hadoop/hdfs/data1

chmod -R 777 /hadoop/hdfs/data1

Hi @Sonu Sahi

i tried to Mount to that dir it keep on throwing error

mount: /mnt/sdb is not a block device

Guru

@Sam Red If you are adding 500GB of new disk to the host, then in Linux, you would mount/format the new disk. In Hadoop, via Ambari, you would add the new mount point to the list of dirs in the dfs.datanote.data.dir property. It will need to be mounted/formatted prior to adding the mount point in Ambari.

@Sonu Sahi

Thank you. Yup perfect I mounted on data1 and now i am able to see in lsblk.I restarted the node now again it is node is there any anything to need in fstab. Please let us know how to add in fstab

Guru

@Sam Red Awesome. In /etc/fstab a best practice is to use 'noatime,nodiratime' to help improve performance.

@Sonu Sahi

I added directories and mounted in data nodes only. If i attach to master nodes does it act like datanode ?

Guru

Not just by adding the dirs, you would need to add the datanode service in Ambari to the master hosts also, but this is not recommended. Instead, it would be better to just add another slave node to the cluster vs taking resources from the master nodes

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.