question Re: Add Disks to hadoop cluster in Support Questions

Add Disks to hadoop cluster

sunilreddykallu — Fri, 04 Aug 2017 00:34:26 GMT

we have Hdp cluster running on 2.5 and ambari 2.4 . we setup through ambari i dont know data directory is taken by default and I have disk space in linux. My root folder is 50GB in linux and i think it was taken by default. IN /mnt/sda I have mounted 500GB disk. Please let me know how do i create mount to hdfs directory

Re: Add Disks to hadoop cluster

ssahi — Fri, 04 Aug 2017 00:41:12 GMT

Hi @Sam Red

You can modify the dfs.datanode.data.dir property within Ambari to add the new mountpoint you are creating.

https://community.hortonworks.com/questions/21687/how-to-increase-the-capacity-of-hdfs.html

Re: Add Disks to hadoop cluster

sunilreddykallu — Fri, 04 Aug 2017 00:49:36 GMT

Hi @Sonu Sahi

Thank you. Yes I created a directory in root folder like these and updated in ambari still it is showing same DFS and I am not sure if I followed right steps on it. I Have disk space in /mnt/sda. do i need to add this mount to that dir ?

mkdir /hadoop/hdfs/data1

chown -R hdfs:hadoop /hadoop/hdfs/data1

chmod -R 777 /hadoop/hdfs/data1

Re: Add Disks to hadoop cluster

sunilreddykallu — Fri, 04 Aug 2017 00:55:11 GMT

Hi @Sonu Sahi

i tried to Mount to that dir it keep on throwing error

mount: /mnt/sdb is not a block device

Re: Add Disks to hadoop cluster

ssahi — Fri, 04 Aug 2017 01:15:28 GMT

@Sam Red If you are adding 500GB of new disk to the host, then in Linux, you would mount/format the new disk. In Hadoop, via Ambari, you would add the new mount point to the list of dirs in the dfs.datanote.data.dir property. It will need to be mounted/formatted prior to adding the mount point in Ambari.

Re: Add Disks to hadoop cluster

sunilreddykallu — Fri, 04 Aug 2017 01:56:10 GMT

@Sonu Sahi

Thank you. Yup perfect I mounted on data1 and now i am able to see in lsblk.I restarted the node now again it is node is there any anything to need in fstab. Please let us know how to add in fstab

Re: Add Disks to hadoop cluster

ssahi — Fri, 04 Aug 2017 03:09:10 GMT

@Sam Red Awesome. In /etc/fstab a best practice is to use 'noatime,nodiratime' to help improve performance.

Re: Add Disks to hadoop cluster

sunilreddykallu — Fri, 04 Aug 2017 09:19:34 GMT

@Sonu Sahi

I added directories and mounted in data nodes only. If i attach to master nodes does it act like datanode ?

Re: Add Disks to hadoop cluster

ssahi — Fri, 04 Aug 2017 21:08:52 GMT

Not just by adding the dirs, you would need to add the datanode service in Ambari to the master hosts also, but this is not recommended. Instead, it would be better to just add another slave node to the cluster vs taking resources from the master nodes