Created 08-01-2016 06:11 AM
Hi Team,
I have 3 virtual machines in HDP cluster ,if i have huge capacity in data nodes disk in TBs so can i use the same disk with diff mount points to store Data nodes data, NN namenode data, SN data, JT data (master node data) and /usr and /var .
I know then if my disk has some issue then all data will be affected
basically i wanted to know if my data node disks have lot of space in TBs, so do you recommend creating diff mounts on same data node disks for diff purposes like /usr,/var and storing NN SN JT data
Also each HDP version data is in /usr/hdp
Created 08-01-2016 06:32 AM
Don't mount all the partitions on same disk, it will create lot of disk I/O.
I would suggest to partition the disk according to your requirement and use dedicated disks for each component like DNs/NNs etc.
Also, Please have a look at below links for Hadoop performance tuning
http://crazyadmins.com/tune-hadoop-cluster-to-get-maximum-performance-part-1/
http://crazyadmins.com/tune-hadoop-cluster-to-get-maximum-performance-part-2/
Created 08-01-2016 06:26 AM
Is this a sandbox, just for testing and experimenting? If yes, then it's fine. For anything else, no. This is not recommended.
Created 08-01-2016 06:32 AM
Don't mount all the partitions on same disk, it will create lot of disk I/O.
I would suggest to partition the disk according to your requirement and use dedicated disks for each component like DNs/NNs etc.
Also, Please have a look at below links for Hadoop performance tuning
http://crazyadmins.com/tune-hadoop-cluster-to-get-maximum-performance-part-1/
http://crazyadmins.com/tune-hadoop-cluster-to-get-maximum-performance-part-2/
Created 08-01-2016 07:09 AM
Thanks a lot Kuldeep i agree and thats why i wanted suggestions from experts like you 🙂
Created 08-08-2016 02:07 AM
I think there is an HCC article on this very topic, but https://martin.atlassian.net/wiki/x/EoC3Ag is a blog post I wrote back in mid-2015 on this subject as well in case it helps any. Good luck!