Setting up 5 Node Cluster on Linux. What are the requirements for Linux and Raid, any recommended OS level settings. Please help.

[#1]

I am new to setting up Linux for Hadoop, we are using Linux (Santiago) and wondering what are the requirements for settingup Linux, I know the Ambari and Hadoop Part but not sure for OS level requirements. Like what is the RAID requirements for OS and Hortonworks software. I know we dont need it for data. Any details will be a great help.

[#2 · Accepted solution]

Raid 10 for OS disks

No RAID for HDFS disks

make sure you do this

/ - enough space

/usr/hdp - separate mount of 20 to 50 GB

/var/log - separate mount of 200 to 300GB

/hadoop - HDFS data/disk ( size is based on your use case) - No RAID

Do check this http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_Installing_HDP_AMB/content/_operating_sys...

Important link

[#3]

Thanks for the answer one more thing when you say /usr/hdp seperate mount does it has to be seperate or can I use /usr as a seperate mount of 50GB and create a directory of hdp under it.

Also does all the nodes need to have the same mount points.

third do we need any other mount points for any additional software or can I install them under /usr/hdp like Drill, oziee, spark etc.

[#4]

@Syed Naqvi That works as long as /usr has enough space.

All nodes needs that - yes

/usr/hdp will have everything

example

[root@phdns01 ~]# ls -lrt /usr/hdp

total 8

drwxr-xr-x. 2 root root 4096 Nov 19 11:19 current

drwxr-xr-x. 35 root root 4096 Jan 5 10:55 2.3.4.0-3276

[root@phdns01 ~]# pwd

/root

[#5]

Thanks much your help is appreciated.

[#6]

@Syed Naqvi Please accept the answer if it was helpful to close the loop.

[#7]

If I have the internet access do I still need to create the mirror server or I can jump directly to step2) installing server after making sure the other requirements are met m liks ssh etc or still do I have to create the mirror server and download all the repos individually. . .

[#8]

another question if you can answer does Hortonworks has any thing like Drill ?

[#9]

@Syed Naqvi Good question for another thread but I can answer this. Look at this http://hortonworks.com/innovation/stinger/

[#10]

@Syed Naqvi Apache Drill is not a part of the HDP out of the box, but its pretty simple to set that up and point that to your cluster. Is there a specific use case you have in mind that requires the use of Apache Drill?