Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Setting up 5 Node Cluster on Linux. What are the requirements for Linux and Raid, any recommended OS level settings. Please help.

avatar
Contributor

I am new to setting up Linux for Hadoop, we are using Linux (Santiago) and wondering what are the requirements for settingup Linux, I know the Ambari and Hadoop Part but not sure for OS level requirements. Like what is the RAID requirements for OS and Hortonworks software. I know we dont need it for data. Any details will be a great help.

1 ACCEPTED SOLUTION

avatar
Master Mentor

Raid 10 for OS disks

No RAID for HDFS disks

make sure you do this

/ - enough space

/usr/hdp - separate mount of 20 to 50 GB

/var/log - separate mount of 200 to 300GB

/hadoop - HDFS data/disk ( size is based on your use case) - No RAID

Do check this http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_Installing_HDP_AMB/content/_operating_sys...

Important link

View solution in original post

9 REPLIES 9

avatar
Master Mentor

Raid 10 for OS disks

No RAID for HDFS disks

make sure you do this

/ - enough space

/usr/hdp - separate mount of 20 to 50 GB

/var/log - separate mount of 200 to 300GB

/hadoop - HDFS data/disk ( size is based on your use case) - No RAID

Do check this http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_Installing_HDP_AMB/content/_operating_sys...

Important link

avatar
Contributor

Thanks for the answer one more thing when you say /usr/hdp seperate mount does it has to be seperate or can I use /usr as a seperate mount of 50GB and create a directory of hdp under it.

Also does all the nodes need to have the same mount points.

third do we need any other mount points for any additional software or can I install them under /usr/hdp like Drill, oziee, spark etc.

avatar
Master Mentor

@Syed Naqvi That works as long as /usr has enough space.

All nodes needs that - yes

/usr/hdp will have everything

example

[root@phdns01 ~]# ls -lrt /usr/hdp

total 8

drwxr-xr-x. 2 root root 4096 Nov 19 11:19 current

drwxr-xr-x. 35 root root 4096 Jan 5 10:55 2.3.4.0-3276

[root@phdns01 ~]# pwd

/root

avatar
Contributor

Thanks much your help is appreciated.

avatar
Master Mentor

@Syed Naqvi Please accept the answer if it was helpful to close the loop.

avatar
Contributor

If I have the internet access do I still need to create the mirror server or I can jump directly to step2) installing server after making sure the other requirements are met m liks ssh etc or still do I have to create the mirror server and download all the repos individually. . .

avatar
Contributor

another question if you can answer does Hortonworks has any thing like Drill ?

avatar
Master Mentor

@Syed Naqvi Good question for another thread but I can answer this. Look at this http://hortonworks.com/innovation/stinger/

avatar
Explorer

@Syed Naqvi Apache Drill is not a part of the HDP out of the box, but its pretty simple to set that up and point that to your cluster. Is there a specific use case you have in mind that requires the use of Apache Drill?