question Re: Setting up 5 Node Cluster on Linux. What are the requirements for Linux and Raid, any recommended OS level settings. Please help. in Support Questions

Setting up 5 Node Cluster on Linux. What are the requirements for Linux and Raid, any recommended OS level settings. Please help.

syedjamalnaqvi — Fri, 16 Sep 2022 09:58:24 GMT

I am new to setting up Linux for Hadoop, we are using Linux (Santiago) and wondering what are the requirements for settingup Linux, I know the Ambari and Hadoop Part but not sure for OS level requirements. Like what is the RAID requirements for OS and Hortonworks software. I know we dont need it for data. Any details will be a great help.

Re: Setting up 5 Node Cluster on Linux. What are the requirements for Linux and Raid, any recommended OS level settings. Please help.

nsabharwal — Wed, 20 Jan 2016 04:32:25 GMT

Raid 10 for OS disks

No RAID for HDFS disks

make sure you do this

/ - enough space

/usr/hdp - separate mount of 20 to 50 GB

/var/log - separate mount of 200 to 300GB

/hadoop - HDFS data/disk ( size is based on your use case) - No RAID

Do check this http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_Installing_HDP_AMB/content/_operating_systems_requirements.html

Important link

Re: Setting up 5 Node Cluster on Linux. What are the requirements for Linux and Raid, any recommended OS level settings. Please help.

syedjamalnaqvi — Thu, 21 Jan 2016 02:38:41 GMT

Thanks for the answer one more thing when you say /usr/hdp seperate mount does it has to be seperate or can I use /usr as a seperate mount of 50GB and create a directory of hdp under it.

Also does all the nodes need to have the same mount points.

third do we need any other mount points for any additional software or can I install them under /usr/hdp like Drill, oziee, spark etc.

Re: Setting up 5 Node Cluster on Linux. What are the requirements for Linux and Raid, any recommended OS level settings. Please help.

nsabharwal — Thu, 21 Jan 2016 02:42:06 GMT

@Syed Naqvi That works as long as /usr has enough space.

All nodes needs that - yes

/usr/hdp will have everything

example

[root@phdns01 ~]# ls -lrt /usr/hdp

total 8

drwxr-xr-x. 2 root root 4096 Nov 19 11:19 current

drwxr-xr-x. 35 root root 4096 Jan 5 10:55 2.3.4.0-3276

[root@phdns01 ~]# pwd

/root

Re: Setting up 5 Node Cluster on Linux. What are the requirements for Linux and Raid, any recommended OS level settings. Please help.

syedjamalnaqvi — Thu, 21 Jan 2016 05:56:42 GMT

Thanks much your help is appreciated.

Re: Setting up 5 Node Cluster on Linux. What are the requirements for Linux and Raid, any recommended OS level settings. Please help.

syedjamalnaqvi — Fri, 22 Jan 2016 01:57:38 GMT

another question if you can answer does Hortonworks has any thing like Drill ?

Re: Setting up 5 Node Cluster on Linux. What are the requirements for Linux and Raid, any recommended OS level settings. Please help.

nsabharwal — Fri, 22 Jan 2016 02:09:42 GMT

@Syed Naqvi Please accept the answer if it was helpful to close the loop.

Re: Setting up 5 Node Cluster on Linux. What are the requirements for Linux and Raid, any recommended OS level settings. Please help.

nsabharwal — Fri, 22 Jan 2016 02:11:01 GMT

@Syed Naqvi Good question for another thread but I can answer this. Look at this http://hortonworks.com/innovation/stinger/

Re: Setting up 5 Node Cluster on Linux. What are the requirements for Linux and Raid, any recommended OS level settings. Please help.

syedjamalnaqvi — Thu, 04 Feb 2016 16:26:05 GMT

If I have the internet access do I still need to create the mirror server or I can jump directly to step2) installing server after making sure the other requirements are met m liks ssh etc or still do I have to create the mirror server and download all the repos individually. . .

Re: Setting up 5 Node Cluster on Linux. What are the requirements for Linux and Raid, any recommended OS level settings. Please help.

shaunaka — Sun, 03 Jul 2016 10:06:14 GMT

@Syed Naqvi Apache Drill is not a part of the HDP out of the box, but its pretty simple to set that up and point that to your cluster. Is there a specific use case you have in mind that requires the use of Apache Drill?