Created 07-10-2018 07:32 PM
hi all
typical production cluster should be as the following structure ( based on hadoop 2.6.5 version and ambari 2.6.1 version)
3 master machines ( name node machines , while aambari server installed on master02 machine and active stasndby name node are master01/03 )
3 kafka machines
5-250 workers machines ( data node machines )
we want to know what are the minimal HW requirements for master machines ?
memory ?
CPU ?
/var partition size ?
/ partition size ?
swap ?
Created 07-10-2018 11:19 PM
You can calculate the requirements based upon the amount and type of data using the following guide:
https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.3/bk_cluster-planning/bk_cluster-planning.pdf
Created 07-12-2018 02:25 PM
I don't agree with type of nodes, you should have 03 :
- 02 Admin machines (at least 8 cores / 32 Go RAM) : Ambari / Ranger / ETC
- 03 Master machines (at least 8 cores / 64 Go RAM): NN / NN HA / Hive Server and others master services
- workers machines (at least 16 cores / 128 Go RAM)
@Sparsh Singhal point you to a good document have a look at it too.
Created 07-13-2018 01:53 PM
what about partitions size for /var and / ? and swap policy ?
Created 07-17-2018 12:28 PM
Document has recommended partition size for / and /var. /var mostly has your logs which can usually take up a lot of space. AFAIK swap should be disabled and swappiness should be 0.