Cluster configuration

[#1]

Can somebody tell me what would be the real time cluster configuration? I have setup Hortonworks in my home system however it is standalone in real time project what would be the cluster configuration like how many nodes, Cluster memory & RAM, Node memory & RAM, backup of cluster and all?

And while submitting Spark job to YARN how can we decide executors, memory and all those properties?

[#2 · Accepted solution]

Hi @Hardik Dave

Cluster sizing and planning would require much more detail and in-depth conversation about the use case, the data sizing, etc. A good guide that can help you down the path of sizing your cluster can be found here: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_cluster-planning/bk_cluster-planning.pdf

I would suggest using Ambari to manage your cluster where the memory allocation and settings/config will be much more visible in the UI for each technology/service being used in your cluster.