Heap resigning and Garbage Collection tuning plays a central role in deciding how healthy and efficiently the cluster resources will be utilized.
One of the biggest challenges is to fine tune the heap to make sure neither you are underutilized or over utilizing the resources.
The following heap sizing has been made after a in depth analysis of the health of individual services. Do remember this is the base line you can add more heap to your resources depending on the kind of work load one executes.
Key take away
1. All the services present in HDP are JVM based and all need appropriate heap sizing and GC tuning
2. HDP and YARN are the base of all the services, hence make sure NN, DN, RM and NN are given sufficient heap.
3. Rule of thumb, heap sizing till 20 - 30 GB doesnt require any special tuning, anything above the value need fine tuning. <
4. 99% of time your cluster is not working efficiently because the services are suffering GC and appear RED in the UI.
5. Most of the Services have short lived connections and interactions hence always provide enough space to the young Generation in the order of 5 - 10 GB (depending on load and concurrency).
6. HiveServer2 , Spark Thrift Server, LLAP sever needs special attention as this service interacts with each and every component in cluster, slowness of any of the component will impact the connection Establishment time of these services.
7. Look for "retries/ retry" in your log, to know which services are slow and fine tune it.
HDFS
YARN
MapReduce2
HBase
Oozie
Zookeeper
Ambari Infra
Ambari Metrics - Hbase
Atlas
Spark Thrift Server1
Ambari and Ambari views should run with a heap of at least 15 GB for it to be fast.
