Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Heap resigning and Garbage Collection tuning plays a central role in deciding how healthy and efficiently the cluster resources will be utilized.
One of the biggest challenges is to fine tune the heap to make sure neither you are underutilized or over utilizing the resources.


The following heap sizing has been made after a in depth analysis of the health of individual services. Do remember this is the base line you can add more heap to your resources depending on the kind of work load one executes.

Key take away

1. All the services present in HDP are JVM based and all need appropriate heap sizing and GC tuning
2. HDP and YARN are the base of all the services, hence make sure NN, DN, RM and NN are given sufficient heap.
3. Rule of thumb, heap sizing till 20 - 30 GB doesnt require any special tuning, anything above the value need fine tuning. <
4. 99% of time your cluster is not working efficiently because the services are suffering GC and appear RED in the UI.
5. Most of the Services have short lived connections and interactions hence always provide enough space to the young Generation in the order of 5 - 10 GB (depending on load and concurrency).
6. HiveServer2 , Spark Thrift Server, LLAP sever needs special attention as this service interacts with each and every component in cluster, slowness of any of the component will impact the connection Establishment time of these services.
7. Look for "retries/ retry" in your log, to know which services are slow and fine tune it.

HDFS

85402-nn.jpg

YARN

MapReduce2

HBase

Oozie


Zookeeper


Ambari Infra


Ambari Metrics - Hbase



Atlas



Spark Thrift Server1

Hive

https://community.hortonworks.com/articles/209789/making-hiveserver2-instance-handle-more-than-500-c...

Ambari and Ambari views should run with a heap of at least 15 GB for it to be fast.

647 Views
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 06:49 AM
Updated by:
 
Contributors
Top Kudoed Authors