Yarn Resource Manager Repeated garbage collection

[ISLAND]

I have a cluster.

Recently, it often happens "RESOURCE_MANAGER_GC_DURATION concerning".

Observe the GC Collection Time case, each time it lasts for about 12s to 18s duration.

JVM Heap size configuration situation 4GB, the overall usage of about 92%, in a high water situation.

Please help to give suggestions.

The clusters of  following specifications:

CDH version:5.7.0

Node info	service
HostName: P1 Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz Memory:256G	cloudera-mgmt-HOSTMONITOR, cloudera-mgmt-EVENTSERVER, cloudera-mgmt-SERVICEMONITOR, cloudera-mgmt-ALERTPUBLISHER, cloudera-mgmt-ACITVITYMONITOR postgresql, hdfs-FAILOVERCONTROLLER, hdfs-NAMENODE, hbase-MASTER, yarn-RESOURCEMANAGER, yarn-NODEMANAGER zookeeper-ZOOKEEPER_SERVER
HostName: Q1 Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz Memory:256G	hdfs-FAILOVERCONTROLLER, hdfs-NAMENODE, hdfs-HTTPFS, hbase-MASTER, yarn-RESOURCEMANAGER, yarn-NODEMANAGER, oozie-OOZIE_SERVER zookeeper-ZOOKEEPER_SERVER
HostName: R1 Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz  Memory:512G	hdfs-DATANODE, hdfs-JOURNALNODE, zookeeper-ZOOKEEPER_SERVER, hbase-REGION_SERVER, Hive-HIVE Metastore server Hive-HiveServer2 yarn-NODEMANAGER, Yarn-JobHistory Server spark_on_yarn-SPARK_HISTORY_SERVER
HostName: S1 Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz  Memory:512G	hdfs-DATANODE, hdfs-JOURNALNODE, zookeeper-ZOOKEEPER_SERVER, hbase-REGION_SERVER, yarn-NODEMANAGER
HostName: T1 Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz  Memory:512G	hdfs-DATANODE, hdfs-JOURNALNODE, zookeeper-ZOOKEEPER_SERVER, hbase-REGION_SERVER, yarn-NODEMANAGER

[Consult]

You may choose to include following command line option, in your conf "Java Configuration Options for ResourceManager"

-XX:PermSize=1024m -XX:MaxPermSize=2048m -Xms1024m -Xmx2048m

Please adapt the values as per your environment and test it in your test environment.

[ISLAND]

Thank you for your reply~

For adjusting jvm parameters, But,I have a question for currently operating resources , currently jvm use nearly 4GB ,therefore, adjusting the xmx parameter will make RM out of memory?

Can you explain why this is done by modifying this about jvm parameter?

[Consult · Accepted solution]

In general, GC is proportional to in-adequate heap size. It is depends on type of workload. You can review heap usage ("select jvm_max_memory_mb, jvm_heap_used_mb") for the impacted service and it should give you a good idea on the usage.

As I am not sure about your machine configs, but if your server has the capacity then then increasing heap size will help you.

[ISLAND]

Thank you for your explanation, I will try to adjust jvm heap size, the follow-up situation is there any improvement.