Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Yarn Resource Manager Repeated garbage collection

Solved Go to solution

Yarn Resource Manager Repeated garbage collection

New Contributor

I have a cluster.

Recently, it often happens "RESOURCE_MANAGER_GC_DURATION concerning".

Observe the GC Collection Time case, each time it lasts for about 12s to 18s duration.

JVM Heap size configuration situation 4GB, the overall usage of about 92%, in a high water situation.

Please help to give suggestions.

 

The clusters of  following specifications:

CDH version:5.7.0

 

Node info

service

HostName: P1

Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz

 

Memory:256G

cloudera-mgmt-HOSTMONITOR,

cloudera-mgmt-EVENTSERVER,

cloudera-mgmt-SERVICEMONITOR,

cloudera-mgmt-ALERTPUBLISHER,

cloudera-mgmt-ACITVITYMONITOR

postgresql,

hdfs-FAILOVERCONTROLLER,

hdfs-NAMENODE,

hbase-MASTER,

yarn-RESOURCEMANAGER,

yarn-NODEMANAGER

zookeeper-ZOOKEEPER_SERVER

HostName: Q1

Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz

 

Memory:256G

hdfs-FAILOVERCONTROLLER,

hdfs-NAMENODE,

hdfs-HTTPFS,

hbase-MASTER,

yarn-RESOURCEMANAGER,

yarn-NODEMANAGER,

oozie-OOZIE_SERVER

zookeeper-ZOOKEEPER_SERVER

HostName: R1

Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz 

 

Memory:512G

hdfs-DATANODE,

hdfs-JOURNALNODE,

zookeeper-ZOOKEEPER_SERVER,

hbase-REGION_SERVER,

Hive-HIVE Metastore server

Hive-HiveServer2

yarn-NODEMANAGER,

Yarn-JobHistory Server

spark_on_yarn-SPARK_HISTORY_SERVER

HostName: S1

Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz 

 

Memory:512G

hdfs-DATANODE,

hdfs-JOURNALNODE,

zookeeper-ZOOKEEPER_SERVER,

hbase-REGION_SERVER,

yarn-NODEMANAGER

 

HostName: T1

Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz 

 

Memory:512G

hdfs-DATANODE,

hdfs-JOURNALNODE,

zookeeper-ZOOKEEPER_SERVER,

hbase-REGION_SERVER,

yarn-NODEMANAGER

 

 

 

 

 

擷取選取區域_197.png

擷取選取區域_198.png

擷取選取區域_199.png

 

 

 

 

 

 

 

 

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Yarn Resource Manager Repeated garbage collection

Expert Contributor

In general, GC is proportional to in-adequate heap size. It is depends on type of workload. You can review heap usage ("select jvm_max_memory_mb, jvm_heap_used_mb") for the impacted service and it should give you a good idea on the usage.

 

As I am not sure about your machine configs, but if your server has the capacity then then increasing heap size will help you.

4 REPLIES 4

Re: Yarn Resource Manager Repeated garbage collection

Expert Contributor

You may choose to include following command line option, in your conf "Java Configuration Options for ResourceManager"

 

-XX:PermSize=1024m -XX:MaxPermSize=2048m -Xms1024m -Xmx2048m

 

Please adapt the values as per your environment and test it in your test environment.

Re: Yarn Resource Manager Repeated garbage collection

New Contributor

Thank you for your reply~

For adjusting jvm parameters, But,I have a question for currently operating resources , currently jvm use nearly 4GB ,therefore, adjusting the xmx parameter will make RM out of memory?

Can you explain why this is done by modifying this about jvm parameter?

Highlighted

Re: Yarn Resource Manager Repeated garbage collection

Expert Contributor

In general, GC is proportional to in-adequate heap size. It is depends on type of workload. You can review heap usage ("select jvm_max_memory_mb, jvm_heap_used_mb") for the impacted service and it should give you a good idea on the usage.

 

As I am not sure about your machine configs, but if your server has the capacity then then increasing heap size will help you.

Re: Yarn Resource Manager Repeated garbage collection

New Contributor

Thank you for your explanation, I will try to adjust jvm heap size, the follow-up situation is there any improvement.