Reply
New Contributor
Posts: 5
Registered: ‎01-10-2017
Accepted Solution

Yarn Resource Manager Repeated garbage collection

I have a cluster.

Recently, it often happens "RESOURCE_MANAGER_GC_DURATION concerning".

Observe the GC Collection Time case, each time it lasts for about 12s to 18s duration.

JVM Heap size configuration situation 4GB, the overall usage of about 92%, in a high water situation.

Please help to give suggestions.

 

The clusters of  following specifications:

CDH version:5.7.0

 

Node info

service

HostName: P1

Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz

 

Memory:256G

cloudera-mgmt-HOSTMONITOR,

cloudera-mgmt-EVENTSERVER,

cloudera-mgmt-SERVICEMONITOR,

cloudera-mgmt-ALERTPUBLISHER,

cloudera-mgmt-ACITVITYMONITOR

postgresql,

hdfs-FAILOVERCONTROLLER,

hdfs-NAMENODE,

hbase-MASTER,

yarn-RESOURCEMANAGER,

yarn-NODEMANAGER

zookeeper-ZOOKEEPER_SERVER

HostName: Q1

Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz

 

Memory:256G

hdfs-FAILOVERCONTROLLER,

hdfs-NAMENODE,

hdfs-HTTPFS,

hbase-MASTER,

yarn-RESOURCEMANAGER,

yarn-NODEMANAGER,

oozie-OOZIE_SERVER

zookeeper-ZOOKEEPER_SERVER

HostName: R1

Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz 

 

Memory:512G

hdfs-DATANODE,

hdfs-JOURNALNODE,

zookeeper-ZOOKEEPER_SERVER,

hbase-REGION_SERVER,

Hive-HIVE Metastore server

Hive-HiveServer2

yarn-NODEMANAGER,

Yarn-JobHistory Server

spark_on_yarn-SPARK_HISTORY_SERVER

HostName: S1

Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz 

 

Memory:512G

hdfs-DATANODE,

hdfs-JOURNALNODE,

zookeeper-ZOOKEEPER_SERVER,

hbase-REGION_SERVER,

yarn-NODEMANAGER

 

HostName: T1

Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz 

 

Memory:512G

hdfs-DATANODE,

hdfs-JOURNALNODE,

zookeeper-ZOOKEEPER_SERVER,

hbase-REGION_SERVER,

yarn-NODEMANAGER

 

 

 

 

 

擷取選取區域_197.png

擷取選取區域_198.png

擷取選取區域_199.png

 

 

 

 

 

 

 

 

Contributor
Posts: 52
Registered: ‎01-08-2016

Re: Yarn Resource Manager Repeated garbage collection

You may choose to include following command line option, in your conf "Java Configuration Options for ResourceManager"

 

-XX:PermSize=1024m -XX:MaxPermSize=2048m -Xms1024m -Xmx2048m

 

Please adapt the values as per your environment and test it in your test environment.

Highlighted
New Contributor
Posts: 5
Registered: ‎01-10-2017

Re: Yarn Resource Manager Repeated garbage collection

Thank you for your reply~

For adjusting jvm parameters, But,I have a question for currently operating resources , currently jvm use nearly 4GB ,therefore, adjusting the xmx parameter will make RM out of memory?

Can you explain why this is done by modifying this about jvm parameter?

Contributor
Posts: 52
Registered: ‎01-08-2016

Re: Yarn Resource Manager Repeated garbage collection

In general, GC is proportional to in-adequate heap size. It is depends on type of workload. You can review heap usage ("select jvm_max_memory_mb, jvm_heap_used_mb") for the impacted service and it should give you a good idea on the usage.

 

As I am not sure about your machine configs, but if your server has the capacity then then increasing heap size will help you.

New Contributor
Posts: 5
Registered: ‎01-10-2017

Re: Yarn Resource Manager Repeated garbage collection

Thank you for your explanation, I will try to adjust jvm heap size, the follow-up situation is there any improvement.

Announcements