question Re: Yarn Resource Manager Repeated garbage collection in Support Questions

Yarn Resource Manager Repeated garbage collection

ISLAND — Fri, 16 Sep 2022 12:41:49 GMT

I have a cluster.

Recently, it often happens "RESOURCE_MANAGER_GC_DURATION concerning".

Observe the GC Collection Time case, each time it lasts for about 12s to 18s duration.

JVM Heap size configuration situation 4GB, the overall usage of about 92%, in a high water situation.

Please help to give suggestions.

The clusters of following specifications:

CDH version:5.7.0

Node info

service

HostName: P1

Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz

Memory:256G

cloudera-mgmt-HOSTMONITOR,

cloudera-mgmt-EVENTSERVER,

cloudera-mgmt-SERVICEMONITOR,

cloudera-mgmt-ALERTPUBLISHER,

cloudera-mgmt-ACITVITYMONITOR

postgresql,

hdfs-FAILOVERCONTROLLER,

hdfs-NAMENODE,

hbase-MASTER,

yarn-RESOURCEMANAGER,

yarn-NODEMANAGER

zookeeper-ZOOKEEPER_SERVER

HostName: Q1

Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz

Memory:256G

hdfs-FAILOVERCONTROLLER,

hdfs-NAMENODE,

hdfs-HTTPFS,

hbase-MASTER,

yarn-RESOURCEMANAGER,

yarn-NODEMANAGER,

oozie-OOZIE_SERVER

zookeeper-ZOOKEEPER_SERVER

HostName: R1

Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz

Memory:512G

hdfs-DATANODE,

hdfs-JOURNALNODE,

zookeeper-ZOOKEEPER_SERVER,

hbase-REGION_SERVER,

Hive-HIVE Metastore server

Hive-HiveServer2

yarn-NODEMANAGER,

Yarn-JobHistory Server

spark_on_yarn-SPARK_HISTORY_SERVER

HostName: S1

Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz

Memory:512G

hdfs-DATANODE,

hdfs-JOURNALNODE,

zookeeper-ZOOKEEPER_SERVER,

hbase-REGION_SERVER,

yarn-NODEMANAGER

HostName: T1

Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz

Memory:512G

hdfs-DATANODE,

hdfs-JOURNALNODE,

zookeeper-ZOOKEEPER_SERVER,

hbase-REGION_SERVER,

yarn-NODEMANAGER

Re: Yarn Resource Manager Repeated garbage collection

Consult — Tue, 09 Jan 2018 08:13:17 GMT

You may choose to include following command line option, in your conf "Java Configuration Options for ResourceManager"

-XX:PermSize=1024m -XX:MaxPermSize=2048m -Xms1024m -Xmx2048m

Please adapt the values as per your environment and test it in your test environment.

Re: Yarn Resource Manager Repeated garbage collection

ISLAND — Tue, 09 Jan 2018 08:33:00 GMT

Thank you for your reply~

For adjusting jvm parameters, But,I have a question for currently operating resources , currently jvm use nearly 4GB ,therefore, adjusting the xmx parameter will make RM out of memory?

Can you explain why this is done by modifying this about jvm parameter?

Re: Yarn Resource Manager Repeated garbage collection

Consult — Thu, 11 Jan 2018 09:23:42 GMT

In general, GC is proportional to in-adequate heap size. It is depends on type of workload. You can review heap usage ("select jvm_max_memory_mb, jvm_heap_used_mb") for the impacted service and it should give you a good idea on the usage.

As I am not sure about your machine configs, but if your server has the capacity then then increasing heap size will help you.

Re: Yarn Resource Manager Repeated garbage collection

ISLAND — Fri, 12 Jan 2018 01:57:38 GMT

Thank you for your explanation, I will try to adjust jvm heap size, the follow-up situation is there any improvement.