01-03-2018 06:44 AM
I have a cluster.
Recently, it often happens "RESOURCE_MANAGER_GC_DURATION concerning".
Observe the GC Collection Time case, each time it lasts for about 12s to 18s duration.
JVM Heap size configuration situation 4GB, the overall usage of about 92%, in a high water situation.
Please help to give suggestions.
The clusters of following specifications:
CDH version:5.7.0
Node info | service |
HostName: P1 Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
Memory:256G | cloudera-mgmt-HOSTMONITOR, cloudera-mgmt-EVENTSERVER, cloudera-mgmt-SERVICEMONITOR, cloudera-mgmt-ALERTPUBLISHER, cloudera-mgmt-ACITVITYMONITOR postgresql, hdfs-FAILOVERCONTROLLER, hdfs-NAMENODE, hbase-MASTER, yarn-RESOURCEMANAGER, yarn-NODEMANAGER zookeeper-ZOOKEEPER_SERVER |
HostName: Q1 Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
Memory:256G | hdfs-FAILOVERCONTROLLER, hdfs-NAMENODE, hdfs-HTTPFS, hbase-MASTER, yarn-RESOURCEMANAGER, yarn-NODEMANAGER, oozie-OOZIE_SERVER zookeeper-ZOOKEEPER_SERVER |
HostName: R1 Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
Memory:512G | hdfs-DATANODE, hdfs-JOURNALNODE, zookeeper-ZOOKEEPER_SERVER, hbase-REGION_SERVER, Hive-HIVE Metastore server Hive-HiveServer2 yarn-NODEMANAGER, Yarn-JobHistory Server spark_on_yarn-SPARK_HISTORY_SERVER |
HostName: S1 Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
Memory:512G | hdfs-DATANODE, hdfs-JOURNALNODE, zookeeper-ZOOKEEPER_SERVER, hbase-REGION_SERVER, yarn-NODEMANAGER
|
HostName: T1 Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
Memory:512G | hdfs-DATANODE, hdfs-JOURNALNODE, zookeeper-ZOOKEEPER_SERVER, hbase-REGION_SERVER, yarn-NODEMANAGER |
Solved! Go to Solution.
01-09-2018 12:13 AM
You may choose to include following command line option, in your conf "Java Configuration Options for ResourceManager"
-XX:PermSize=1024m -XX:MaxPermSize=2048m -Xms1024m -Xmx2048m
Please adapt the values as per your environment and test it in your test environment.
01-09-2018 12:33 AM
Thank you for your reply~
For adjusting jvm parameters, But,I have a question for currently operating resources , currently jvm use nearly 4GB ,therefore, adjusting the xmx parameter will make RM out of memory?
Can you explain why this is done by modifying this about jvm parameter?
01-11-2018 01:23 AM
In general, GC is proportional to in-adequate heap size. It is depends on type of workload. You can review heap usage ("select jvm_max_memory_mb, jvm_heap_used_mb") for the impacted service and it should give you a good idea on the usage.
As I am not sure about your machine configs, but if your server has the capacity then then increasing heap size will help you.
01-11-2018 05:57 PM
Thank you for your explanation, I will try to adjust jvm heap size, the follow-up situation is there any improvement.