Optimize the Cloudera Manager Server

[#1]

Summary

Cloudera Manager is the best-in-class holistic interface that provides end-to-end cluster management and key enterprise features to provide granular visibility into and control over every part of an open data lakehouse. The optimization steps below complement Cloudera’s Optimize the Cloudera Manager Server page.

Investigation & Resolution

Monitor your Cloudera Manager Server Heap

We have provided a useful suite of Cloudera Manager dashboards in the blog Deploy your Cloudera Manager Dashboards. The dashboard called “MB - MGMT Cluster - JVM GC Sizing” includes a set of charts that focus on the Cloudera Manager Server heap which will enable you to easily visualize abnormal characteristics of the heap. This portion shows a 6hr time series set:

Tune your Cloudera Manager Server Heap

The Cloudera Manager Server must be configured from the command line within Cloudera Manager Server itself.

Log in as root to your Cloudera Manager Server:

mbush@mbush-MBP16 CDSW % ssh root@<CM SERVER FQDN>

Check the Cloudera Manager Server config file:

[root@<CM SERVER FQDN> ~]# cat /etc/default/cloudera-scm-server

#

# Specify any command line arguments for the Cloudera SCM Server here.

#

CMF_SERVER_ARGS=""

#

# Locate the JDBC driver jar file.

#

# The default value is the default system mysql driver on RHEL/CentOS/Ubuntu

# and the standard, documented location for where to put the oracle jar in CM

# deployments.

#

export CMF_JDBC_DRIVER_JAR="/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar"

#

# You can override JAVA_HOME here if your java is not on the normal search path

# export JAVA_HOME=/usr/java/default

#

# Java Options.

#

# Default value sets Java maximum heap size to 2GB, and Java maximum permanent

# generation size to 256MB.

#

export CMF_JAVA_OPTS="-Xmx4G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp"

The key line to control the tuning of the Cloudera Manager Server heap is highlighted in red.

Above, you can see the default settings that we get with any vanilla deployment of the Cloudera Manager service.  These parameters will need to be tuned as your Cloudera CDP cluster becomes larger and busier, and that tuning is particularly important when any serious use of the Cloudera Manager API is introduced.

We recommend that you suitably raise the overall heap size based on your cluster size and CM API needs, and disable adaptive heap sizing & control the JVM heap ratio.

This can be done by amending the CMF_JAVA_OPTS and restarting the CM Server:

export CMF_JAVA_OPTS="-Xms16G -Xmx16G -XX:NewRatio=2 -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp"

#RESTART THE CLOUDERA MANAGER SERVICE TO COMMIT THE AMENDMENT

[root@<CM SERVER FQDN> ~]# systemctl restart cloudera-scm-server

[#2]

The JDK 8 HotSpot JVM is now using native memory for the representation of class metadata and is called Metaspace.

The permanent generation has been removed. The PermSize and MaxPermSize are ignored and a warning is issued if they are present on the command line.