New method to capturing Cloudera Manager (CM 7.x+) performance profile information

[#1]

Introduction

The new method of capturing Cloudera Manager performance profile utilizes the Java Flight Recorder (JFR) feature of JVM. (Running Java Flight Recorder) 

JFR works on OpenJDK 11 and higher,  also OpenJDK 8u262 and higher.  For older JDK8, use the older method for capturing performance profiles. (Steps to collect Cloudera Manager profiling data to troubleshoot performance issues - you need subscription to access this article)

The performance capture files can be analyzed with the JMC tool - JDK Mission Control (JMC) 

Pre-requisites

Ensure recent JDK

# /usr/java/default/bin/java -version
openjdk version "1.8.0_232"

• OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_232-b09)
• OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.232-b09, mixed mode)
• JDK 8u232 is not recent enough to support JFR.  Will not work.  Must be OpenJDK 8u262 or higher.

—---

# /usr/java/default/bin/java -version
openjdk version "11.0.3" 2019-04-16 LTS

• OpenJDK Runtime Environment Zulu11.31+11-CA (build 11.0.3+7-LTS)
• OpenJDK 64-Bit Server VM Zulu11.31+11-CA (build 11.0.3+7-LTS, mixed mode)
• Any JDK 11 should work.

Ensure jcmd tool

This procedure uses the jcmd command line tool. This tool is shipped with JDK devel packages.  Please ensure the appropriate JDK devel package is installed on the CM Server host. For example, if using the java-11-openjdk package, ensure that java-11-openjdk-devel is installed. The jcmd tool is located in the same directory as the java executable.  Ensure that the jcmd tool matches the java command used to run the CM Server.

Oracle JDK

1. When using Oracle JDK, it may be necessary to add the following flag to the Java command line of CM server.  

   -XX:+UnlockCommercialFeatures 

2. Check the Oracle JDK docs for Java Flight Recorder to see if this is necessary.
3. Edit the file /etc/default/cloudera-scm-server and add the flag to the CMF_JAVA_OPTS line near the end of the file. 
4. Then restart the CM server via

   # systemctl restart cloudera-scm-server

           This flag is not needed for OpenJDK.

Capture Steps

Find PID for CM server

# /usr/java/default/bin/jcmd | grep cmf.Main

17444 com.cloudera.server.cmf.Main

Built-in help for jcmd tool

This section is informational only, and can be skipped.

# /usr/java/default/bin/jcmd 17444 help

17444:

The following commands are available:

 . . .

JFR.check

JFR.configure

JFR.dump

JFR.start

JFR.stop

 . . .

VM.version

help

For more information about a specific command use 'help <command>'.

# /usr/java/default/bin/jcmd 17444 help JFR.start
. . .

Possible settings are found here:

# ls /usr/java/default/lib/jfr/

default.jfc  profile.jfc

Capturing performance profile

To start a performance capture for 10 minutes.  [Substitute the correct PID for the running CM server process].

# /usr/java/default/bin/jcmd 17444 JFR.configure stackdepth=128

17444:

Stack depth: 128

# /usr/java/default/bin/jcmd 17444 JFR.start name=cm-capture \ settings=profile duration=10m disk=true filename=/tmp/cm-profile.jfr path-to-gc-roots=true

17444:

Started recording 2. The result will be written to:

/tmp/cm-profile.jfr

Now perform whatever actions on Cloudera Manager that need to be profiled.  If the actions to be recorded take longer than 10 minutes, increase the “10m” duration value in the command above as needed.

NOTE: It will take about 15 minutes for the JVM to finalize the performance capture after the capture duration.  So if capture duration is 10 minutes, then it may take 10 + 15 = 25 minutes for the capture file to be written out.  During this time the capture file size will remain zero (0).

This process will create a performance capture file called cm-profile.jfr in /tmp directory.  A 10 minute capture could be 20MB to 40MB in size.

Recommend compressing the file using gzip.

# gzip /tmp/cm-profile.jfr

Upload the compressed file to Cloudera case ticket for analysis.

Check on a running capture

# /usr/java/default/bin/jcmd 17444 JFR.check

17444:

Recording 3: name=cm-capture duration=10m (running)

When the capture is completed, the result will be:

# /usr/java/default/bin/jcmd 17444 JFR.check

17444:

No available recordings.

Possible problems

Problem  1:

Could not start recording, not able to write to file /root/cm-profile.jfr

Solution: Target directory needs to be writable by cloudera-scm user (CM server running as that user).  Either change permissions/owner on the target directory, or use a different path for the capture file.

Problem 2:  jcmd tool not able to communicate with CM server JVM.

Solution 2: The jcmd tool does not have permissions to communicate with CM Server JVM running as cloudera-scm user.  Add the prefix “sudo -u cloudera-scm” user to the jcmd line above to switch to the correct user.