How/When to take thread dump (Jstack) and heap dumps (jmap)

[#1]

JSTACK and JMAP Collection

Jstack Collection

Step 1: Switch as the service user that started the process.

#su - <service-user-who-started-the-process>

Step 2: Capture the process ID

#ps -ef | grep <process-name>

#ps -ef | grep hive

hive     21887     1  0 Aug01 ?        00:58:04 /usr/jdk64/jdk1.8.0_112/bin/java -Xmx1024m -Dhdp.version=2.6.5.0-292 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.5.0-292 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.5.0-292/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.5.0-292/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.6.5.0-292/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.6.5.0-292/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx2048m -Djava.util.logging.config.file=/usr/hdp/current/hive-server2/conf/conf.server/parquet-logging.properties -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.6.5.0-292/hive/lib/hive-service-1.2.1000.2.6.5.0-292.jar org.apache.hive.service.server.HiveServer2 --hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -hiveconf hive.metastore.uris=  -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/var/log/hive

From above output,

Parent service account is hive

Process ID is 21887

Java version used is /usr/jdk64/jdk1.8.0_112/bin/java

Step 3: Capture the java used by the process to start the service.

From the above output it is /usr/jdk64/jdk1.8.0_112/bin/java

Step 4: (In the order of priority)

NOTE: We would need to consider running the command multiple times (min 5 times ) separated by 20-30 seconds.

4.1: Simple jstack for a responding process

#<jstack-used-by-process>/jstack -l <pid> > <location-to-redirect-the-output>/jstack.out

4.2: Use kill for a hung process

#kill -3 <pid>

Corresponding output is captured in .out file of the process.

4.3: Use -F for a hung process

#<jstack-used-by-process>/jstack -F <pid> > <location-to-redirect-the-output>/jstack.out

JMap Collection

Step 1: #su - <service-user-who-started-the-process>

Step 2: Capture the process ID

Step 3: Capture the java used by the process to start the service.

Step 4: Determining the appropriate flag to be used,

We use "-heap" option to determine if it is needed to use "-dump" option.

#<jmap-used-by-process>/jmap -heap <pid> > jmapHEAP.out

Upon multiple executions of the above command, if the percentage used is above 90% then we use the -dump flag as below,

Sample output of above command is,

Attaching to process ID 21887, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.112-b15

using thread-local object allocation.
Parallel GC with 8 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 0
   MaxHeapFreeRatio         = 100
   MaxHeapSize              = 2147483648 (2048.0MB)
   NewSize                  = 87031808 (83.0MB)
   MaxNewSize               = 715653120 (682.5MB)
   OldSize                  = 175112192 (167.0MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 0 (0.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 141557760 (135.0MB)
   used     = 36859416 (35.151878356933594MB)
   free     = 104698344 (99.8481216430664MB)
   26.038428412543404% used
From Space:
   capacity = 5767168 (5.5MB)
   used     = 4211840 (4.0167236328125MB)
   free     = 1555328 (1.4832763671875MB)
   73.0313387784091% used
To Space:
   capacity = 5767168 (5.5MB)
   used     = 0 (0.0MB)
   free     = 5767168 (5.5MB)
   0.0% used
PS Old Generation
   capacity = 277872640 (265.0MB)
   used     = 161075720 (153.61377716064453MB)
   free     = 116796920 (111.38622283935547MB)
   57.9674630794885% used

From the above output, 57% of heap is being used.

The two general flags that are used while collecting the heapdumps are “-dump” and “-histo”.

While former gives the heapdump in the form of binary file with the collection of objects at a particular time, latter provides the details of live objects in a text format.

#<jmap-used-by-process>/jmap -dump:file=<location-to-redirect-the-output>/heapdump.hprof,format=b <PID>

If histo label needs to be used,

#<jmap-used-by-process>/jmap -histo <pid> > jmap.out

NOTE:

1. Jmap/Jstack is high CPU intensive process, so please use it with caution.

2. Please try not to use -F as much as possible as critical data are missed with this option. If -F option needs to be used by any of the commands,

Example:

#/usr/jdk64/jdk1.8.0_112/bin/jmap -dump:file=/tmp/jmap21887.hprof,format=b -F 21887

#/usr/jdk64/jdk1.8.0_112/bin/jmap -histo -F 21887 > /tmp/jmaphistoF.out

Thanks @Rajkumar Singh @Vinod Bonthu and @Kevin Wong for reviewing.