Created on 08-30-2018 12:27 AM - edited 09-16-2022 01:44 AM
Step 1: Switch as the service user that started the process.
#su - <service-user-who-started-the-process>
Step 2: Capture the process ID
#ps -ef | grep <process-name>
#ps -ef | grep hive hive 21887 1 0 Aug01 ? 00:58:04 /usr/jdk64/jdk1.8.0_112/bin/java -Xmx1024m -Dhdp.version=2.6.5.0-292 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.5.0-292 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.5.0-292/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.5.0-292/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.6.5.0-292/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.6.5.0-292/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx2048m -Djava.util.logging.config.file=/usr/hdp/current/hive-server2/conf/conf.server/parquet-logging.properties -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.6.5.0-292/hive/lib/hive-service-1.2.1000.2.6.5.0-292.jar org.apache.hive.service.server.HiveServer2 --hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -hiveconf hive.metastore.uris= -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/var/log/hive
From above output,
Parent service account is hive
Process ID is 21887
Java version used is /usr/jdk64/jdk1.8.0_112/bin/java
Step 3: Capture the java used by the process to start the service.
From the above output it is /usr/jdk64/jdk1.8.0_112/bin/java
Step 4: (In the order of priority)
NOTE: We would need to consider running the command multiple times (min 5 times ) separated by 20-30 seconds.
4.1: Simple jstack for a responding process
#<jstack-used-by-process>/jstack -l <pid> > <location-to-redirect-the-output>/jstack.out
4.2: Use kill for a hung process
#kill -3 <pid>
Corresponding output is captured in .out file of the process.
4.3: Use -F for a hung process
#<jstack-used-by-process>/jstack -F <pid> > <location-to-redirect-the-output>/jstack.out
Step 1: #su - <service-user-who-started-the-process>
Step 2: Capture the process ID
Step 3: Capture the java used by the process to start the service.
Step 4: Determining the appropriate flag to be used,
We use "-heap" option to determine if it is needed to use "-dump" option.
#<jmap-used-by-process>/jmap -heap <pid> > jmapHEAP.out
Upon multiple executions of the above command, if the percentage used is above 90% then we use the -dump flag as below,
Sample output of above command is,
Attaching to process ID 21887, please wait... Debugger attached successfully. Server compiler detected. JVM version is 25.112-b15 using thread-local object allocation. Parallel GC with 8 thread(s) Heap Configuration: MinHeapFreeRatio = 0 MaxHeapFreeRatio = 100 MaxHeapSize = 2147483648 (2048.0MB) NewSize = 87031808 (83.0MB) MaxNewSize = 715653120 (682.5MB) OldSize = 175112192 (167.0MB) NewRatio = 2 SurvivorRatio = 8 MetaspaceSize = 21807104 (20.796875MB) CompressedClassSpaceSize = 1073741824 (1024.0MB) MaxMetaspaceSize = 17592186044415 MB G1HeapRegionSize = 0 (0.0MB) Heap Usage: PS Young Generation Eden Space: capacity = 141557760 (135.0MB) used = 36859416 (35.151878356933594MB) free = 104698344 (99.8481216430664MB) 26.038428412543404% used From Space: capacity = 5767168 (5.5MB) used = 4211840 (4.0167236328125MB) free = 1555328 (1.4832763671875MB) 73.0313387784091% used To Space: capacity = 5767168 (5.5MB) used = 0 (0.0MB) free = 5767168 (5.5MB) 0.0% used PS Old Generation capacity = 277872640 (265.0MB) used = 161075720 (153.61377716064453MB) free = 116796920 (111.38622283935547MB) 57.9674630794885% used
From the above output, 57% of heap is being used.
The two general flags that are used while collecting the heapdumps are “-dump” and “-histo”.
While former gives the heapdump in the form of binary file with the collection of objects at a particular time, latter provides the details of live objects in a text format.
#<jmap-used-by-process>/jmap -dump:file=<location-to-redirect-the-output>/heapdump.hprof,format=b <PID>
If histo label needs to be used,
#<jmap-used-by-process>/jmap -histo <pid> > jmap.out
NOTE:
1. Jmap/Jstack is high CPU intensive process, so please use it with caution.
2. Please try not to use -F as much as possible as critical data are missed with this option. If -F option needs to be used by any of the commands,
Example:
#/usr/jdk64/jdk1.8.0_112/bin/jmap -dump:file=/tmp/jmap21887.hprof,format=b -F 21887
#/usr/jdk64/jdk1.8.0_112/bin/jmap -histo -F 21887 > /tmp/jmaphistoF.out
Thanks @Rajkumar Singh @Vinod Bonthu and @Kevin Wong for reviewing.