Support Questions
Find answers, ask questions, and share your expertise

HiveServer2 JVM crashed when G1GC used

New Contributor

We run HiveServer2 on Java 1.8.0u151 with G1GC.

Sometimes, HiveServer2 is shut down because JVM is crashed.

The reason is described as follows:

raise, /lib64/libc.so.6
abort, /lib64/libc.so.6
os::abort(bool), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
VMError::report_and_die(), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
JVM_handle_linux_signal, /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
signalHandler(int, siginfo*, void*), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
[NULL], /lib64/libpthread.so.0
InstanceKlass::oop_oop_iterate_backwards_nv(oopDesc*, G1ParScanClosure*), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
InstanceMirrorKlass::oop_oop_iterate_backwards_nv(oopDesc*, G1ParScanClosure*), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
G1ParScanThreadState::copy_to_survivor_space(InCSetState, oopDesc*, markOopDesc*), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
G1ParCopyClosure<(G1Barrier)2, (G1Mark)0>::do_oop(oopDesc**), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
G1KlassScanClosure::do_klass(Klass*), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
ClassLoaderData::oops_do(OopClosure*, KlassClosure*, bool), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
ClassLoaderDataGraph::roots_cld_do(CLDClosure*, CLDClosure*), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
G1RootProcessor::process_java_roots(OopClosure*, CLDClosure*, CLDClosure*, CLDClosure*, CodeBlobClosure*, G1GCPhaseTimes*, unsigned int), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
G1RootProcessor::evacuate_roots(OopClosure*, OopClosure*, CLDClosure*, CLDClosure*, bool, unsigned int), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
G1ParTask::work(unsigned int), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
GangWorker::loop(), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
java_start(Thread*), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
start_thread, /lib64/libpthread.so.0
__clone, /lib64/libc.so.6"

Does anyone help us? I couldn't find the reported bug in Oracle's Java Bug Database.

2 REPLIES 2

Re: HiveServer2 JVM crashed when G1GC used

Hi, All

I am Jaehyung's colleague.

The core dumps are as follows:

CoreDump #1

Stack: [0x00007f43dabec000,0x00007f43daced000], sp=0x00007f43dace7240, free space=1004k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x651649] InstanceKlass::oop_oop_iterate_backwards_nv(oopDesc*, G1ParScanClosure*)+0xc9
V [libjvm.so+0x661140] InstanceMirrorKlass::oop_oop_iterate_backwards_nv(oopDesc*, G1ParScanClosure*)+0x20
V [libjvm.so+0x5c7845] G1ParScanThreadState::copy_to_survivor_space(InCSetState, oopDesc*, markOopDesc*)+0x4c5
V [libjvm.so+0x5aa7b1] G1ParCopyClosure<(G1Barrier)2, (G1Mark)0>::do_oop(oopDesc**)+0x51
V [libjvm.so+0x5aa0e4] G1KlassScanClosure::do_klass(Klass*)+0x34
V [libjvm.so+0x46bc7c] ClassLoaderData::oops_do(OopClosure*, KlassClosure*, bool)+0x8c
V [libjvm.so+0x46afc8] ClassLoaderDataGraph::roots_cld_do(CLDClosure*, CLDClosure*)+0x38
V [libjvm.so+0x5ce4de] G1RootProcessor::process_java_roots(OopClosure*, CLDClosure*, CLDClosure*, CLDClosure*, CodeBlobClosure*, G1GCPhaseTimes*, unsigned int)+0x6e
V [libjvm.so+0x5ced81] G1RootProcessor::evacuate_roots(OopClosure*, OopClosure*, CLDClosure*, CLDClosure*, bool, unsigned int)+0x561
V [libjvm.so+0x5ae1a8] G1ParTask::work(unsigned int)+0x3b8
V [libjvm.so+0xaed0ff] GangWorker::loop()+0xcf
V [libjvm.so+0x92a728] java_start(Thread*)+0x108

Heap:
garbage-first heap total 67108864K, used 41487682K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 682 young (22347776K), 17 survivors (557056K)
Metaspace used 144974K, capacity 149252K, committed 149440K, reserved 149504K

GC Heap History (10 events):
Event: 1095307.920 GC heap after
Heap after GC invocations=9809 (full 0):
garbage-first heap total 67108864K, used 19507107K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 12 young (393216K), 12 survivors (393216K)
Metaspace used 144921K, capacity 149223K, committed 149440K, reserved 149504K
}
Event: 1095337.303 GC heap before
{Heap before GC invocations=9809 (full 0):
garbage-first heap total 67108864K, used 41461667K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 682 young (22347776K), 12 survivors (393216K)
Metaspace used 144921K, capacity 149223K, committed 149440K, reserved 149504K
Event: 1095337.411 GC heap after
Heap after GC invocations=9810 (full 0):
garbage-first heap total 67108864K, used 19527369K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 13 young (425984K), 13 survivors (425984K)
Metaspace used 144921K, capacity 149223K, committed 149440K, reserved 149504K
}
Event: 1095368.891 GC heap before
{Heap before GC invocations=9810 (full 0):
garbage-first heap total 67108864K, used 41449161K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 682 young (22347776K), 13 survivors (425984K)
Metaspace used 144921K, capacity 149223K, committed 149440K, reserved 149504K
Event: 1095368.988 GC heap after
Heap after GC invocations=9811 (full 0):
garbage-first heap total 67108864K, used 19582126K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 14 young (458752K), 14 survivors (458752K)
Metaspace used 144921K, capacity 149223K, committed 149440K, reserved 149504K
}
Event: 1095559.738 GC heap before
{Heap before GC invocations=9811 (full 0):
garbage-first heap total 67108864K, used 41471150K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 682 young (22347776K), 14 survivors (458752K)
Metaspace used 144921K, capacity 149223K, committed 149440K, reserved 149504K
Event: 1095559.885 GC heap after
Heap after GC invocations=9812 (full 0):
garbage-first heap total 67108864K, used 19671570K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 17 young (557056K), 17 survivors (557056K)
Metaspace used 144921K, capacity 149223K, committed 149440K, reserved 149504K
}
Event: 1095746.491 GC heap before
{Heap before GC invocations=9812 (full 0):
garbage-first heap total 67108864K, used 41462290K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 682 young (22347776K), 17 survivors (557056K)
Metaspace used 144938K, capacity 149244K, committed 149440K, reserved 149504K
Event: 1095746.629 GC heap after
Heap after GC invocations=9813 (full 0):
garbage-first heap total 67108864K, used 19696962K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 17 young (557056K), 17 survivors (557056K)
Metaspace used 144938K, capacity 149244K, committed 149440K, reserved 149504K
}
Event: 1095881.378 GC heap before
{Heap before GC invocations=9813 (full 0):
garbage-first heap total 67108864K, used 41487682K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 682 young (22347776K), 17 survivors (557056K)
Metaspace used 144974K, capacity 149252K, committed 149440K, reserved 149504K

--------

CoreDump #2

Stack: [0x00007f7aa0973000,0x00007f7aa0a74000], sp=0x00007f7aa0a6e1c0, free space=1004k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x651649] InstanceKlass::oop_oop_iterate_backwards_nv(oopDesc*, G1ParScanClosure*)+0xc9
V [libjvm.so+0x661140] InstanceMirrorKlass::oop_oop_iterate_backwards_nv(oopDesc*, G1ParScanClosure*)+0x20
V [libjvm.so+0x5c7845] G1ParScanThreadState::copy_to_survivor_space(InCSetState, oopDesc*, markOopDesc*)+0x4c5
V [libjvm.so+0x5aa7b1] G1ParCopyClosure<(G1Barrier)2, (G1Mark)0>::do_oop(oopDesc**)+0x51
V [libjvm.so+0x5aa0e4] G1KlassScanClosure::do_klass(Klass*)+0x34
V [libjvm.so+0x46bc7c] ClassLoaderData::oops_do(OopClosure*, KlassClosure*, bool)+0x8c
V [libjvm.so+0x46afc8] ClassLoaderDataGraph::roots_cld_do(CLDClosure*, CLDClosure*)+0x38
V [libjvm.so+0x5ce4de] G1RootProcessor::process_java_roots(OopClosure*, CLDClosure*, CLDClosure*, CLDClosure*, CodeBlobClosure*, G1GCPhaseTimes*, unsigned int)+0x6e
V [libjvm.so+0x5ced81] G1RootProcessor::evacuate_roots(OopClosure*, OopClosure*, CLDClosure*, CLDClosure*, bool, unsigned int)+0x561
V [libjvm.so+0x5ae1a8] G1ParTask::work(unsigned int)+0x3b8
V [libjvm.so+0xaed0ff] GangWorker::loop()+0xcf
V [libjvm.so+0x92a728] java_start(Thread*)+0x108

Heap:
garbage-first heap total 67108864K, used 44362593K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 682 young (22347776K), 5 survivors (163840K)
Metaspace used 150062K, capacity 155003K, committed 155360K, reserved 155648K

GC Heap History (10 events):
Event: 2031681.264 GC heap after
Heap after GC invocations=15747 (full 0):
garbage-first heap total 67108864K, used 22165995K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 4 young (131072K), 4 survivors (131072K)
Metaspace used 150034K, capacity 155003K, committed 155360K, reserved 155648K
}
Event: 2031720.454 GC heap before
{Heap before GC invocations=15747 (full 0):
garbage-first heap total 67108864K, used 44382699K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 682 young (22347776K), 4 survivors (131072K)
Metaspace used 150034K, capacity 155003K, committed 155360K, reserved 155648K
Event: 2031720.538 GC heap after
Heap after GC invocations=15748 (full 0):
garbage-first heap total 67108864K, used 22175438K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 5 young (163840K), 5 survivors (163840K)
Metaspace used 150034K, capacity 155003K, committed 155360K, reserved 155648K
}
Event: 2031754.375 GC heap before
{Heap before GC invocations=15748 (full 0):
garbage-first heap total 67108864K, used 44359374K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 682 young (22347776K), 5 survivors (163840K)
Metaspace used 150034K, capacity 155003K, committed 155360K, reserved 155648K
Event: 2031754.461 GC heap after
Heap after GC invocations=15749 (full 0):
garbage-first heap total 67108864K, used 22158646K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 4 young (131072K), 4 survivors (131072K)
Metaspace used 150034K, capacity 155003K, committed 155360K, reserved 155648K
}
Event: 2031795.794 GC heap before
{Heap before GC invocations=15749 (full 0):
garbage-first heap total 67108864K, used 44375350K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 682 young (22347776K), 4 survivors (131072K)
Metaspace used 150034K, capacity 155003K, committed 155360K, reserved 155648K
Event: 2031795.892 GC heap after
Heap after GC invocations=15750 (full 0):
garbage-first heap total 67108864K, used 22165753K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 4 young (131072K), 4 survivors (131072K)
Metaspace used 150034K, capacity 155003K, committed 155360K, reserved 155648K
}
Event: 2031844.619 GC heap before
{Heap before GC invocations=15750 (full 0):
garbage-first heap total 67108864K, used 44382457K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 682 young (22347776K), 4 survivors (131072K)
Metaspace used 150034K, capacity 155003K, committed 155360K, reserved 155648K
Event: 2031844.716 GC heap after
Heap after GC invocations=15751 (full 0):
garbage-first heap total 67108864K, used 22178657K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 5 young (163840K), 5 survivors (163840K)
Metaspace used 150034K, capacity 155003K, committed 155360K, reserved 155648K
}
Event: 2031889.871 GC heap before
{Heap before GC invocations=15751 (full 0):
garbage-first heap total 67108864K, used 44362593K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 682 young (22347776K), 5 survivors (163840K)

Metaspace used 150062K, capacity 155003K, committed 155360K, reserved 155648K

Re: HiveServer2 JVM crashed when G1GC used

and jvm arguments are as follows;

VM Arguments:
jvm_args: -Xmx1024m -Dhdp.version=2.6.3.0-235 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.3.0-235 -Dhadoop.log.dir=/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.3.0-235/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.3.0-235/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.6.3.0-235/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.6.3.0-235/hadoop/lib... -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:MaxGCPauseMillis=1000 -XX:InitiatingHeapOccupancyPercent=45 -XX:NewRatio=2 -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=15 -XX:G1ReservePercent=10 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -hiveconf hive.metastore.uris= -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/log/hive