HiveServer2 JVM crashed when G1GC used

[#1]

We run HiveServer2 on Java 1.8.0u151 with G1GC.

Sometimes, HiveServer2 is shut down because JVM is crashed.

The reason is described as follows:

raise, /lib64/libc.so.6
abort, /lib64/libc.so.6
os::abort(bool), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
VMError::report_and_die(), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
JVM_handle_linux_signal, /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
signalHandler(int, siginfo*, void*), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
[NULL], /lib64/libpthread.so.0
InstanceKlass::oop_oop_iterate_backwards_nv(oopDesc*, G1ParScanClosure*), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
InstanceMirrorKlass::oop_oop_iterate_backwards_nv(oopDesc*, G1ParScanClosure*), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
G1ParScanThreadState::copy_to_survivor_space(InCSetState, oopDesc*, markOopDesc*), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
G1ParCopyClosure<(G1Barrier)2, (G1Mark)0>::do_oop(oopDesc**), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
G1KlassScanClosure::do_klass(Klass*), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
ClassLoaderData::oops_do(OopClosure*, KlassClosure*, bool), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
ClassLoaderDataGraph::roots_cld_do(CLDClosure*, CLDClosure*), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
G1RootProcessor::process_java_roots(OopClosure*, CLDClosure*, CLDClosure*, CLDClosure*, CodeBlobClosure*, G1GCPhaseTimes*, unsigned int), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
G1RootProcessor::evacuate_roots(OopClosure*, OopClosure*, CLDClosure*, CLDClosure*, bool, unsigned int), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
G1ParTask::work(unsigned int), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
GangWorker::loop(), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
java_start(Thread*), /usr/java/jdk1.8.0_151/jre/lib/amd64/server/libjvm.so
start_thread, /lib64/libpthread.so.0
__clone, /lib64/libc.so.6"

Does anyone help us? I couldn't find the reported bug in Oracle's Java Bug Database.

[#2]

Hi, All

I am Jaehyung's colleague.

The core dumps are as follows:

CoreDump #1

Stack: [0x00007f43dabec000,0x00007f43daced000], sp=0x00007f43dace7240, free space=1004k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x651649] InstanceKlass::oop_oop_iterate_backwards_nv(oopDesc*, G1ParScanClosure*)+0xc9
V [libjvm.so+0x661140] InstanceMirrorKlass::oop_oop_iterate_backwards_nv(oopDesc*, G1ParScanClosure*)+0x20
V [libjvm.so+0x5c7845] G1ParScanThreadState::copy_to_survivor_space(InCSetState, oopDesc*, markOopDesc*)+0x4c5
V [libjvm.so+0x5aa7b1] G1ParCopyClosure<(G1Barrier)2, (G1Mark)0>::do_oop(oopDesc**)+0x51
V [libjvm.so+0x5aa0e4] G1KlassScanClosure::do_klass(Klass*)+0x34
V [libjvm.so+0x46bc7c] ClassLoaderData::oops_do(OopClosure*, KlassClosure*, bool)+0x8c
V [libjvm.so+0x46afc8] ClassLoaderDataGraph::roots_cld_do(CLDClosure*, CLDClosure*)+0x38
V [libjvm.so+0x5ce4de] G1RootProcessor::process_java_roots(OopClosure*, CLDClosure*, CLDClosure*, CLDClosure*, CodeBlobClosure*, G1GCPhaseTimes*, unsigned int)+0x6e
V [libjvm.so+0x5ced81] G1RootProcessor::evacuate_roots(OopClosure*, OopClosure*, CLDClosure*, CLDClosure*, bool, unsigned int)+0x561
V [libjvm.so+0x5ae1a8] G1ParTask::work(unsigned int)+0x3b8
V [libjvm.so+0xaed0ff] GangWorker::loop()+0xcf
V [libjvm.so+0x92a728] java_start(Thread*)+0x108

Heap:
garbage-first heap total 67108864K, used 41487682K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 682 young (22347776K), 17 survivors (557056K)
Metaspace used 144974K, capacity 149252K, committed 149440K, reserved 149504K

GC Heap History (10 events):
Event: 1095307.920 GC heap after
Heap after GC invocations=9809 (full 0):
garbage-first heap total 67108864K, used 19507107K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 12 young (393216K), 12 survivors (393216K)
Metaspace used 144921K, capacity 149223K, committed 149440K, reserved 149504K
}
Event: 1095337.303 GC heap before
{Heap before GC invocations=9809 (full 0):
garbage-first heap total 67108864K, used 41461667K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 682 young (22347776K), 12 survivors (393216K)
Metaspace used 144921K, capacity 149223K, committed 149440K, reserved 149504K
Event: 1095337.411 GC heap after
Heap after GC invocations=9810 (full 0):
garbage-first heap total 67108864K, used 19527369K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 13 young (425984K), 13 survivors (425984K)
Metaspace used 144921K, capacity 149223K, committed 149440K, reserved 149504K
}
Event: 1095368.891 GC heap before
{Heap before GC invocations=9810 (full 0):
garbage-first heap total 67108864K, used 41449161K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 682 young (22347776K), 13 survivors (425984K)
Metaspace used 144921K, capacity 149223K, committed 149440K, reserved 149504K
Event: 1095368.988 GC heap after
Heap after GC invocations=9811 (full 0):
garbage-first heap total 67108864K, used 19582126K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 14 young (458752K), 14 survivors (458752K)
Metaspace used 144921K, capacity 149223K, committed 149440K, reserved 149504K
}
Event: 1095559.738 GC heap before
{Heap before GC invocations=9811 (full 0):
garbage-first heap total 67108864K, used 41471150K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 682 young (22347776K), 14 survivors (458752K)
Metaspace used 144921K, capacity 149223K, committed 149440K, reserved 149504K
Event: 1095559.885 GC heap after
Heap after GC invocations=9812 (full 0):
garbage-first heap total 67108864K, used 19671570K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 17 young (557056K), 17 survivors (557056K)
Metaspace used 144921K, capacity 149223K, committed 149440K, reserved 149504K
}
Event: 1095746.491 GC heap before
{Heap before GC invocations=9812 (full 0):
garbage-first heap total 67108864K, used 41462290K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 682 young (22347776K), 17 survivors (557056K)
Metaspace used 144938K, capacity 149244K, committed 149440K, reserved 149504K
Event: 1095746.629 GC heap after
Heap after GC invocations=9813 (full 0):
garbage-first heap total 67108864K, used 19696962K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 17 young (557056K), 17 survivors (557056K)
Metaspace used 144938K, capacity 149244K, committed 149440K, reserved 149504K
}
Event: 1095881.378 GC heap before
{Heap before GC invocations=9813 (full 0):
garbage-first heap total 67108864K, used 41487682K [0x00007f33d2000000, 0x00007f33d4004000, 0x00007f43d2000000)
region size 32768K, 682 young (22347776K), 17 survivors (557056K)
Metaspace used 144974K, capacity 149252K, committed 149440K, reserved 149504K

--------

CoreDump #2

Stack: [0x00007f7aa0973000,0x00007f7aa0a74000], sp=0x00007f7aa0a6e1c0, free space=1004k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x651649] InstanceKlass::oop_oop_iterate_backwards_nv(oopDesc*, G1ParScanClosure*)+0xc9
V [libjvm.so+0x661140] InstanceMirrorKlass::oop_oop_iterate_backwards_nv(oopDesc*, G1ParScanClosure*)+0x20
V [libjvm.so+0x5c7845] G1ParScanThreadState::copy_to_survivor_space(InCSetState, oopDesc*, markOopDesc*)+0x4c5
V [libjvm.so+0x5aa7b1] G1ParCopyClosure<(G1Barrier)2, (G1Mark)0>::do_oop(oopDesc**)+0x51
V [libjvm.so+0x5aa0e4] G1KlassScanClosure::do_klass(Klass*)+0x34
V [libjvm.so+0x46bc7c] ClassLoaderData::oops_do(OopClosure*, KlassClosure*, bool)+0x8c
V [libjvm.so+0x46afc8] ClassLoaderDataGraph::roots_cld_do(CLDClosure*, CLDClosure*)+0x38
V [libjvm.so+0x5ce4de] G1RootProcessor::process_java_roots(OopClosure*, CLDClosure*, CLDClosure*, CLDClosure*, CodeBlobClosure*, G1GCPhaseTimes*, unsigned int)+0x6e
V [libjvm.so+0x5ced81] G1RootProcessor::evacuate_roots(OopClosure*, OopClosure*, CLDClosure*, CLDClosure*, bool, unsigned int)+0x561
V [libjvm.so+0x5ae1a8] G1ParTask::work(unsigned int)+0x3b8
V [libjvm.so+0xaed0ff] GangWorker::loop()+0xcf
V [libjvm.so+0x92a728] java_start(Thread*)+0x108

Heap:
garbage-first heap total 67108864K, used 44362593K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 682 young (22347776K), 5 survivors (163840K)
Metaspace used 150062K, capacity 155003K, committed 155360K, reserved 155648K

GC Heap History (10 events):
Event: 2031681.264 GC heap after
Heap after GC invocations=15747 (full 0):
garbage-first heap total 67108864K, used 22165995K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 4 young (131072K), 4 survivors (131072K)
Metaspace used 150034K, capacity 155003K, committed 155360K, reserved 155648K
}
Event: 2031720.454 GC heap before
{Heap before GC invocations=15747 (full 0):
garbage-first heap total 67108864K, used 44382699K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 682 young (22347776K), 4 survivors (131072K)
Metaspace used 150034K, capacity 155003K, committed 155360K, reserved 155648K
Event: 2031720.538 GC heap after
Heap after GC invocations=15748 (full 0):
garbage-first heap total 67108864K, used 22175438K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 5 young (163840K), 5 survivors (163840K)
Metaspace used 150034K, capacity 155003K, committed 155360K, reserved 155648K
}
Event: 2031754.375 GC heap before
{Heap before GC invocations=15748 (full 0):
garbage-first heap total 67108864K, used 44359374K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 682 young (22347776K), 5 survivors (163840K)
Metaspace used 150034K, capacity 155003K, committed 155360K, reserved 155648K
Event: 2031754.461 GC heap after
Heap after GC invocations=15749 (full 0):
garbage-first heap total 67108864K, used 22158646K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 4 young (131072K), 4 survivors (131072K)
Metaspace used 150034K, capacity 155003K, committed 155360K, reserved 155648K
}
Event: 2031795.794 GC heap before
{Heap before GC invocations=15749 (full 0):
garbage-first heap total 67108864K, used 44375350K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 682 young (22347776K), 4 survivors (131072K)
Metaspace used 150034K, capacity 155003K, committed 155360K, reserved 155648K
Event: 2031795.892 GC heap after
Heap after GC invocations=15750 (full 0):
garbage-first heap total 67108864K, used 22165753K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 4 young (131072K), 4 survivors (131072K)
Metaspace used 150034K, capacity 155003K, committed 155360K, reserved 155648K
}
Event: 2031844.619 GC heap before
{Heap before GC invocations=15750 (full 0):
garbage-first heap total 67108864K, used 44382457K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 682 young (22347776K), 4 survivors (131072K)
Metaspace used 150034K, capacity 155003K, committed 155360K, reserved 155648K
Event: 2031844.716 GC heap after
Heap after GC invocations=15751 (full 0):
garbage-first heap total 67108864K, used 22178657K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 5 young (163840K), 5 survivors (163840K)
Metaspace used 150034K, capacity 155003K, committed 155360K, reserved 155648K
}
Event: 2031889.871 GC heap before
{Heap before GC invocations=15751 (full 0):
garbage-first heap total 67108864K, used 44362593K [0x00007f6a7e000000, 0x00007f6a80004000, 0x00007f7a7e000000)
region size 32768K, 682 young (22347776K), 5 survivors (163840K)

Metaspace used 150062K, capacity 155003K, committed 155360K, reserved 155648K

[#3]

and jvm arguments are as follows;

VM Arguments:
jvm_args: -Xmx1024m -Dhdp.version=2.6.3.0-235 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.3.0-235 -Dhadoop.log.dir=/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.3.0-235/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.3.0-235/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.6.3.0-235/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.6.3.0-235/hadoop/lib... -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:MaxGCPauseMillis=1000 -XX:InitiatingHeapOccupancyPercent=45 -XX:NewRatio=2 -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=15 -XX:G1ReservePercent=10 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -hiveconf hive.metastore.uris= -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/log/hive