Hi:
I use HBASE 2.1.6 version, and enable MOB feature on my tables
then I start to input data to my table,
after a while, 4 regionservers are all down, and the error log hs_err_pidxxx.log generated in the /var/log/hbase/.
after to restart the hbase, regionservers still down, it can not start again
I have no idea how to check the hs_err_pidxxx.log
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007efe5ae4ae40, pid=11890, tid=0x00007efe585d5700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_241-b07) (build 1.8.0_241-b07)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.241-b07 mixed mode linux-amd64 )
# Problematic frame:
# V [libjvm.so+0x649e40] void G1ParScanClosure::do_oop_nv<oopDesc*>(oopDesc**)+0x30
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x00007efe5405d800): GCTaskThread [stack: 0x00007efe584d5000,0x00007efe585d6000] [id=12112]
siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: 0x00007efe53916e78
Registers:
RAX=0x00000000ed02a6a0, RBX=0x00007efdb1724bc0, RCX=0x00000000000000ed, RDX=0x00007efe54037810
RSP=0x00007efe585d45e0, RBP=0x00007efe585d45f0, RSI=0x00007efe53916d8b, RDI=0x00007efe585d4dc0
R8 =0x0000000000000000, R9 =0x00000000000000a8, R10=0x0000000000000001, R11=0x00007efe5b9733b0
R12=0x00007efdb1724ba8, R13=0x00007ef3ac13f2c8, R14=0x00007ef3ac13f048, R15=0x00007efdb1724bc0
RIP=0x00007efe5ae4ae40, EFLAGS=0x0000000000010206, CSGSFS=0x002b000000000033, ERR=0x0000000000000004
TRAPNO=0x000000000000000e
Top of Stack: (sp=0x00007efe585d45e0)
0x00007efe585d45e0: 00007efe585d4dc0 00007efdb1724ba8
0x00007efe585d45f0: 00007efe585d4640 00007efe5ae48427
0x00007efe585d4600: 00007ef3ac13f2c8 00007efdb1724b98
0x00007efe585d4610: 00007efe585d4dc0 00007efe21ffbca8
0x00007efe585d4620: 00007efe585d4dc0 00007efdb1724b98
0x00007efe585d4630: 00007ef3ac13f048 00007efe585d4c50
0x00007efe585d4640: 00007efe585d4680 00007efe5ae505ff
0x00007efe585d4650: 00007efe585d4670 00007efe21ffbca8
0x00007efe585d4660: 00007efdb1724b98 0000000000000016
................
Register to memory mapping:
RAX=0x00000000ed02a6a0 is an unknown value
RBX=0x00007efdb1724bc0 is pointing into object: 0x00007efdb1724b98
java.lang.Class
- klass: 'java/lang/Class'
RCX=0x00000000000000ed is an unknown value
RDX=0x00007efe54037810 is an unknown value
RSP=0x00007efe585d45e0 is an unknown value
RBP=0x00007efe585d45f0 is an unknown value
RSI=0x00007efe53916d8b is an unknown value
RDI=0x00007efe585d4dc0 is an unknown value
R8 =0x0000000000000000 is an unknown value
R9 =0x00000000000000a8 is an unknown value
R10=0x0000000000000001 is an unknown value
R11=0x00007efe5b9733b0: <offset 0x1823b0> in /lib64/libc.so.6 at 0x00007efe5b7f1000
R12=0x00007efdb1724ba8 is pointing into object: 0x00007efdb1724b98
java.lang.Class
- klass: 'java/lang/Class'
R13=0x00007ef3ac13f2c8 is pointing into metadata
R14=0x00007ef3ac13f048 is pointing into metadata
R15=0x00007efdb1724bc0 is pointing into object: 0x00007efdb1724b98
java.lang.Class
- klass: 'java/lang/Class'
Stack: [0x00007efe584d5000,0x00007efe585d6000], sp=0x00007efe585d45e0, free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x649e40] void G1ParScanClosure::do_oop_nv<oopDesc*>(oopDesc**)+0x30
V [libjvm.so+0x647427] InstanceKlass::oop_oop_iterate_backwards_nv(oopDesc*, G1ParScanClosure*)+0x167
V [libjvm.so+0x64f5ff] InstanceMirrorKlass::oop_oop_iterate_backwards_nv(oopDesc*, G1ParScanClosure*)+0x1f
V [libjvm.so+0x5b14e9] G1ParScanThreadState::copy_to_survivor_space(InCSetState, oopDesc*, markOopDesc*)+0x569
V [libjvm.so+0x5b1e53] G1ParScanThreadState::trim_queue()+0x643
V [libjvm.so+0x58aefb] G1ParEvacuateFollowersClosure::do_void()+0x26b
V [libjvm.so+0x59b51c] G1ParTask::work(unsigned int)+0x44c
V [libjvm.so+0xaf9918] GangWorker::loop()+0xd8
V [libjvm.so+0x90f542] java_start(Thread*)+0x102
................
0x00007efe55d69800 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=12233, stack(0x00007ef3a5424000,0x00007ef3a5525000)]
0x00007efe55d68000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=12232, stack(0x00007ef3a5525000,0x00007ef3a5626000)]
0x00007efe55d63000 JavaThread "Surrogate Locker Thread (Concurrent GC)" daemon [_thread_blocked, id=12231, stack(0x00007ef3a5626000,0x00007ef3a5727000)]
0x00007efe55d35000 JavaThread "Finalizer" daemon [_thread_blocked, id=12227, stack(0x00007ef3abc4e000,0x00007ef3abd4f000)]
0x00007efe55d30000 JavaThread "Reference Handler" daemon [_thread_blocked, id=12226, stack(0x00007ef3abd4f000,0x00007ef3abe50000)]
0x00007efe54023000 JavaThread "main" [_thread_blocked, id=12093, stack(0x00007efe5c2f7000,0x00007efe5c3f8000)]
Other Threads:
0x00007efe55d26800 VMThread [stack: 0x00007ef3abe50000,0x00007ef3abf51000] [id=12225]
0x00007efe55dad800 WatcherThread [stack: 0x00007ef3a4111000,0x00007ef3a4212000] [id=12254]
=>0x00007efe5405d800 (exited) GCTaskThread [stack: 0x00007efe584d5000,0x00007efe585d6000] [id=12112]
VM state:at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x00007efe54021810] Threads_lock - owner thread: 0x00007efe55d26800
[0x00007efe54021d10] Heap_lock - owner thread: 0x00000000021c1800
Heap:
garbage-first heap total 41943040K, used 2107094K [0x00007ef435000000, 0x00007ef436005000, 0x00007efe35000000)
region size 16384K, 128 young (2097152K), 4 survivors (65536K)
Metaspace used 96765K, capacity 99604K, committed 99840K, reserved 100352K
Heap Regions: (Y=young(eden), SU=young(survivor), HS=humongous(starts), HC=humongous(continues), CS=collection set, F=free, TS=gc time stamp, PTAMS=previous top-at-mark-start, NTAMS=next top-at-mark-start)
AC 0 HS TS 0 PTAMS 0x00007ef435800018 NTAMS 0x00007ef435800018 space 16384K, 50% used [0x00007ef435000000, 0x00007ef436000000)
AC 0 HS TS 0 PTAMS 0x00007ef436800018 NTAMS 0x00007ef436800018 space 16384K, 50% used [0x00007ef436000000, 0x00007ef437000000)
AC 0 F TS 0 PTAMS 0x00007ef437000000 NTAMS 0x00007ef437000000 space 16384K, 0% used [0x00007ef437000000, 0x00007ef438000000)
AC 0 F TS 0 PTAMS 0x00007ef438000000 NTAMS 0x00007ef438000000 space 16384K, 0% used [0x00007ef438000000, 0x00007ef439000000)
