Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

hbase regionserver down for JVM crashed , and does not start again

avatar

Hi:

I use HBASE 2.1.6 version, and enable MOB feature on my tables

then I start to input data to my table,

after a while, 4 regionservers are all down, and the error log hs_err_pidxxx.log  generated in the /var/log/hbase/.

after to restart the hbase, regionservers still down, it can not start again

I have no idea how to check the hs_err_pidxxx.log

 

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007efe5ae4ae40, pid=11890, tid=0x00007efe585d5700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_241-b07) (build 1.8.0_241-b07)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.241-b07 mixed mode linux-amd64 )
# Problematic frame:
# V  [libjvm.so+0x649e40]  void G1ParScanClosure::do_oop_nv<oopDesc*>(oopDesc**)+0x30
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007efe5405d800):  GCTaskThread [stack: 0x00007efe584d5000,0x00007efe585d6000] [id=12112]

siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: 0x00007efe53916e78

Registers:
RAX=0x00000000ed02a6a0, RBX=0x00007efdb1724bc0, RCX=0x00000000000000ed, RDX=0x00007efe54037810
RSP=0x00007efe585d45e0, RBP=0x00007efe585d45f0, RSI=0x00007efe53916d8b, RDI=0x00007efe585d4dc0
R8 =0x0000000000000000, R9 =0x00000000000000a8, R10=0x0000000000000001, R11=0x00007efe5b9733b0
R12=0x00007efdb1724ba8, R13=0x00007ef3ac13f2c8, R14=0x00007ef3ac13f048, R15=0x00007efdb1724bc0
RIP=0x00007efe5ae4ae40, EFLAGS=0x0000000000010206, CSGSFS=0x002b000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007efe585d45e0)
0x00007efe585d45e0:   00007efe585d4dc0 00007efdb1724ba8
0x00007efe585d45f0:   00007efe585d4640 00007efe5ae48427
0x00007efe585d4600:   00007ef3ac13f2c8 00007efdb1724b98
0x00007efe585d4610:   00007efe585d4dc0 00007efe21ffbca8
0x00007efe585d4620:   00007efe585d4dc0 00007efdb1724b98
0x00007efe585d4630:   00007ef3ac13f048 00007efe585d4c50
0x00007efe585d4640:   00007efe585d4680 00007efe5ae505ff
0x00007efe585d4650:   00007efe585d4670 00007efe21ffbca8
0x00007efe585d4660:   00007efdb1724b98 0000000000000016

................


Register to memory mapping:

RAX=0x00000000ed02a6a0 is an unknown value
RBX=0x00007efdb1724bc0 is pointing into object: 0x00007efdb1724b98
java.lang.Class
 - klass: 'java/lang/Class'
RCX=0x00000000000000ed is an unknown value
RDX=0x00007efe54037810 is an unknown value
RSP=0x00007efe585d45e0 is an unknown value
RBP=0x00007efe585d45f0 is an unknown value
RSI=0x00007efe53916d8b is an unknown value
RDI=0x00007efe585d4dc0 is an unknown value
R8 =0x0000000000000000 is an unknown value
R9 =0x00000000000000a8 is an unknown value
R10=0x0000000000000001 is an unknown value
R11=0x00007efe5b9733b0: <offset 0x1823b0> in /lib64/libc.so.6 at 0x00007efe5b7f1000
R12=0x00007efdb1724ba8 is pointing into object: 0x00007efdb1724b98
java.lang.Class
 - klass: 'java/lang/Class'
R13=0x00007ef3ac13f2c8 is pointing into metadata
R14=0x00007ef3ac13f048 is pointing into metadata
R15=0x00007efdb1724bc0 is pointing into object: 0x00007efdb1724b98
java.lang.Class
 - klass: 'java/lang/Class'


Stack: [0x00007efe584d5000,0x00007efe585d6000],  sp=0x00007efe585d45e0,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x649e40]  void G1ParScanClosure::do_oop_nv<oopDesc*>(oopDesc**)+0x30
V  [libjvm.so+0x647427]  InstanceKlass::oop_oop_iterate_backwards_nv(oopDesc*, G1ParScanClosure*)+0x167
V  [libjvm.so+0x64f5ff]  InstanceMirrorKlass::oop_oop_iterate_backwards_nv(oopDesc*, G1ParScanClosure*)+0x1f
V  [libjvm.so+0x5b14e9]  G1ParScanThreadState::copy_to_survivor_space(InCSetState, oopDesc*, markOopDesc*)+0x569
V  [libjvm.so+0x5b1e53]  G1ParScanThreadState::trim_queue()+0x643
V  [libjvm.so+0x58aefb]  G1ParEvacuateFollowersClosure::do_void()+0x26b
V  [libjvm.so+0x59b51c]  G1ParTask::work(unsigned int)+0x44c
V  [libjvm.so+0xaf9918]  GangWorker::loop()+0xd8
V  [libjvm.so+0x90f542]  java_start(Thread*)+0x102


................


  0x00007efe55d69800 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=12233, stack(0x00007ef3a5424000,0x00007ef3a5525000)]
  0x00007efe55d68000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=12232, stack(0x00007ef3a5525000,0x00007ef3a5626000)]
  0x00007efe55d63000 JavaThread "Surrogate Locker Thread (Concurrent GC)" daemon [_thread_blocked, id=12231, stack(0x00007ef3a5626000,0x00007ef3a5727000)]
  0x00007efe55d35000 JavaThread "Finalizer" daemon [_thread_blocked, id=12227, stack(0x00007ef3abc4e000,0x00007ef3abd4f000)]
  0x00007efe55d30000 JavaThread "Reference Handler" daemon [_thread_blocked, id=12226, stack(0x00007ef3abd4f000,0x00007ef3abe50000)]
  0x00007efe54023000 JavaThread "main" [_thread_blocked, id=12093, stack(0x00007efe5c2f7000,0x00007efe5c3f8000)]

Other Threads:
  0x00007efe55d26800 VMThread [stack: 0x00007ef3abe50000,0x00007ef3abf51000] [id=12225]
  0x00007efe55dad800 WatcherThread [stack: 0x00007ef3a4111000,0x00007ef3a4212000] [id=12254]

=>0x00007efe5405d800 (exited) GCTaskThread [stack: 0x00007efe584d5000,0x00007efe585d6000] [id=12112]

VM state:at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x00007efe54021810] Threads_lock - owner thread: 0x00007efe55d26800
[0x00007efe54021d10] Heap_lock - owner thread: 0x00000000021c1800

Heap:
 garbage-first heap   total 41943040K, used 2107094K [0x00007ef435000000, 0x00007ef436005000, 0x00007efe35000000)
  region size 16384K, 128 young (2097152K), 4 survivors (65536K)
 Metaspace       used 96765K, capacity 99604K, committed 99840K, reserved 100352K

Heap Regions: (Y=young(eden), SU=young(survivor), HS=humongous(starts), HC=humongous(continues), CS=collection set, F=free, TS=gc time stamp, PTAMS=previous top-at-mark-start, NTAMS=next top-at-mark-start)
AC   0 HS    TS     0 PTAMS 0x00007ef435800018 NTAMS 0x00007ef435800018 space 16384K,  50% used [0x00007ef435000000, 0x00007ef436000000)
AC   0 HS    TS     0 PTAMS 0x00007ef436800018 NTAMS 0x00007ef436800018 space 16384K,  50% used [0x00007ef436000000, 0x00007ef437000000)
AC   0  F    TS     0 PTAMS 0x00007ef437000000 NTAMS 0x00007ef437000000 space 16384K,   0% used [0x00007ef437000000, 0x00007ef438000000)
AC   0  F    TS     0 PTAMS 0x00007ef438000000 NTAMS 0x00007ef438000000 space 16384K,   0% used [0x00007ef438000000, 0x00007ef439000000)

 

1 REPLY 1

avatar
Super Collaborator

hi @polingsky202 could you please paste the runtime error stack