Hbase Regionserver doesn't start

[#1]

Please help me out,

Unable to figure out what the problem is while starting regionserver, It exits with following message

starting regionserver, logging to /var/log/hbase/hbase-root-regionserver-ip-172-31-19-88.out
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000300000000, 20401094656, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 20401094656 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /var/log/hbase/hs_err_pid31016.log

below are the free -m o/p:

            total |used |free|shared|buffers|cached
Mem:         68555 |21134 |47420|0|181 |14850 -/+ buffers/cache:       6102      62452
Swap:            0          0          0

I am not not sure if it is a Java Error or OS error

Below is are system configs:

Ram : 70gb
os: Redhat centos 6.7
Ambari : 2.2.2  
HDP : 2.4.2
Java Version (Ambari): 1.8.0_40 

[#2]

That error message is saying that the JVM failed to allocate ~20GB of memory via the Linux system call (mmap). It looks like that would be very close to the total unused memory on your system which is likely why the call failed. The operating system couldn't actually allocate that much memory for your process.

Try setting -Xmx16G in the variable HBASE_REGIONSERVER_OPTS in hbase-env.sh instead of 20G.

[#3]

I have changed the configuration in Ambari it resulted as follows:

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000400000000, 16106127360, 0) failed; error='Cannot allocate memory' (errno=12) # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 16106127360 bytes for committing reserved memory. # An error report file with more information is saved as: # /var/log/hbase/hs_err_pid35924.log

[#4]

Also, can you share that JVM crash report? It probably has some signs about what was trying to do that allocation.

[#5]

Please find the log file attached

hs-err-pid41985.zip

[#6]

Additional Information:

HBase Region Server that fails immediately on start from Ambari shows this error in logfile:

2016-05-06 18:53:46,507 ERROR [ShortCircuitCache_SlotReleaser] shortcircuit.ShortCircuitCache: ShortCircuitCache(0x196d0093): failed to release short-circuit shared memory slot Slot(slotIdx=126, shm=DfsClientShm(4ceb7a14af49cdf5ee906542fea6dfee)) by sending ReleaseShortCircuitAccessRequestProto to /var/lib/hadoop-hdfs/dn_socket. Closing shared memory segment. java.io.IOException: ERROR_INVALID: there is no shared memory segment registered with shmId 4ceb7a14af49cdf5ee906542fea6dfee at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:208) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-05-06 18:53:46,507 WARN [ShortCircuitCache_SlotReleaser] shortcircuit.DfsClientShmManager: EndpointShmManager(DatanodeInfoWithStorage[172.31.19.89:50010,DS-80a6e92c-8fcc-4ee4-a172-0cfa57a6768a,DISK], parent=ShortCircuitShmManager(4c54fd52)): error shutting down shm: got IOException calling shutdown(SHUT_RDWR)

[#7]

Current thread (0x00007f2e7c01d000):  JavaThread "Unknown thread" [_thread_in_vm, id=42067, stack(0x00007f2e83bc8000,0x00007f2e83cc9000)]

Stack: [0x00007f2e83bc8000,0x00007f2e83cc9000],  sp=0x00007f2e83cc72f0,  free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xaaca9a]  VMError::report_and_die()+0x2ba
V  [libjvm.so+0x4f333b]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x8b
V  [libjvm.so+0x90e8c3]  os::Linux::commit_memory_impl(char*, unsigned long, bool)+0x103
V  [libjvm.so+0x90ee19]  os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x29
V  [libjvm.so+0x90877a]  os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a
V  [libjvm.so+0xaa8749]  VirtualSpace::expand_by(unsigned long, bool)+0x1c9
V  [libjvm.so+0xaa921e]  VirtualSpace::initialize(ReservedSpace, unsigned long)+0xee
V  [libjvm.so+0x5eda21]  CardGeneration::CardGeneration(ReservedSpace, unsigned long, int, GenRemSet*)+0xf1
V  [libjvm.so+0x4cb3fb]  ConcurrentMarkSweepGeneration::ConcurrentMarkSweepGeneration(ReservedSpace, unsigned long, int, CardTableRS*, bool, FreeBlockDictionary<FreeChunk>::DictionaryChoice)+0x4b
V  [libjvm.so+0x5eead2]  GenerationSpec::init(ReservedSpace, int, GenRemSet*)+0xf2
V  [libjvm.so+0x5dd77e]  GenCollectedHeap::initialize()+0x1ee
V  [libjvm.so+0xa75bab]  Universe::initialize_heap()+0xfb
V  [libjvm.so+0xa75f1e]  universe_init()+0x3e
V  [libjvm.so+0x62f665]  init_globals()+0x65
V  [libjvm.so+0xa5a12e]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x23e
V  [libjvm.so+0x6c3274]  JNI_CreateJavaVM+0x74
C  [libjli.so+0x745e]  JavaMain+0x9e
C  [libpthread.so.0+0x7aa1]  start_thread+0xd1

Is telling us that the JVM was still trying to create the JVM and failed to do so.

jvm_args: -Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p -Dhdp.version=2.4.2.0-258 -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hbase/hs_err_pid%p.log -Djava.io.tmpdir=/tmp -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/hbase/gc.log-201605191913 -Xmn1024m -XX:CMSInitiatingOccupancyFraction=70 -Xms16384m -Xmx16384m -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-root-regionserver-ip-172-31-19-88.log -Dhbase.home.dir=/usr/hdp/current/hbase-regionserver/bin/.. -Dhbase.id.str=root -Dhbase.root.logger=DEBUG,RFA -Djava.library.path=:/usr/hdp/2.4.2.0-258/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.2.0-258/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS

We can also see all of the JVM arguments. You set the young generation size to 1G, and the min and max heap size to 16G. These seem fine.

I apparently misread your free memory earlier. You have 40GB free and 20G in caches, so you should have plenty of free memory for HBase.

Can you also please share the output of `ulimit -a` as the hbase user and your hbase-site.xml?

[#8]

 ulimit -a 
core file size (blocks, -c) 0 
data seg size (kbytes, -d) 
unlimited scheduling priority (-e) 0 
file size (blocks, -f) 
unlimited pending signals (-i) 274125 
max locked memory (kbytes, -l) 64 
max memory size (kbytes, -m) 
unlimited open files (-n) 32000 
pipe size (512 bytes, -p) 8 
POSIX message queues (bytes, -q) 819200 
real-time priority (-r) 0 
stack size (kbytes, -s) 10240 
cpu time (seconds, -t) 
unlimited max user processes (-u) 16000 
virtual memory (kbytes, -v) 
unlimited file locks (-x) unlimited

[#9]

Thank you for getting back, below is the output for ulimit -a

[#10]

Hi @Raghu Gurrala,

Would you mind taking a look of below page.

http://www.theshell.guru/native-memory-allocation-mmap-failed-to-map-xx-bytes-for-committing-reserve...

This guy also faced the same problem and I think he has the working fix.