question Re: Hbase Regionserver doesn't start in Support Questions

Hbase Regionserver doesn't start

Joyglobal — Fri, 20 May 2016 00:40:30 GMT

Please help me out,

Unable to figure out what the problem is while starting regionserver, It exits with following message

starting regionserver, logging to /var/log/hbase/hbase-root-regionserver-ip-172-31-19-88.out
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000300000000, 20401094656, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 20401094656 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /var/log/hbase/hs_err_pid31016.log

below are the free -m o/p:

            total |used |free|shared|buffers|cached
Mem:         68555 |21134 |47420|0|181 |14850 -/+ buffers/cache:       6102      62452
Swap:            0          0          0

I am not not sure if it is a Java Error or OS error

Below is are system configs:

Ram : 70gb
os: Redhat centos 6.7
Ambari : 2.2.2  
HDP : 2.4.2
Java Version (Ambari): 1.8.0_40

Re: Hbase Regionserver doesn't start

elserj — Fri, 20 May 2016 00:49:19 GMT

That error message is saying that the JVM failed to allocate ~20GB of memory via the Linux system call (mmap). It looks like that would be very close to the total unused memory on your system which is likely why the call failed. The operating system couldn't actually allocate that much memory for your process.

Try setting -Xmx16G in the variable HBASE_REGIONSERVER_OPTS in hbase-env.sh instead of 20G.

Re: Hbase Regionserver doesn't start

Joyglobal — Fri, 20 May 2016 01:15:55 GMT

I have changed the configuration in Ambari it resulted as follows:

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000400000000, 16106127360, 0) failed; error='Cannot allocate memory' (errno=12) # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 16106127360 bytes for committing reserved memory. # An error report file with more information is saved as: # /var/log/hbase/hs_err_pid35924.log

Re: Hbase Regionserver doesn't start

Stry2K — Fri, 20 May 2016 01:55:17 GMT

Additional Information:

HBase Region Server that fails immediately on start from Ambari shows this error in logfile:

2016-05-06 18:53:46,507 ERROR [ShortCircuitCache_SlotReleaser] shortcircuit.ShortCircuitCache: ShortCircuitCache(0x196d0093): failed to release short-circuit shared memory slot Slot(slotIdx=126, shm=DfsClientShm(4ceb7a14af49cdf5ee906542fea6dfee)) by sending ReleaseShortCircuitAccessRequestProto to /var/lib/hadoop-hdfs/dn_socket. Closing shared memory segment. java.io.IOException: ERROR_INVALID: there is no shared memory segment registered with shmId 4ceb7a14af49cdf5ee906542fea6dfee at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:208) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-05-06 18:53:46,507 WARN [ShortCircuitCache_SlotReleaser] shortcircuit.DfsClientShmManager: EndpointShmManager(DatanodeInfoWithStorage[172.31.19.89:50010,DS-80a6e92c-8fcc-4ee4-a172-0cfa57a6768a,DISK], parent=ShortCircuitShmManager(4c54fd52)): error shutting down shm: got IOException calling shutdown(SHUT_RDWR)

Re: Hbase Regionserver doesn't start

elserj — Fri, 20 May 2016 01:56:07 GMT

Also, can you share that JVM crash report? It probably has some signs about what was trying to do that allocation.

Re: Hbase Regionserver doesn't start

Joyglobal — Fri, 20 May 2016 02:14:17 GMT

Please find the log file attached

hs-err-pid41985.zip

Re: Hbase Regionserver doesn't start

elserj — Fri, 20 May 2016 03:34:09 GMT

Current thread (0x00007f2e7c01d000):  JavaThread "Unknown thread" [_thread_in_vm, id=42067, stack(0x00007f2e83bc8000,0x00007f2e83cc9000)]

Stack: [0x00007f2e83bc8000,0x00007f2e83cc9000],  sp=0x00007f2e83cc72f0,  free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xaaca9a]  VMError::report_and_die()+0x2ba
V  [libjvm.so+0x4f333b]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x8b
V  [libjvm.so+0x90e8c3]  os::Linux::commit_memory_impl(char*, unsigned long, bool)+0x103
V  [libjvm.so+0x90ee19]  os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x29
V  [libjvm.so+0x90877a]  os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a
V  [libjvm.so+0xaa8749]  VirtualSpace::expand_by(unsigned long, bool)+0x1c9
V  [libjvm.so+0xaa921e]  VirtualSpace::initialize(ReservedSpace, unsigned long)+0xee
V  [libjvm.so+0x5eda21]  CardGeneration::CardGeneration(ReservedSpace, unsigned long, int, GenRemSet*)+0xf1
V  [libjvm.so+0x4cb3fb]  ConcurrentMarkSweepGeneration::ConcurrentMarkSweepGeneration(ReservedSpace, unsigned long, int, CardTableRS*, bool, FreeBlockDictionary<FreeChunk>::DictionaryChoice)+0x4b
V  [libjvm.so+0x5eead2]  GenerationSpec::init(ReservedSpace, int, GenRemSet*)+0xf2
V  [libjvm.so+0x5dd77e]  GenCollectedHeap::initialize()+0x1ee
V  [libjvm.so+0xa75bab]  Universe::initialize_heap()+0xfb
V  [libjvm.so+0xa75f1e]  universe_init()+0x3e
V  [libjvm.so+0x62f665]  init_globals()+0x65
V  [libjvm.so+0xa5a12e]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x23e
V  [libjvm.so+0x6c3274]  JNI_CreateJavaVM+0x74
C  [libjli.so+0x745e]  JavaMain+0x9e
C  [libpthread.so.0+0x7aa1]  start_thread+0xd1

Is telling us that the JVM was still trying to create the JVM and failed to do so.

jvm_args: -Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p -Dhdp.version=2.4.2.0-258 -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hbase/hs_err_pid%p.log -Djava.io.tmpdir=/tmp -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/hbase/gc.log-201605191913 -Xmn1024m -XX:CMSInitiatingOccupancyFraction=70 -Xms16384m -Xmx16384m -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-root-regionserver-ip-172-31-19-88.log -Dhbase.home.dir=/usr/hdp/current/hbase-regionserver/bin/.. -Dhbase.id.str=root -Dhbase.root.logger=DEBUG,RFA -Djava.library.path=:/usr/hdp/2.4.2.0-258/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.2.0-258/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS

We can also see all of the JVM arguments. You set the young generation size to 1G, and the min and max heap size to 16G. These seem fine.

I apparently misread your free memory earlier. You have 40GB free and 20G in caches, so you should have plenty of free memory for HBase.

Can you also please share the output of `ulimit -a` as the hbase user and your hbase-site.xml?

Re: Hbase Regionserver doesn't start

asinghal — Fri, 20 May 2016 14:52:14 GMT

Hi @Raghu Gurrala,

Would you mind taking a look of below page.

http://www.theshell.guru/native-memory-allocation-mmap-failed-to-map-xx-bytes-for-committing-reserved-memory-jdk-1-8-centos-6-7/

This guy also faced the same problem and I think he has the working fix.

Re: Hbase Regionserver doesn't start

Joyglobal — Sat, 21 May 2016 01:56:02 GMT

 ulimit -a 
core file size (blocks, -c) 0 
data seg size (kbytes, -d) 
unlimited scheduling priority (-e) 0 
file size (blocks, -f) 
unlimited pending signals (-i) 274125 
max locked memory (kbytes, -l) 64 
max memory size (kbytes, -m) 
unlimited open files (-n) 32000 
pipe size (512 bytes, -p) 8 
POSIX message queues (bytes, -q) 819200 
real-time priority (-r) 0 
stack size (kbytes, -s) 10240 
cpu time (seconds, -t) 
unlimited max user processes (-u) 16000 
virtual memory (kbytes, -v) 
unlimited file locks (-x) unlimited

Re: Hbase Regionserver doesn't start

Joyglobal — Sat, 21 May 2016 01:57:45 GMT

Thank you for getting back, below is the output for ulimit -a

Re: Hbase Regionserver doesn't start

Joyglobal — Sat, 21 May 2016 01:58:39 GMT

Thank you for the response,

Tried that one but seem to have no impact

Re: Hbase Regionserver doesn't start

Joyglobal — Mon, 13 Jun 2016 21:43:59 GMT

Here is the solution

Hawq installation have caused a change to /etc/sysctl.conf

Had to change it back to default value and got rid of all the OOM errors.

vm.overcommit_memory=2

Had to change to default value 
vm.overcommit_memory=0