Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

FATAL org.apache.hadoop.hbase.regionserver.RSRpcServices: Run out of memory

FATAL org.apache.hadoop.hbase.regionserver.RSRpcServices: Run out of memory

New Contributor

 

 

 

 

We are seeing frequient failures of region servers with the below error. This was started suddenly from last week.

 

2018-06-07 03:30:44,971 FATAL org.apache.hadoop.hbase.regionserver.RSRpcServices: Run out of memory; RSRpcServices will abort itself immediately
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360)
at java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:181)
at org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:218)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceWorkerThread.shipEdits(ReplicationSource.java:946)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceWorkerThread.run(ReplicationSource.java:631)
2018-06-07 03:30:44,971 WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /10.0.25.222:50010 for block, add to deadNodes and continue. java.io.IOException: Connection reset by peer
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.jav

2 REPLIES 2

Re: FATAL org.apache.hadoop.hbase.regionserver.RSRpcServices: Run out of memory

Cloudera Employee

Region server seems to be running out of memory/file descriptors. you can try increasing the "memory" &  "Max open files".

Re: FATAL org.apache.hadoop.hbase.regionserver.RSRpcServices: Run out of memory

Cloudera Employee

As mentioned by Srinivasulu, your HBase RegionServer instances are hitting the limitation of available native threads that the Operating System will allow the JVM to have.

 

You may need to investigate the ulimit definitions on your hosts to see if they need to be increased further.

 

Depending on the current configurations you have for ulimit of these hosts, it may require further investigation to understand why so many threads are being consumed by HBase. Over-utilized RegionServer instances can be indicative of hotspotting regions due to poor rowkey/table design, so understanding and comparing the number of requests per region amongst all servers is imperative to identifying hotspotting regions.