Reply
New Contributor
Posts: 1
Registered: ‎06-06-2018

FATAL org.apache.hadoop.hbase.regionserver.RSRpcServices: Run out of memory

 

 

 

 

We are seeing frequient failures of region servers with the below error. This was started suddenly from last week.

 

2018-06-07 03:30:44,971 FATAL org.apache.hadoop.hbase.regionserver.RSRpcServices: Run out of memory; RSRpcServices will abort itself immediately
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360)
at java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:181)
at org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:218)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceWorkerThread.shipEdits(ReplicationSource.java:946)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceWorkerThread.run(ReplicationSource.java:631)
2018-06-07 03:30:44,971 WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /10.0.25.222:50010 for block, add to deadNodes and continue. java.io.IOException: Connection reset by peer
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.jav

Highlighted
Cloudera Employee
Posts: 4
Registered: ‎07-30-2018

Re: FATAL org.apache.hadoop.hbase.regionserver.RSRpcServices: Run out of memory

Region server seems to be running out of memory/file descriptors. you can try increasing the "memory" &  "Max open files".

Cloudera Employee
Posts: 53
Registered: ‎09-08-2017

Re: FATAL org.apache.hadoop.hbase.regionserver.RSRpcServices: Run out of memory

As mentioned by Srinivasulu, your HBase RegionServer instances are hitting the limitation of available native threads that the Operating System will allow the JVM to have.

 

You may need to investigate the ulimit definitions on your hosts to see if they need to be increased further.

 

Depending on the current configurations you have for ulimit of these hosts, it may require further investigation to understand why so many threads are being consumed by HBase. Over-utilized RegionServer instances can be indicative of hotspotting regions due to poor rowkey/table design, so understanding and comparing the number of requests per region amongst all servers is imperative to identifying hotspotting regions.

Announcements

Our community is getting a little larger. And a lot better.


Learn More about the Cloudera and Hortonworks community merger planned for late July and early August.