Reply
Contributor
Posts: 126
Registered: ‎10-15-2014

Regions dying too many regions in transition

 

Getting the following stack and I don't know what is causing it

We are using CDH 5.4.8 hbase 1.0 on Ubuntu 14.04

we use storm to feed hbase data via thrift API 

 

2015-11-09 18:45:33,286 ERROR org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: Unable to accept edit because:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 5 actions: RegionOpeningException: 2 times, IOException: 3 times,
	at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:227)
	at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:207)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.getErrors(AsyncProcess.java:1563)
	at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:1003)
	at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:1017)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.batch(ReplicationSink.java:236)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.replicateEntries(ReplicationSink.java:160)
	at org.apache.hadoop.hbase.replication.regionserver.Replication.replicateLogEntries(Replication.java:198)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.replicateWALEntry(RSRpcServices.java:1584)
	at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20880)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
	at java.lang.Thread.run(Thread.java:745)

 

Any help would be appreciated

 

Posts: 1,896
Kudos: 433
Solutions: 303
Registered: ‎07-31-2013

Re: Regions dying too many regions in transition

Given the error comes from ReplicationSink, it appears that the target
cluster (i.e. not the one you are seeing these errors on, but the other
'peer' cluster you've added, or a Lily-HBase indexer if you have one such
configured) is not accepting the writes due to some errors on its own side.
Your next step would be to look at this other cluster's RS logs (of the
indexer service logs) to see what the true problem is.

Contributor
Posts: 126
Registered: ‎10-15-2014

Re: Regions dying too many regions in transition

Thanks for the feedback 

yes I did remove my peers 

makes sense

Posts: 1,896
Kudos: 433
Solutions: 303
Registered: ‎07-31-2013

Re: Regions dying too many regions in transition

Have you checked the Master log to see if there are a lot of splits in progress? If so, try increasing the split threshold limit for your table to avoid it.
Announcements