Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Regions dying too many regions in transition

Regions dying too many regions in transition

Contributor

 

Getting the following stack and I don't know what is causing it

We are using CDH 5.4.8 hbase 1.0 on Ubuntu 14.04

we use storm to feed hbase data via thrift API 

 

2015-11-09 18:45:33,286 ERROR org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: Unable to accept edit because:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 5 actions: RegionOpeningException: 2 times, IOException: 3 times,
	at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:227)
	at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:207)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.getErrors(AsyncProcess.java:1563)
	at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:1003)
	at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:1017)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.batch(ReplicationSink.java:236)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.replicateEntries(ReplicationSink.java:160)
	at org.apache.hadoop.hbase.replication.regionserver.Replication.replicateLogEntries(Replication.java:198)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.replicateWALEntry(RSRpcServices.java:1584)
	at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20880)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
	at java.lang.Thread.run(Thread.java:745)

 

Any help would be appreciated

 

3 REPLIES 3

Re: Regions dying too many regions in transition

Master Guru
Given the error comes from ReplicationSink, it appears that the target
cluster (i.e. not the one you are seeing these errors on, but the other
'peer' cluster you've added, or a Lily-HBase indexer if you have one such
configured) is not accepting the writes due to some errors on its own side.
Your next step would be to look at this other cluster's RS logs (of the
indexer service logs) to see what the true problem is.

Re: Regions dying too many regions in transition

Contributor

Thanks for the feedback 

yes I did remove my peers 

makes sense

Re: Regions dying too many regions in transition

Master Guru
Have you checked the Master log to see if there are a lot of splits in progress? If so, try increasing the split threshold limit for your table to avoid it.