Support Questions
Find answers, ask questions, and share your expertise

HBase-FailedSanityCheckException: Requested row out of range for doMiniBatchMutation

HBase-FailedSanityCheckException: Requested row out of range for doMiniBatchMutation

Explorer

Hello HCC,

We are reading data/messages from Kafka and writing into HBase, while writing data into HBase we are seeing below exceptions in the logs. As mentioned in one of the HCC article from HortonWorks engineers we ran HBase Hbck and repair holes command but none of them helped we are still seeing exceptions. The namespace got over 396 online regions.

It was working fine for last couple of months, but recently we started to see this exceptions.

we are HDP:2.5.3.0-37

HBase:Version 1.1.2.2.5.3.0-37

is it something related to https://issues.apache.org/jira/browse/HBASE-17276 ?

any help will be greatly appreciated

@Kuldeep Kulkarni @Rohan Pednekar @Geoffrey Shelton Okot @Aditya Sirna @Jay Kumar SenSharma @kkanchu @Artem Ervits @Enis @Joe Young @Orlando Teixeira

org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: org.apache.hadoop.hbase.exceptions.FailedSanityCheckException: Requested row out of range for doMiniBatchMutation on HRegion prodnamespace:contactdetails-v2,69498218,1518378381699.8b7a1df21ed54a47facb99845845305a., startKey='69498218', getEndKey()='697452052100488', row='535021032017430'
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:729)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:679)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2056)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2127)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
	at java.lang.Thread.run(Thread.java:745)
: 1 time, servers with issues: xyz.testing.com,16020,1518641778110, 
	at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:260)
	at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$2400(AsyncProcess.java:240)
	at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1886)
	at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:247)
	at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:197)
	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1461)
	at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1029)
	at com.tgt.move.WlmInvData$$anonfun$main$1$$anonfun$apply$1.apply(WlmInvData.scala:129)
	at com.tgt.move.WlmInvData$$anonfun$main$1$$anonfun$apply$1.apply(WlmInvData.scala:70)
	at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
	at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2069)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2069)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:108)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Don't have an account?