Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HBase-FailedSanityCheckException: Requested row out of range for doMiniBatchMutation

Highlighted

HBase-FailedSanityCheckException: Requested row out of range for doMiniBatchMutation

New Contributor

Hello HCC,

We are reading data/messages from Kafka and writing into HBase, while writing data into HBase we are seeing below exceptions in the logs. As mentioned in one of the HCC article from HortonWorks engineers we ran HBase Hbck and repair holes command but none of them helped we are still seeing exceptions. The namespace got over 396 online regions.

It was working fine for last couple of months, but recently we started to see this exceptions.

we are HDP:2.5.3.0-37

HBase:Version 1.1.2.2.5.3.0-37

is it something related to https://issues.apache.org/jira/browse/HBASE-17276 ?

any help will be greatly appreciated

@Kuldeep Kulkarni @Rohan Pednekar @Geoffrey Shelton Okot @Aditya Sirna @Jay Kumar SenSharma @kkanchu @Artem Ervits @Enis @Joe Young @Orlando Teixeira

org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: org.apache.hadoop.hbase.exceptions.FailedSanityCheckException: Requested row out of range for doMiniBatchMutation on HRegion prodnamespace:contactdetails-v2,69498218,1518378381699.8b7a1df21ed54a47facb99845845305a., startKey='69498218', getEndKey()='697452052100488', row='535021032017430'
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:729)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:679)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2056)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2127)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
	at java.lang.Thread.run(Thread.java:745)
: 1 time, servers with issues: xyz.testing.com,16020,1518641778110, 
	at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:260)
	at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$2400(AsyncProcess.java:240)
	at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1886)
	at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:247)
	at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:197)
	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1461)
	at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1029)
	at com.tgt.move.WlmInvData$$anonfun$main$1$$anonfun$apply$1.apply(WlmInvData.scala:129)
	at com.tgt.move.WlmInvData$$anonfun$main$1$$anonfun$apply$1.apply(WlmInvData.scala:70)
	at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
	at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2069)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2069)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:108)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Don't have an account?
Coming from Hortonworks? Activate your account here