Reply
New Contributor
Posts: 11
Registered: ‎02-11-2014

HRegionInfo was null or empty in Meta with HBase and YCSB

Hi.

 

I have a cluster with 4 servers m1.xlarge. (3 Region Servers)

 

I'm seeing these errors when executing.

~/ycsb-0.1.4/bin/ycsb load hbase -P ~/ycsb-0.1.4/workloads/workloada -p columnfamily=family -p recordcount=209715200 -p threadcount=4 -s

 

14/02/12 22:27:20 WARN client.HConnectionManager$HConnectionImplementation: Encountered problems when prefetch hbase:meta table:
java.io.IOException: HRegionInfo was null or empty in Meta for usertable, row=usertable,user4150970021147505785,99999999999999
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:152)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1102)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1162)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1054)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1011)
at org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:349)
at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:477)
at org.apache.hadoop.hbase.client.AsyncProcess.logAndResubmit(AsyncProcess.java:686)
at org.apache.hadoop.hbase.client.AsyncProcess.receiveMultiAction(AsyncProcess.java:782)
at org.apache.hadoop.hbase.client.AsyncProcess.access$300(AsyncProcess.java:90)
at org.apache.hadoop.hbase.client.AsyncProcess$1.run(AsyncProcess.java:530)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
14/02/12 22:27:20 WARN client.HConnectionManager$HConnectionImplementation: Encountered problems when prefetch hbase:meta table:
java.io.IOException: HRegionInfo was null or empty in Meta for usertable, row=usertable,user4907736609192555414,99999999999999
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:152)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1102)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1162)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1054)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1011)
at org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:349)
at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:477)
at org.apache.hadoop.hbase.client.AsyncProcess.logAndResubmit(AsyncProcess.java:686)
at org.apache.hadoop.hbase.client.AsyncProcess.receiveMultiAction(AsyncProcess.java:782)
at org.apache.hadoop.hbase.client.AsyncProcess.access$300(AsyncProcess.java:90)
at org.apache.hadoop.hbase.client.AsyncProcess$1.run(AsyncProcess.java:530)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)

 

 

I don't know if it's too much workload for the server

 

I also see these messages:

14/02/12 22:31:55 INFO client.AsyncProcess: #3, table=usertable, attempt=15/35 failed 445 ops, last exception: org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, regionName=usertable,user6185654260443757784,1392244009628.b1dc62b4029ef31f2d03384bfb1ec28a., server=ip-10-238-168-39.ec2.internal,60020,1392175489003, memstoreSize=272166720, blockingMemStoreSize=268435456
at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2561)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1963)
at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4048)
at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3359)
at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3263)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)
on ip-10-238-168-39.ec2.internal,60020,1392175489003, tracking started Wed Feb 12 22:30:05 UTC 2014, retrying after 20030 ms, replay 445 ops.
14/02/12 22:31:55 INFO client.AsyncProcess: #4, table=usertable, attempt=15/35 failed 60 ops, last exception: org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, regionName=usertable,user6185654260443757784,1392244009628.b1dc62b4029ef31f2d03384bfb1ec28a., server=ip-10-238-168-39.ec2.internal,60020,1392175489003, memstoreSize=272166720, blockingMemStoreSize=268435456
at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2561)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1963)
at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4048)
at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3359)
at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3263)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)
on ip-10-238-168-39.ec2.internal,60020,1392175489003, tracking started Wed Feb 12 22:30:06 UTC 2014, retrying after 20070 ms, replay 60 ops.

 

Regards!

Cloudera Employee
Posts: 35
Registered: ‎09-04-2013

Re: HRegionInfo was null or empty in Meta with HBase and YCSB

You could have run into an issue similar to HBASE-10349. Have you pre-split your table? If you pre-split your table into enough regions, you should not run into this issue.

New Contributor
Posts: 11
Registered: ‎02-11-2014

Re: HRegionInfo was null or empty in Meta with HBase and YCSB

Thanks.
That worked.

As reference, I will leave the command I used to create the usertable.

create 'usertable', 'family', {SPLITS => (1..200).map {|i| "user#{1000+i*(9999-1000)/200}"}, MAX_FILESIZE => 4*1024**3}
Highlighted
Cloudera Employee
Posts: 35
Registered: ‎09-04-2013

Re: HRegionInfo was null or empty in Meta with HBase and YCSB

Cool, it's great to know it helps. Thanks for sharing the command used.

Announcements