Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HRegionInfo was null or empty in Meta with HBase and YCSB

HRegionInfo was null or empty in Meta with HBase and YCSB

New Contributor

Hi.

 

I have a cluster with 4 servers m1.xlarge. (3 Region Servers)

 

I'm seeing these errors when executing.

~/ycsb-0.1.4/bin/ycsb load hbase -P ~/ycsb-0.1.4/workloads/workloada -p columnfamily=family -p recordcount=209715200 -p threadcount=4 -s

 

14/02/12 22:27:20 WARN client.HConnectionManager$HConnectionImplementation: Encountered problems when prefetch hbase:meta table:
java.io.IOException: HRegionInfo was null or empty in Meta for usertable, row=usertable,user4150970021147505785,99999999999999
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:152)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1102)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1162)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1054)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1011)
at org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:349)
at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:477)
at org.apache.hadoop.hbase.client.AsyncProcess.logAndResubmit(AsyncProcess.java:686)
at org.apache.hadoop.hbase.client.AsyncProcess.receiveMultiAction(AsyncProcess.java:782)
at org.apache.hadoop.hbase.client.AsyncProcess.access$300(AsyncProcess.java:90)
at org.apache.hadoop.hbase.client.AsyncProcess$1.run(AsyncProcess.java:530)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
14/02/12 22:27:20 WARN client.HConnectionManager$HConnectionImplementation: Encountered problems when prefetch hbase:meta table:
java.io.IOException: HRegionInfo was null or empty in Meta for usertable, row=usertable,user4907736609192555414,99999999999999
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:152)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1102)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1162)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1054)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1011)
at org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:349)
at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:477)
at org.apache.hadoop.hbase.client.AsyncProcess.logAndResubmit(AsyncProcess.java:686)
at org.apache.hadoop.hbase.client.AsyncProcess.receiveMultiAction(AsyncProcess.java:782)
at org.apache.hadoop.hbase.client.AsyncProcess.access$300(AsyncProcess.java:90)
at org.apache.hadoop.hbase.client.AsyncProcess$1.run(AsyncProcess.java:530)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)

 

 

I don't know if it's too much workload for the server

 

I also see these messages:

14/02/12 22:31:55 INFO client.AsyncProcess: #3, table=usertable, attempt=15/35 failed 445 ops, last exception: org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, regionName=usertable,user6185654260443757784,1392244009628.b1dc62b4029ef31f2d03384bfb1ec28a., server=ip-10-238-168-39.ec2.internal,60020,1392175489003, memstoreSize=272166720, blockingMemStoreSize=268435456
at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2561)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1963)
at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4048)
at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3359)
at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3263)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)
on ip-10-238-168-39.ec2.internal,60020,1392175489003, tracking started Wed Feb 12 22:30:05 UTC 2014, retrying after 20030 ms, replay 445 ops.
14/02/12 22:31:55 INFO client.AsyncProcess: #4, table=usertable, attempt=15/35 failed 60 ops, last exception: org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, regionName=usertable,user6185654260443757784,1392244009628.b1dc62b4029ef31f2d03384bfb1ec28a., server=ip-10-238-168-39.ec2.internal,60020,1392175489003, memstoreSize=272166720, blockingMemStoreSize=268435456
at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2561)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1963)
at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4048)
at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3359)
at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3263)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)
on ip-10-238-168-39.ec2.internal,60020,1392175489003, tracking started Wed Feb 12 22:30:06 UTC 2014, retrying after 20070 ms, replay 60 ops.

 

Regards!

3 REPLIES 3

Re: HRegionInfo was null or empty in Meta with HBase and YCSB

Contributor

You could have run into an issue similar to HBASE-10349. Have you pre-split your table? If you pre-split your table into enough regions, you should not run into this issue.

Re: HRegionInfo was null or empty in Meta with HBase and YCSB

New Contributor
Thanks.
That worked.

As reference, I will leave the command I used to create the usertable.

create 'usertable', 'family', {SPLITS => (1..200).map {|i| "user#{1000+i*(9999-1000)/200}"}, MAX_FILESIZE => 4*1024**3}

Re: HRegionInfo was null or empty in Meta with HBase and YCSB

Contributor

Cool, it's great to know it helps. Thanks for sharing the command used.