Reply
Explorer
Posts: 9
Registered: ‎03-30-2016

HBASE throwing weird error.

Hello Members

We are having a weird problem with HBASE. We connect to HBASE manually and fire of queries using the API and everything works perfect. When we integrate it to a Spring web application, it starts throwing errors on the second call. Can anyone help? Here is the stack trace.

 

authToken: - #3, Caught throwable while calling. This is unexpected. Retrying. Server is dend1b31bda03.test.tiaa-cref.org,60020,1453819888718, tableName=HomePageStore
java.lang.RuntimeException: java.lang.RuntimeException: Cannot get region location for multi-region request
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208) ~[hbase-client-1.0.0-cdh5.4.0.jar:na]
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:701) ~[hbase-client-1.0.0-cdh5.4.0.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_51]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
Caused by: java.lang.RuntimeException: Cannot get region location for multi-region request
at org.apache.hadoop.hbase.client.MultiServerCallable.getLocation(MultiServerCallable.java:72) ~[hbase-client-1.0.0-cdh5.4.0.jar:na]
at org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java) ~[hbase-client-1.0.0-cdh5.4.0.jar:na]
at org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:53) ~[hbase-client-1.0.0-cdh5.4.0.jar:na]
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) ~[hbase-client-1.0.0-cdh5.4.0.jar:na]
... 6 common frames omitted

 

 

We went through the source code of the HBASE client API - and for some reason, it is creating an instance of 

https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/clien...

and inserting to the SingleServerRequestRunnable, 

 

(https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/clien... - Line 761)

 

If we look at this class - it throws a RuntimeException...

 

@Override
 protected HRegionLocation getLocation() {
 throw new RuntimeException("Cannot get region location for multi-region request");
 

}

 

 

 

Any ideas as to what makes it use this instance of the class?

 

Thanks

Venky

Highlighted
Explorer
Posts: 9
Registered: ‎03-30-2016

Re: HBASE throwing weird error.

Any ideas?

When we do a singular GET call - it works perfectly. When we try to do a batched get call by passing a list of GET objects - it throws this weird error and exits.

 

java.lang.RuntimeException: java.lang.RuntimeException: Cannot get region location for multi-region request
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208) ~[hbase-client-1.0.0-cdh5.4.0.jar:?]
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:701) [hbase-client-1.0.0-cdh5.4.0.jar:?]
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:953) [hbase-client-1.0.0-cdh5.4.0.jar:?]
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:862) [hbase-client-1.0.0-cdh5.4.0.jar:?]
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1156) [hbase-client-1.0.0-cdh5.4.0.jar:?]
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1123) [hbase-client-1.0.0-cdh5.4.0.jar:?]
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:574) [hbase-client-1.0.0-cdh5.4.0.jar:?]
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:711) [hbase-client-1.0.0-cdh5.4.0.jar:?]
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:953) [hbase-client-1.0.0-cdh5.4.0.jar:?]
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:862) [hbase-client-1.0.0-cdh5.4.0.jar:?]
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1156) [hbase-client-1.0.0-cdh5.4.0.jar:?]
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1123) [hbase-client-1.0.0-cdh5.4.0.jar:?]
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:574) [hbase-client-1.0.0-cdh5.4.0.jar:?]
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:711) [hbase-client-1.0.0-cdh5.4.0.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_51]
	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_51]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_51]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_51]
	at java.lang.Thread.run(Thread.java:744) [?:1.7.0_51]
Caused by: java.lang.RuntimeException: Cannot get region location for multi-region request
	at org.apache.hadoop.hbase.client.MultiServerCallable.getLocation(MultiServerCallable.java:72) ~[hbase-client-1.0.0-cdh5.4.0.jar:?]
	at org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java) ~[hbase-client-1.0.0-cdh5.4.0.jar:?]
	at org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:53) ~[hbase-client-1.0.0-cdh5.4.0.jar:?]
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) ~[hbase-client-1.0.0-cdh5.4.0.jar:?]

Explorer
Posts: 9
Registered: ‎03-30-2016

Re: HBASE throwing weird error.

Some more information. THis happens ONLY when running inside Weblogic server - The same code works in tomcat with no issues. Does anyone think that the ExecutorService customizations done in WLS can make this fail?

Expert Contributor
Posts: 101
Registered: ‎01-24-2014

Re: HBASE throwing weird error.

I'll agree, seems like something wrong with weblogic or weblogic config

 

here is where that error comes from, looks like something is calling getLocation() in the Multi and that is what throws that exception.

 

https://hbase.apache.org/devapidocs/src-html/org/apache/hadoop/hbase/client/MultiServerCallable.html

 

052class MultiServerCallable<R> extends PayloadCarryingServerCallable<MultiResponse> {
053 private final MultiAction<R> multiAction;
054 private final boolean cellBlock;
055
056 MultiServerCallable(final ClusterConnection connection, final TableName tableName,
057 final ServerName location, RpcControllerFactory rpcFactory, final MultiAction<R> multi) {
058 super(connection, tableName, null, rpcFactory);
059 this.multiAction = multi;
060 // RegionServerCallable has HRegionLocation field, but this is a multi-region request.
061 // Using region info from parent HRegionLocation would be a mistake for this class; so
062 // we will store the server here, and throw if someone tries to obtain location/regioninfo.
063 this.location = new HRegionLocation(null, location);
064 this.cellBlock = isCellBlock();
065 }
066
067 @Override
068 protected HRegionLocation getLocation() {
069 throw new RuntimeException("Cannot get region location for multi-region request");
070 }
071
072 @Override
073 public HRegionInfo getHRegionInfo() {
074 throw new RuntimeException("Cannot get region info for multi-region request");
075 }
076
077 MultiAction<R> getMulti() {
078 return this.multiAction;
079 }
080

Announcements

Our community is getting a little larger. And a lot better.


Learn More about the Cloudera and Hortonworks community merger planned for late July and early August.