Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

All the jobs are failing with exception`org.apache.hadoop.hbase.CallQueueTooBigException: Call queue is ful`

All the jobs are failing with exception`org.apache.hadoop.hbase.CallQueueTooBigException: Call queue is ful`

New Contributor

Thu Jan 18 05:19:15 UTC 2018, RpcRetryingCaller{globalStartTime=1516252204444, pause=100, retries=35}, org.apache.hadoop.hbase.CallQueueTooBigException: Call queue is full on /ip:16020, too many items queued ?

	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147)
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:107)
	at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:56)
	at org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService$Stub.getVersion(MetaDataProtos.java:14699)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl$6.call(ConnectionQueryServicesImpl.java:1209)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl$6.call(ConnectionQueryServicesImpl.java:1201)
	at org.apache.hadoop.hbase.client.HTable$15.call(HTable.java:1724)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.CallQueueTooBigException: Call queue is full on /my_ip:16020, too many items queued ?
	at sun.reflect.GeneratedConstructorAccessor8.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:334)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1632)
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:104)
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:94)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
	... 10 more
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException): Call queue is full on /myip:16020, too many items queued ?
	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1225)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:32855)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1628)
2 REPLIES 2

Re: All the jobs are failing with exception`org.apache.hadoop.hbase.CallQueueTooBigException: Call queue is ful`

Contributor

you can check queue size from regionserver UI (host-address:16030/rs-status) under Queues tab.

Queue size increases when there are too many requests and your RS cannot handle it. This can happen in case of

  • too many requests (CPU crunch)
  • ongoing compation
  • ongoing GC

You should investigate and identify the cause. and accordingly you can have more cpu, schedule compactions during off-peak hours, optimize GC.

Hope it helps!

Highlighted

Re: All the jobs are failing with exception`org.apache.hadoop.hbase.CallQueueTooBigException: Call queue is ful`

New Contributor

We had the same error. Additional to the answer above we did change specific the following properties:

hbase-site.xml

hbase.regionserver.handler.count from 30 to 40

phoenix.regionserver.index.handler.count from 30 to 40

hbase-env

add (at export HBASE_REGIONSERVER_OPTS) -XX:ParallelGCThreads=8

----------------

Additional you can increase the hbase.ipc.server.max.callqueue.size (default 1GB)

Regards,

Michael

Don't have an account?
Coming from Hortonworks? Activate your account here