All the jobs are failing with exception`org.apache.hadoop.hbase.CallQueueTooBigException: Call queue is ful`

[#1]

Thu Jan 18 05:19:15 UTC 2018, RpcRetryingCaller{globalStartTime=1516252204444, pause=100, retries=35}, org.apache.hadoop.hbase.CallQueueTooBigException: Call queue is full on /ip:16020, too many items queued ?

	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147)
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:107)
	at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:56)
	at org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService$Stub.getVersion(MetaDataProtos.java:14699)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl$6.call(ConnectionQueryServicesImpl.java:1209)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl$6.call(ConnectionQueryServicesImpl.java:1201)
	at org.apache.hadoop.hbase.client.HTable$15.call(HTable.java:1724)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.CallQueueTooBigException: Call queue is full on /my_ip:16020, too many items queued ?
	at sun.reflect.GeneratedConstructorAccessor8.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:334)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1632)
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:104)
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:94)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
	... 10 more
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException): Call queue is full on /myip:16020, too many items queued ?
	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1225)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:32855)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1628)

[#2]

you can check queue size from regionserver UI (host-address:16030/rs-status) under Queues tab.

Queue size increases when there are too many requests and your RS cannot handle it. This can happen in case of

• too many requests (CPU crunch)
• ongoing compation
• ongoing GC

You should investigate and identify the cause. and accordingly you can have more cpu, schedule compactions during off-peak hours, optimize GC.

Hope it helps!

[#3]

We had the same error. Additional to the answer above we did change specific the following properties:

hbase-site.xml

hbase.regionserver.handler.count from 30 to 40

phoenix.regionserver.index.handler.count from 30 to 40

hbase-env

add (at export HBASE_REGIONSERVER_OPTS) -XX:ParallelGCThreads=8

----------------

Additional you can increase the hbase.ipc.server.max.callqueue.size (default 1GB)

Regards,

Michael

[#4]

Be careful with setting ParallelGCThreads to 8 as it will decrease threads if the system has more processing power.

=> https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/parallel.html

On a machine with N hardware threads where N is greater than 8, the parallel collector uses a fixed fraction of N as the number of garbage collector threads. The fraction is approximately 5/8 for large values of N. At values of N below 8, the number used is N. On selected platforms, the fraction drops to 5/16. The specific number of garbage collector threads can be adjusted with a command-line option (which is described later). On a host with one processor, the parallel collector will likely not perform as well as the serial collector because of the overhead required for parallel execution (for example, synchronization). However, when running applications with medium-sized to large-sized heaps, it generally outperforms the serial collector by a modest amount on machines with two processors, and usually performs significantly better than the serial collector when more than two processors are available.