Support Questions

Find answers, ask questions, and share your expertise

hbase PriorityRpcServer.handler Waiting for a call (since 406hrs)

avatar
Explorer

Hi admin users, I am using hbase cluster CM 5.7. often i check the hmaster status thru webUI. i noticed something is waiting for queue to process. does anyone help me understand what this means. what is this waiting for??

 

Master: hmaster01.abc.net

Fri Jul 29 12:55:49 CDT 2016B.defaultRpcServer.handler=29,queue=2,port=60000WAITING (since 12sec ago)Waiting for a call (since 12sec ago)
Fri Jul 29 12:55:49 CDT 2016PriorityRpcServer.handler=0,queue=0,port=60000WAITING (since 406hrs, 23mins, 18sec ago)Waiting for a call (since 406hrs, 23mins, 18sec ago)
Fri Jul 29 12:55:49 CDT 2016PriorityRpcServer.handler=1,queue=1,port=60000WAITING (since 406hrs, 23mins, 18sec ago)Waiting for a call (since 406hrs, 23mins, 18sec ago)
 
 
1 ACCEPTED SOLUTION

avatar
Mentor
The cause for crashes would be unrelated to this observance.

I'd recommend starting a new topic by posting the logs of the service that crashes for you, specifically the earliest FATAL message it produces before it aborts, if there is one.

View solution in original post

3 REPLIES 3

avatar
Mentor
The HMaster has two RPC handler pools, one for priority tasks, the other for regular ones. The regular ones see activity regularly and are represented by the section "B.defaultRpcServer" and you'll find that these continually process client or RS calls made to them.

The priority ones, on the other hand, only see limited activity relating to certain specific work, and will mostly be idle depending on your cluster's situation. It is normal to see a very large "Waiting since" value on the priority handlers, as they are used only in special situations.

Some more bits for your information:
- "Waiting for a call" means that the thread is idle and is waiting for some work to flow in.
- The "queue=X" value here is a queue identifier associated to the specific handler thread ID (there are multiple parallel queues that the handlers feed themselves from) and not the size of queue.

These waiting statuses on the HMaster UI only appear if you switch the Tasks tabs to show all monitored tasks or all RPC tasks regardless of state. If you're interested in finding actual calls, use the "Show Active RPC Calls" tab under it instead, and you may be able to catch some of the calls that regularly come in.

avatar
Explorer

so it is mostly like info kind message we dont need to worry. currently my cluster is occuring frequent crashes, so this post. 

 

Thanks for clarification!!

avatar
Mentor
The cause for crashes would be unrelated to this observance.

I'd recommend starting a new topic by posting the logs of the service that crashes for you, specifically the earliest FATAL message it produces before it aborts, if there is one.