Kudu Issue

[#1]

Hello All,

We are observing below error on our Kudu Master server, 

CDH / CM = 5.16.2
Kudu 1.7.0
memory_limit_hard_bytes = 35GB

W1125 14:29:26.059293 39728 service_pool.cc:130] GetTableSchema request on kudu.master.MasterService from KUDU MASTER IP:60098 dropped due to backpressure. The service queue is full; it has 50 items.
W1125 16:12:31.944316 14678 negotiation.cc:313] Failed RPC negotiation. Trace:
1125 16:12:28.943331 (+     0us) reactor.cc:577] Submitting negotiation task for server connection from 172.12.345.678:33960
1125 16:12:28.943495 (+   164us) server_negotiation.cc:176] Beginning negotiation
1125 16:12:28.943498 (+     3us) server_negotiation.cc:365] Waiting for connection header
1125 16:12:28.944585 (+  1087us) server_negotiation.cc:373] Connection header received
1125 16:12:31.944237 (+2999652us) negotiation.cc:304] Negotiation complete: Timed out: Server connection negotiation failed: server connection from 172.12.345.678:33960
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":54,"threads_started":1}
W1125 16:32:01.273672 26481 negotiation.cc:313] Failed RPC negotiation. Trace:
1125 16:31:58.273342 (+     0us) reactor.cc:577] Submitting negotiation task for server connection from 172.12.345.678:34488
1125 16:31:58.273518 (+   176us) server_negotiation.cc:176] Beginning negotiation
1125 16:31:58.273521 (+     3us) server_negotiation.cc:365] Waiting for connection header
1125 16:31:58.274365 (+   844us) server_negotiation.cc:373] Connection header received
1125 16:32:01.273565 (+2999200us) negotiation.cc:304] Negotiation complete: Timed out: Server connection negotiation failed: server connection from 172.12.345.678:34488
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":59,"threads_started":1}
W1125 16:42:01.186538 39726 service_pool.cc:130] GetTableSchema request on kudu.master.MasterService from TSERVER1:49284 dropped due to backpressure. The service queue is full; it has 50 items.
W1125 16:46:10.786643 39729 service_pool.cc:130] GetTableSchema request on kudu.master.MasterService from TSERVER2:34404 dropped due to backpressure. The service queue is full; it has 50 items.
W1125 17:40:18.836066 39726 service_pool.cc:130] GetTableSchema request on kudu.master.MasterService from TSERVER3:43694 dropped due to backpressure. The service queue is full; it has 50 items.

Request some assistance in resolving / fixing this issue 

Regards

Amn

[#2]

Hi Amn,

It seems there are too many requests to the MasterService which causes requests to be dropped when the service queue is full.

While this happens frequently with other services across TabletServers, it's unusual on the MasterService. To investigate this further I'd need to understand the access patterns that could be responsible for this. How are the clients accessing?

Attila

[#3]

Hi Attila,

Appreciate  your reply, not quite sure when you say 'how are clients accessing" as far as I know we use Impala to query kudu tables, if that makes sense…. still trying to learn the ropes of Kudu. Where and how can I check ?

Regards

Amn

[#4]

Hello @Amn_468 

Did you find a solution? In our case we have increased the service queue to 500, still we see the master flooded with backpressure issues pointing to service queue full.

Regards
SohamR

[#5]

@SohamR As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. Thanks