Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kudu Issue

Highlighted

Kudu Issue

Explorer

 

 

Hello All,

We are observing below error on our Kudu Master server, 

CDH / CM = 5.16.2
Kudu 1.7.0
memory_limit_hard_bytes = 35GB

 

W1125 14:29:26.059293 39728 service_pool.cc:130] GetTableSchema request on kudu.master.MasterService from KUDU MASTER IP:60098 dropped due to backpressure. The service queue is full; it has 50 items.
W1125 16:12:31.944316 14678 negotiation.cc:313] Failed RPC negotiation. Trace:
1125 16:12:28.943331 (+     0us) reactor.cc:577] Submitting negotiation task for server connection from 172.12.345.678:33960
1125 16:12:28.943495 (+   164us) server_negotiation.cc:176] Beginning negotiation
1125 16:12:28.943498 (+     3us) server_negotiation.cc:365] Waiting for connection header
1125 16:12:28.944585 (+  1087us) server_negotiation.cc:373] Connection header received
1125 16:12:31.944237 (+2999652us) negotiation.cc:304] Negotiation complete: Timed out: Server connection negotiation failed: server connection from 172.12.345.678:33960
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":54,"threads_started":1}
W1125 16:32:01.273672 26481 negotiation.cc:313] Failed RPC negotiation. Trace:
1125 16:31:58.273342 (+     0us) reactor.cc:577] Submitting negotiation task for server connection from 172.12.345.678:34488
1125 16:31:58.273518 (+   176us) server_negotiation.cc:176] Beginning negotiation
1125 16:31:58.273521 (+     3us) server_negotiation.cc:365] Waiting for connection header
1125 16:31:58.274365 (+   844us) server_negotiation.cc:373] Connection header received
1125 16:32:01.273565 (+2999200us) negotiation.cc:304] Negotiation complete: Timed out: Server connection negotiation failed: server connection from 172.12.345.678:34488
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":59,"threads_started":1}
W1125 16:42:01.186538 39726 service_pool.cc:130] GetTableSchema request on kudu.master.MasterService from TSERVER1:49284 dropped due to backpressure. The service queue is full; it has 50 items.
W1125 16:46:10.786643 39729 service_pool.cc:130] GetTableSchema request on kudu.master.MasterService from TSERVER2:34404 dropped due to backpressure. The service queue is full; it has 50 items.
W1125 17:40:18.836066 39726 service_pool.cc:130] GetTableSchema request on kudu.master.MasterService from TSERVER3:43694 dropped due to backpressure. The service queue is full; it has 50 items.

 

 

Request some assistance in resolving / fixing this issue 

 

Regards

Amn

2 REPLIES 2

Re: Kudu Issue

Cloudera Employee

Hi Amn,

 

It seems there are too many requests to the MasterService which causes requests to be dropped when the service queue is full.

 

While this happens frequently with other services across TabletServers, it's unusual on the MasterService. To investigate this further I'd need to understand the access patterns that could be responsible for this. How are the clients accessing?

 

Attila

Re: Kudu Issue

Explorer

Hi Attila,

 

Appreciate  your reply, not quite sure when you say 'how are clients accessing" as far as I know we use Impala to query kudu tables, if that makes sense…. still trying to learn the ropes of Kudu. Where and how can I check ?

 

Regards

Amn

Don't have an account?
Coming from Hortonworks? Activate your account here