Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error when stressing the cluster

Solved Go to solution

Error when stressing the cluster

Explorer

Hi,

 

We are stressing the Kudu cluster (inserting a lot of information) and we are getting errors of timeouts when inserting the data in the tablets:

 

 

W0807 12:53:47.136150 31391 meta_cache.cc:207] Tablet d687c05ffe5e48d19fbfe2f71bd136f7: Replica 0cf3c1866a094ee0b2305bca770f5e70 (bigdata09dev:7050) has failed: Timed out: Write RPC to 192.168.10.124:7050 timed out after 9.989s (SENT)
W0807 12:53:47.136211 31391 batcher.cc:329] Timed out: Failed to write batch of 805 ops to tablet d687c05ffe5e48d19fbfe2f71bd136f7 after 1 attempt(s): Failed to write to server: 0cf3c1866a094ee0b2305bca770f5e70 (bigdata09dev:7050): Write RPC to 192.168.10.124:7050 timed out after 9.989s (SENT)

 

 

This is causing data loss. My question is: Is the only option to avoid this (avoid data loss) to control the errors by software when programming the loader and retrying the insert? Or is it possible to configure the cluster to retry the insert by default until it gets loaded?

 

Thank you very much and best regards

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Error when stressing the cluster

Expert Contributor
It looks like you're using the C++ client. Given that, you can use the
KuduSession::SetTimeout() API:

https://kudu.apache.org/cpp-client-api/classkudu_1_1client_1_1KuduSession.html#a25b22362650d7120f59c...

-Todd
3 REPLIES 3

Re: Error when stressing the cluster

Expert Contributor
Hi,

If you simply increase your timeout, the client itself has built-in retries
and will keep trying to complete the insert until the given time has
elapsed. In a scenario that is not latency-sensitive I would recommend
increasing the timeout to a minute or two.

-Todd

Re: Error when stressing the cluster

Explorer

Thanks a lot. And do you know how to change that timeout?

Highlighted

Re: Error when stressing the cluster

Expert Contributor
It looks like you're using the C++ client. Given that, you can use the
KuduSession::SetTimeout() API:

https://kudu.apache.org/cpp-client-api/classkudu_1_1client_1_1KuduSession.html#a25b22362650d7120f59c...

-Todd