question Re: Kudu Tablet taking too long to sync with Kudu Master and Kudu Tablet using 90% memory in Support Questions

Kudu Tablet taking too long to sync with Kudu Master and Kudu Tablet using 90% memory

yagoaparecidoti — Mon, 28 Mar 2022 19:56:55 GMT

We would like to understand two behaviors that Kudu is showing.

1. When restarting the Kudu Tablet, it takes between 5 and 10min for this Kudu Tablet to communicate with the Kudu Master again, why is this happening?

2. There are 5 Kudu Tablets and two of those Kudu Tablets is using more than 90% of the memory that was set in "memory_limit_hard_bytes", why is this happening?

Re: Kudu Tablet taking too long to sync with Kudu Master and Kudu Tablet using 90% memory

jromero — Mon, 28 Mar 2022 21:01:25 GMT

With regards to your questions:

1. To further understand this behavior, it would be needed to check the tablet server logs during startup. Likely the block processing time is taking a good part of those 10 mins.

2. It could be possible that those 2 servers are overloaded, that can happen if the kudu cluster is not balanced.

To further understand, the output of the ksck and:

kudu table list ${MASTER_ADDRS} -list_tablets | grep "^ " | cut -d' ' -f6,7 | sort

Would throw a good output to gain a better understanding of your cluster situation.

If you can attach those here, would be great for further analysis.

Re: Kudu Tablet taking too long to sync with Kudu Master and Kudu Tablet using 90% memory

yagoaparecidoti — Tue, 29 Mar 2022 15:54:12 GMT

hi @jromero

kudu is balanced correctly.

there are 5 kudu tablet and a total of 1173 replicas and each kudu tablet has 234 tablets, this shows that it is balanced.

even being balanced, it presents these two mentioned problems.

Re: Kudu Tablet taking too long to sync with Kudu Master and Kudu Tablet using 90% memory

jromero — Tue, 29 Mar 2022 20:53:42 GMT

Hellio @yagoaparecidoti

Being balanced is a good thing. How about hotspotting?

Can you check in the kudu web page > Tablet servers > Click on those affected > check the metrics and the RPC pages on those two.

> How does it look?

> How many RPC calls compared to a healthy tablet server?

Re: Kudu Tablet taking too long to sync with Kudu Master and Kudu Tablet using 90% memory

yagoaparecidoti — Tue, 29 Mar 2022 21:03:59 GMT

Hi @jromero

the "metrics[1]" and "rpcs[2]" web pages are too long:

what should we actually look at?
what parameter do we need to look at?

[1] - http://ip_host:8050/metrics
[2] - http://ip_host:8050/rpcz

Re: Kudu Tablet taking too long to sync with Kudu Master and Kudu Tablet using 90% memory

jromero — Tue, 29 Mar 2022 21:28:02 GMT

Inbound/Outbound connections on the rpc page.

rpc_* metrics on the metrics page. Sorry I don't recall the exact metric names.

Re: Kudu Tablet taking too long to sync with Kudu Master and Kudu Tablet using 90% memory

yagoaparecidoti — Tue, 29 Mar 2022 23:53:04 GMT

hi @jromero

on the page "http://ip_host:8050/rpcz":

in "inbound_connections" shows the addresses of other tablet servers with "open" state.

in "outbound_connections" it shows the addresses of other tablet servers and master with "open" state.

this above also happens with other tablet servers.

I didn't find anything that could be related to the mentioned problems.

Re: Kudu Tablet taking too long to sync with Kudu Master and Kudu Tablet using 90% memory

jromero — Wed, 30 Mar 2022 19:27:21 GMT

Hi @yagoaparecidoti

Thanks, in that case further investigation will be needed, we would need to check what is happening in those 2 tablet servers. If you are able to share the logs from those TS that would be great, if not it will be quite hard to tell and your best bet would be to open a support case to have it checked.

> Are you able to check the charts on cloudera manager > Kudu > instances > tablet server > Chart library > Replicas? Can you compare those with a non affected TS?

Re: Kudu Tablet taking too long to sync with Kudu Master and Kudu Tablet using 90% memory

VidyaSargur — Mon, 04 Apr 2022 06:32:48 GMT

@yagoaparecidoti, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. If you are still experiencing the issue, can you provide the information @jromero has requested?

Re: Kudu Tablet taking too long to sync with Kudu Master and Kudu Tablet using 90% memory

yagoaparecidoti — Mon, 04 Apr 2022 15:40:54 GMT

hi @jromero

thanks for the feedback.

unfortunately we cannot make the TS logs available as they contain sensitive information.

we will try to open a ticket with support.

checking the TS replica charts "Total Tablet Size On Disk Across Kudu Replicas", they all contain the same size.