Kudu t-server online issues

[#1]

Hello,

We have an upcoming OS patching activity for our cluster and we would be bringing the cluster down, in past when we have conducted such activity we have seen that Kudu T-Servers take a long time to come back online/ live around 4-5 hours for all t-servers to be online

Upon reviewing the logs we find that it continues to give message ‘Opened block log’ etc. Would like to know if there is a way to quicken this process to reduce the time all t-servers are online

CM/ CDH- 5.16.2
T-Servers – 10 / Master – 3

Appreciate any help/guidance.

Thanks

[#2]

Hi Team,

I believe your issue has not been resolved till now hence i am pointing toward a direction where you need to look..!!

if you TS is taking 3-4 hours it means there is a massive overload on them, Please check the scheme/scala from kudu runbook

https://kudu.apache.org/docs/scaling_guide.html

Points need to check:

1) Please check how many Tablet per TS you have, we don't recommend more than 2K per TS post replication

2) Does your master WAL and Data directory is on same disk, as its not recommended too

3) How many data directory you have per TS, keep from 3-12..!!

4) Please run your kudu cluster ksck <master_addrss>  command and check if your cluster is in healthy state..!!

Thanks..!!