Kudu Tablet Server High CPU

Question

I have a cluster with 3 tservers. When running a workload that heavily reads from the cluster, 1 of the 3 tservers is reaching nearly 100% CPU utilization while the other two are less than 10%. The tablets are equally balanced amongst the 3.

I am thinking that by chance, all my data used in this particular workload happens to reside on the 1 tserver.

thoughts? How might I diagnose this further?

adar · Accepted Answer

No, the rebalancer doesn't fix leader skew. It may in a future release. Leaders can cluster onto one tserver when individual tservers are restarted; if you restart the entire cluster all at once you might be able to redistribute leadership more evenly.

You're right that if you're only using one host to initiate reads, the reads will go to the local tserver rather than round-robin across the cluster. The master doesn't directly tell where clients to scan; it just provides them with enough information to make that decision based on their replica selection policy. There's also no way to do round robin (or randomized) replica selection.

adar · Answer

Before we get into metrics and other lower-level troubleshooting techniques, let's start with how you're reading.

What are you using to read? If it's the raw Kudu API, are you using the LEADER_ONLY replica selection policy? If so, and if your three node cluster is heavily skewed so that the majority of leader replicas are on one node, it's possible for that node to be servicing the majority of your scans.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.