I have a cluster with 3 tservers. When running a workload that heavily reads from the cluster, 1 of the 3 tservers is reaching nearly 100% CPU utilization while the other two are less than 10%. The tablets are equally balanced amongst the 3.
I am thinking that by chance, all my data used in this particular workload happens to reside on the 1 tserver.
thoughts? How might I diagnose this further?
Before we get into metrics and other lower-level troubleshooting techniques, let's start with how you're reading.
What are you using to read? If it's the raw Kudu API, are you using the LEADER_ONLY replica selection policy? If so, and if your three node cluster is heavily skewed so that the majority of leader replicas are on one node, it's possible for that node to be servicing the majority of your scans.
Will the rebalancer distribute the leaders evenly amongst the cluster. It is not clear from the docs, seems it only balances the replica's which should result in leaders also being balanced as well?
Let’s say “I only have 1 host reading from the cluster and I select closest_replica. Won’t I end up in the same situation? How does the master distribute load? IP address hash? Can I change this to RR or this something controlled from the client side?
For others reading this post, I was able to identify the leader tablet distribution without using the rebalance tool. I am on an older Kudu that does not provide the tool.
I was able to copy and paste the live tablet info from the UI of the t-servers into excel and found that 95% of the of leaders are on the first t-server.