Replace your Kudu Disks (single node)
Summary
After you experience a disk failure on a worker node then replace the disk, you’ll need to ensure that the disk is suitably rebalanced within the Kudu Service at the local level.
Investigation & Resolution
Purging a Tablet Server
There isn’t currently a method to rebalance the replicas on a single Tablet Server disk array. This means that we need to empty the node and reintroduce it so that it can be used again from scratch. We begin by quiescing the Tablet Server.
Quiesce the Tablet Server
Quiesce essentially means to stop the Tablet Server from hosting any leaders in order to:
- Make other replicas on live Tablet Servers become the leaders
- Prevent this Tablet Server from becoming a leader for any other reason
- Allow this Tablet Server to be read from (the replicas that are still present)
Check Quiesce Status
| sudo -u kudu kudu tserver quiesce status <Worker-Node-FQDN> Quiescing | Tablet Leaders | Active Scanners -----------+----------------+----------------- true | 0 | 0 |
Quiesce Start
| sudo -u kudu kudu tserver quiesce start <Worker-Node-FQDN> |
Put the Tablet Server into Maintenance Mode
Maintenance Mode stops the Tablet Server from being used completely. The maintenance mode commands require you to retrieve the UUID of the Tablet Server first. We can get this information from a tserver list command:
| sudo -u kudu kudu tserver list <Master-Node1-FQDN>,<Master-Node2-FQDN>,<Master-Note 3-FQDN> |
An example that then targets the server you want to work on
| sudo -u kudu kudu tserver list <Master-Node1-FQDN>,<Master-Node2-FQDN>,<Master-Node3-FQDN> | grep <Worker-Node-FQDN> 5e103ac84707495e843a4553ac622f20 | <Worker-Node-FQDN>:7050 |
Put the Tablet Server into Maintenance Mode
| sudo -u kudu kudu tserver state enter_maintenance <Master-Node1-FQDN>,<Master-Node2-FQDN>,<Master-Note 3-FQDN> 5e103ac84707495e843a4553ac622f20 |
Exit the Tablet Server from Maintenance Mode
| sudo -u kudu kudu tserver state exit_maintenance <Master-Node1-FQDN>,<Master-Node2-FQDN>,<Master-Note 3-FQDN> 5e103ac84707495e843a4553ac622f20 |
Run ksck to check the status of Kudu Service / TS to be purged
This will confirm the status of both Quiesce and Maintenance Mode for every Tablet Server in the cluster, (in our example - <Worker-Node-FQDN>😞
| sudo -u kudu kudu cluster ksck <Master-Node1-FQDN>,<Master-Node2-FQDN>,<Master-Node3-FQDN> 2>&1 | tee ksck.out |
The above command outputs the ksck to both the terminal and a file called ‘ksck.out’. This allows us to review the information from both perspectives and also create a record of the output in the file. But taking our example of purging <Worker-Node-FQDN> into account, the following information is key:
Tablet Server Summary
This is a list of all Tablet Servers in the cluster. We’ve focused on just <Worker-Node-FQDN> and the surrounding TS’s for illustrative purposes. Notice the text in RED - <Worker-Node-FQDN> is quiescing and has no leaders running on it.
| Tablet Server Summary UUID | Address | Status | Location | Quiescing | Tablet Leaders | Active Scanners ----------------------------------+---------------------------------+---------+-------------+-----------+----------------+----------------- … 59e6ca5107754c24b649ee9c9acfccfb | <Worker-Node-FQDN>:7050 | HEALTHY | /CabinetE01 | false | 47 | 0 5e103ac84707495e843a4553ac622f20 | <Worker-Node-FQDN>:7050 | HEALTHY | /CabinetA08 | true | 0 | 0 5edf82f0516b4897b3a7991a7e67d71c | <Worker-Node-FQDN>:7050 | HEALTHY | /CabinetA07 | false | 1452 | 0 … |
Tablet Server State (maintenance mode)
This section shows that the TS is in maintenance mode.
| Tablet Server States Server | State ----------------------------------+------------------ 5e103ac84707495e843a4553ac622f20 | MAINTENANCE_MODE |
Purge the Tablet Server
The following command instructs kudu to ignore the <Worker-Node-FQDN> node AND move replicas away from it:
| sudo -u kudu /tmp/kudu cluster rebalance <Master-Node1-FQDN>,<Master-Node2-FQDN>,<Master-Note 3-FQDN> -ignored_tservers=5e103ac84707495e843a4553ac622f20 -move_replicas_from_ignored_tservers |
Again, importantly, the Tablet Server has to have been successfully quiesced and put into maintenance mode to avoid any issues with the Kudu service.
A simple break in VPN or shell terminal will kill the rebalance command. This won't affect Kudu, but it will stop the process. In order to work around this and retain information during the process, use the following command to output the rebalance status into the active terminal session as well as a file:
| sudo -u kudu /tmp/kudu cluster rebalance <Master-Node1-FQDN>,<Master-Node2-FQDN>,<Master-Node3-FQDN> -ignored_tservers=5e103ac84707495e843a4553ac622f20 -move_replicas_from_ignored_tservers 2>&1 | tee <Worker-Node-FQDN>-rebalance.out & |
Re-introduce the Tablet Server
After the Kudu Tablet Server has been purged, it’s time to reintroduce it into the Kudu service so that it can be used again.
Exit the Tablet Server from Maintenance Mode
| sudo -u kudu kudu tserver state exit_maintenance <Master-Node1-FQDN>,<Master-Node2-FQDN>,<Master-Note 3-FQDN> 5e103ac84707495e843a4553ac622f20 |
Unquiesce the Tablet Server
| sudo -u kudu kudu tserver quiesce stop <Worker-Node-FQDN> |
Rebalance the Kudu Service
We now have a Kudu Tablet Server that has been quiesced and purged. It’s time to rebalance the Kudu service and share the Tablets back onto the recently purged Kudu Tablet Server.
Go to CM - Kudu - Actions - Run Kudu Rebalancer Tool:
