Reply
Expert Contributor
Posts: 115
Registered: ‎07-01-2015

Add new disk to Kudu

I tried to increase the number of existing TableServer drives (/data/disk4,/data/disk5), but adding an new path (/data/disk1) in CM configuration did not worked. TabletServer refused to start, because it did not found anything on the new disk and tried to create a new filesystem, but then failed because /data/disk4 is not empty

Is there any specific command for adding a new drive?

Thanks 

Champion
Posts: 545
Registered: ‎05-16-2016

Re: Add new disk to Kudu

[ Edited ]

since you have already used /data/disk4 there might be still data lying around . 

perform the below step

 

 

rm -rf  in the  data directories and WAL directory contents 
of tablet server that you had configured .

 

then kick off the tablet server using .

sudo service kudu-tserver start 

 

finally run the health check against the cluster using 

 kudu cluster ksck  master_address

 

finally could you let me know what type of filesystem that are you are configuring because it should support hole punching if not the start up will fail but cloudera has a work around though .  

Highlighted
Expert Contributor
Posts: 115
Registered: ‎07-01-2015

Re: Add new disk to Kudu

Regarding the first suggestion: I can't delete the existing data and WAL directories, because I will loose the data.

I know, that in case of tables with replica more than 1 it should not be a problem. But what If every Kudu table server has to be extended? Do I understand it correctly that you are suggesting a rolling "extension" proces?:

 

1. shutdown the kudu tablet server

2. remove data and WAL directories (all tables must have replica more than 1)

3. reconfigure JUST the actual table server in CM and add the new disks (data directories)

4. kick off the tablet server

5. Run health check (maybe wait for the data to be replicated?)

6. Continue with another table server

 

 

 

The filesystem is ext4.

 

In the meantime I have found the answer to my question, that Kudu does not support adding new directories:

 

https://kudu.apache.org/docs/known_issues.html#_server_management

"Data directories cannot be added/removed; all must be reformatted to change the set of directories."

 

Champion
Posts: 545
Registered: ‎05-16-2016

Re: Add new disk to Kudu

Kudu table server has to be extended " i did not understand this part , if you have logical volume configured on that mount that you have ur filesystem it shouldnt be a problem i aint a linux expert but thats my kind of thinking goes . 

to check the health of the kudu  / consistent of the metadata 

you can run the below command 

kudu cluster ksck <master_addresses> [-checksum_cache_blocks] [-checksum_scan] [-checksum_scan_concurrency=<concurrency>] [-nochecksum_snapshot] [-color=<color>] [-noconsensus] [-tables=<tables>] [-tablets=<tablets>] 

Thanks for the link mate

Cloudera Employee
Posts: 46
Registered: ‎02-05-2016

Re: Add new disk to Kudu

The "rolling extension" process you're suggesting looks reasonable. It should serve as a workaround for Kudu's current inability to add new data directories.
Announcements