kudu - tablet_id with different volume between tablet servers

[#1]

hello cloudera community,

we identified that there is tablet_id with different volume among the cluster's tablet servers

there is tablet_id with 700MB on one tablet server and the same tablet_id with 190MB on another tablet server

how can we equalize this volumetry between tablet_id on all tablet servers

we believe that with this different volumetry between the tablet_id, some hosts are using a lot of memory, practically 90% of memory of what was defined.

1 - we are using cloudera express 5.16.x
2 - a rebalance has already been performed on kudu recently

[#2]

Hi,

kudu balancer balance tablet per TS but not the size of tablet, 

1) what replication factor you have provided for kudu

2) please run the kudu cluster ksck master_address_fqdn <tablet_id>. << it will show you all the replica of this tablet and cross- check the size of tablet in all 3TS

My assumption is this tablet is new and kudu will be downloading the blocks on the same time thats why the replica is showing differ size but its an assumption, you need to trace the same via kudu TS logs which used to be under /var/log/kudu

[#3]

oi  @Asfahan 

1 - this parameter was defined in the kudu settings on cloudera:

default_num_replicas = 3

2 - below is the result of the "fsck" command you asked for:

command did not return tablet_id size on TS hosts

[#4]

the command is to show you where all the replica of that tabletid exist now login to those TS go to consensus-meta and check the size in all three TS,

[#5]

hi @Asfahan 

checking the file with the same name as tablet_id in the "consensus-meta" directory, it shows that the file is 11K in all TS

as you can see in the screenshot below, the tablet_id has different sizes between the 3 TS