Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

kudu - tablet_id with different volume between tablet servers

avatar
Expert Contributor

hello cloudera community,

 

we identified that there is tablet_id with different volume among the cluster's tablet servers

 

there is tablet_id with 700MB on one tablet server and the same tablet_id with 190MB on another tablet server

 

how can we equalize this volumetry between tablet_id on all tablet servers

 

we believe that with this different volumetry between the tablet_id, some hosts are using a lot of memory, practically 90% of memory of what was defined.

 

1 - we are using cloudera express 5.16.x
2 - a rebalance has already been performed on kudu recently

 

4 REPLIES 4

avatar
Expert Contributor

Hi,

 

kudu balancer balance tablet per TS but not the size of tablet, 

 

1) what replication factor you have provided for kudu

2) please run the kudu cluster ksck master_address_fqdn <tablet_id>. << it will show you all the replica of this tablet and cross- check the size of tablet in all 3TS

 

My assumption is this tablet is new and kudu will be downloading the blocks on the same time thats why the replica is showing differ size but its an assumption, you need to trace the same via kudu TS logs which used to be under /var/log/kudu

avatar
Expert Contributor

oi  @Asfahan 

 

1 - this parameter was defined in the kudu settings on cloudera:

 

default_num_replicas = 3

 

2 - below is the result of the "fsck" command you asked for:

 

tablets_id.png

 

command did not return tablet_id size on TS hosts

avatar
Expert Contributor

the command is to show you where all the replica of that tabletid exist now login to those TS go to consensus-meta and check the size in all three TS,

avatar
Expert Contributor

hi @Asfahan 

 

checking the file with the same name as tablet_id in the "consensus-meta" directory, it shows that the file is 11K in all TS

 

as you can see in the screenshot below, the tablet_id has different sizes between the 3 TS

 

tablets_id_size_diferents.png