Skip to main content
wert_1311
wert_1311
Gold ⭐️
June 17, 2021
Solved

Kudu T-server data distribution

Hello,
I would like some guidance/ information on data distribution in Kudu T-Servers.
We have Kudu cluster of 3 Masters and 9 T-Servers (each t-server has storage of 1TB). We are noticing that space in some t-server is getting consumed rapidly whereas in other its not that much being consumed. Would like to know why this is happening and is there any way this can be overcome, so that data can be distributed evenly across of 9 t-servers.

Kudu 1.7.0-cdh5.16.2/ CM 5.16.2

 

Appreciate any assistance in this regard.

 

Thanks

Wert

 

kingpin
kingpinAnswer
Gold ⭐️
June 21, 2021

Hi @wert_1311 

 

That's right, balancer just balances the tablet across the kudu cluster. If one host is consuming more space, it could be that the size of tablets is huge. 

Thats right, Kudu cant rebalance like HDFS based on dfs usage. 

one of the workaround you can try:-

- Stop that specific kudu TS role

- Run ksck until it comes healthy. 

- once ksck is healthy, rebuild that particular Kudu TS (rebuilding = wiping all data and wal dir)

https://kudu.apache.org/docs/administration.html#rebuilding_kudu

- start that specific TS 

- Run rebalance again

 

That should help. Let me know how did that go.

 

Cheers,

 

~ If that answers your question - Please  give the thumbs up & mark the post as accept as solution. 

View in thread

4 replies

kingpin
Gold ⭐️
kingpin
Gold ⭐️
June 17, 2021

Hi @wert_1311 ,

 

Check for Tablet distribution across tablet servers. For some reason if one tablet server goes down/unavailable, the data will be replicated to other tablet servers. 

 

You get can get number of tablets per tablet server using this command :-

sudo -u kudu kudu table list <csv of master addresses>  -list_tablets | grep "^    " | cut -d' ' -f6,7 | sort | uniq -c

 

If you find the tablet distribution is uneven. You can go ahead with kudu rebalance tool to balance your cluster.

https://docs.cloudera.com/runtime/7.2.2/administering-kudu/topics/kudu-running-tablet-rebalancing-tool.html

 

Let me know how did that go.

 

If that answers your question, Please mark this post as "accept as solution"

 

Regards,

Translate
wert_1311
wert_1311Author
Gold ⭐️
June 18, 2021

Hi @kingpin 

I did execute the script, ran rebalance report & did a rebalance too however the result I was looking for was not archived (space is still over consumed in 1 TS). I think rebalance just distributes tablets evenly to all TS what I am looking to achieve is like HDFS rebalancer and I don’t think it is there in Kudu, correct me if I am wrong.

 

Thanks 

Wert

Translate
kingpin
Gold ⭐️
kingpinAnswer
Gold ⭐️
June 21, 2021

Hi @wert_1311 

 

That's right, balancer just balances the tablet across the kudu cluster. If one host is consuming more space, it could be that the size of tablets is huge. 

Thats right, Kudu cant rebalance like HDFS based on dfs usage. 

one of the workaround you can try:-

- Stop that specific kudu TS role

- Run ksck until it comes healthy. 

- once ksck is healthy, rebuild that particular Kudu TS (rebuilding = wiping all data and wal dir)

https://kudu.apache.org/docs/administration.html#rebuilding_kudu

- start that specific TS 

- Run rebalance again

 

That should help. Let me know how did that go.

 

Cheers,

 

~ If that answers your question - Please  give the thumbs up & mark the post as accept as solution. 

Translate
    Terms & ConditionsAccessibility statement