Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kudu Tablets Servers - Memory usage does not decrease after reducing Reads and Writes

Kudu Tablets Servers - Memory usage does not decrease after reducing Reads and Writes

Explorer

Hello Team,

Following is out Kudu Cluster configuration


Kudu 1.8
Number of Master :- 3
Number of tablet servers :- 5
each tablet server has 64GB RAM and an 8 core CPU


Total number of tables is 254 and total number of tablets is 1265 with replication factor of 3, so post replication there are 3795 replicas

replica distribution across all the tablets is even 759 per tablet server

--block-cache-size is 2 GB --memory-hard-limit 50GB

 

For running queries on top of Kudu we use Impala. For Ingesting data to kudu we use a multi threaded Application built on top of Kudu Java Client

 

We ingest data to kudu in real time using the application which we have built on Java Client.

Under normal load memory usage on all the tablets is between 35-40 GB

 

Issue arises when we backfill the data for some of the tables.

  • Lets say we are back filling the data for a table say table_v1 from last two years.
  • There can be around 200 - 300 million records which will get updated(we are using kudu clients upsert method).
  • The rate of ingestion to kudu during this back fill process is around 5 million records per hour.
  • Well after this the memory usage on all the kudu tablet servers increases by 7-10 GB which is expected, but it never comes down after the back fill is done.
  • We have even tried to stop all the reads and writes to kudu but the memory usage on the tablet servers never come down.

 

So what can be the possible reasons for memory usage not going down. What can I do to bring the memory usage down?

 

Due of this we often hit the memory threshold and kudu stops accepting the writes and we are forced to restart kudu.

Don't have an account?
Coming from Hortonworks? Activate your account here