Hello Team, Following is out Kudu Cluster configuration Kudu 1.8 Number of Master :- 3 Number of tablet servers :- 5 each tablet server has 64GB RAM and an 8 core CPU Total number of tables is 254 and total number of tablets is 1265 with replication factor of 3, so post replication there are 3795 replicas replica distribution across all the tablets is even 759 per tablet server --block-cache-size is 2 GB --memory-hard-limit 50GB   For running queries on top of Kudu we use Impala. For Ingesting data to kudu we use a multi threaded Application built on top of Kudu Java Client   We ingest data to kudu in real time using the application which we have built on Java Client. Under normal load memory usage on all the tablets is between 35-40 GB   Issue arises when we backfill the data for some of the tables. Lets say we are back filling the data for a table say table_v1 from last two years. There can be around 200 - 300 million records which will get updated(we are using kudu clients upsert method). The rate of ingestion to kudu during this back fill process is around 5 million records per hour. Well after this the memory usage on all the kudu tablet servers increases by 7-10 GB which is expected, but it never comes down after the back fill is done. We have even tried to stop all the reads and writes to kudu but the memory usage on the tablet servers never come down.   So what can be the possible reasons for memory usage not going down. What can I do to bring the memory usage down?   Due of this we often hit the memory threshold and kudu stops accepting the writes and we are forced to restart kudu. ... View more

Hello,   We are currently using Cloudera Manager(express edition) to manage our Kudu, HDFS and Impala cluster. Current cluster is deployed in AWS environment. We are planning to use Cloudera Altus Director as it provides good features for auto scaling. So is it possible with Cloudera Altus Directory to use the existing cluster without creating the new one as migrating from old cluster to new one will be a pain for us. I checked the Altus director documentation but could not find anything. What other options do I have with the current setup to auto scale the existing cluster?   Regards Parth  ... View more

Hello Team,   Currently we are using Cloudera manager for managing Impala, Kudu, Hive and HDFS and this existing cluster runs on aws. With increasing query workload we are facing the issue with scaling Impala. Hence wanted to try out Cloudera Atlus Directory and wanted to know about how can I use Atlus Directory with existing cluster managed by Cloudera manager. It will be helpful anybody can suggest other methods for auto scaling existing impala cluster on aws infrastructure.     Regards Parth ... View more

Hello Team,   We use Impala and Hue to query the Kudu tables. We have now installed Sentry for access control, using which we are now able to restrict a users access to a particular column of the table. I wanted to know If we can put restriction to access a particular range partition of the table. For example if Table T1 has five columns say c1,c2, c3, c4 and c5 and 3 Range partitions p1, p2 and p3. As of now using sentry I am able to restrict access only to columns c1, c2 and c5. I also wanted to know if I can restrict access of user to only partition p1.       Regards Parth ... View more

Hi Team,   Greetings!   I am currently using Cloudera Manager 5.16 and CDH 5.16 and want to upgrade to Cloudera Manager 6.1.0. and CDH 6.1.0 . As per the Supported upgrade paths  we can upgrade to 6.1.0 from 5.16(correct me if I am wrong). But according to Upgrade Cloudera Manager and Upgrade CDH suggest that I will have to upgrade to 5.7 first and then I will be able to upgrade to 6.1.0 .   This is a bit confusing can you please help me out with this one.     Regards Parth ... View more

Hi Team,   Greetings!   I am currently using Cloudera Manager 5.16 and CDH 5.16 and want to upgrade to Cloudera Manager 6.1.0. and CDH 5.1.6 . As per the Supported upgrade paths  we can upgrade to 6.1.0 from 5.16(correct me if I am wrong). But according to Upgrade Cloudera Manager and Upgrade CDH suggest that I will have to upgrade to 5.7 first and then I will be able to upgrade to 6.1.0 .   This is a bit confusing can you please help me out with this one.     Regards Parth ... View more