I have a kudu table with schema:     create table test_table ( `time` timestamp not null, -- `id` string not null, -- ..... primary key(`time`,`id`) ) partition by hash(id) partitions 6 stored as kudu;   and I try to use spark to copy the data to a parquet table in hdfs:   val df = spark.read.options(Map("kudu.master" -> kuduMasters, "kudu.table" -> KuduTable)).format("kudu").load .where("time> '2019-10-29 08:05:10' AND time < '2019-10-29 08:05:30'") df.write .mode("append") .parquet("hdfs://parquet")   But the performance is low and the job seems to be doing a full table scan against the kudu table (from spark UI, I can see the "Scan Kudu impala::table" is the number of entire table). For comparison I did a copy using impala's "insert into from" which is much faster and the "where" predicate seems to be working.  Is this full table scan behavior expected or am I missing something here? The kudu version is 1.10.0 and spark client is kudu-spark2_2.11:1.10.0 ... View more

Hi,   We are testing heavy write on kudu-1.8 shipped by CDH 6.1.1. Now we have around 350 "hot replicas" on one t-server. When we are ingesting the data, the memory consumed by each tablet keep on increasing and eventally the total used memory end up being exceeding the memory limit we give to kudu. From the mem-tracker, some replica's memrowset use as much as 500 MB,does that mean we need to provide up to 350 * 500 MB= 175 GB for each t-server?  Is there any way/config to limit/throttle the memory consumed by memrowset(The limit shows in mem-tradcker UI is "none", see below screenshot)? Any suggestion will be appreaciated, thanks! ... View more