Posts: 7
Registered: ‎12-26-2018

kudu compaction did not run

kudu1.7.0 in cdh 5.15

3 master nodes, 4c32g, ubuntu16.04 
3 data nodes, 8c64g, 1.8T ssd, ubuntu16.04


Here is a table : project_construction_record, 62 columns, 170k records, no partition

The table has many crud operations every day

I run a simple sql on it (using impala): 


SELECT * FROM project_construction_record ORDER BY id LIMIT 1

it takes 7 seconds!


By checking the profile, I found this:

KUDU_SCAN_NODE (id=0) (6.06s)
  • BytesRead: 0 byte
  • CollectionItemsRead: 0
  • InactiveTotalTime: 0 ns
  • KuduRemoteScanTokens: 0
  • NumScannerThreadsStarted: 1
  • PeakMemoryUsage: 3.4 MB
  • RowsRead: 177,007
  • RowsReturned: 177,007
  • RowsReturnedRate: 29188/s
  • ScanRangesComplete: 1
  • ScannerThreadsInvoluntaryContextSwitches: 0
  • ScannerThreadsTotalWallClockTime: 6.09s
    • MaterializeTupleTime6.06s
    • ScannerThreadsSysTime: 48ms
    • ScannerThreadsUserTime: 172ms

So i check the scan of this sql, and found this:

columncells readbytes readblocks read

There are so many blocks read.


Then I run the kudu fs list command, and I got a 70M report data, here is the bottom:


0b6ac30b449043a68905e02b797144fc | 25024 | 40310988 | column
 0b6ac30b449043a68905e02b797144fc | 25024 | 40310989 | column
 0b6ac30b449043a68905e02b797144fc | 25024 | 40310990 | column
 0b6ac30b449043a68905e02b797144fc | 25024 | 40310991 | column
 0b6ac30b449043a68905e02b797144fc | 25024 | 40310992 | column
 0b6ac30b449043a68905e02b797144fc | 25024 | 40310993 | column
 0b6ac30b449043a68905e02b797144fc | 25024 | 40310996 | undo
 0b6ac30b449043a68905e02b797144fc | 25024 | 40310994 | bloom
 0b6ac30b449043a68905e02b797144fc | 25024 | 40310995 | adhoc-index


there are 25024 rowsets, and more than 1m blocks in the tablet

I left the maintenance and the compact flags by default, only change the tablet_history_max_age_sec to one day:



It is a production enviroment, and many other tables have same issue, the performance is getting slower and slower.


So my question is: 

why the compaction does not run? is it a bug? and can i do compact manually?

Posts: 7
Registered: ‎12-26-2018

Re: kudu compaction did not run

I found this for that tablet in the tablet server metrics:


                "name": "compact_rs_duration",
                "total_count": 0,
                "min": 0,
                "mean": 0,
                "percentile_75": 0,
                "percentile_95": 0,
                "percentile_99": 0,
                "percentile_99_9": 0,
                "percentile_99_99": 0,
                "max": 0,
                "total_sum": 0

compaction has never run!

Cloudera Employee
Posts: 25
Registered: ‎02-22-2017

Re: kudu compaction did not run

Hi huaj,


It looks like you are hitting KUDU-1400, which before the fix Kudu cmpacts rowsets based on overlap and not based on other criteria like on-disk size.


Unfortunately, there is no way to fix the small rowsets have been flushed. On the other hand, You can rebuild the affected tables, create new tables from this existing tables and see if that helps. Before doing that, please check this doc to see which patten usage pattern caused you to hit this issue and try to prevent that by following the recommendations.


FYI, this fix for KUDU-1400 should land in the next release CDH6.2.



Posts: 7
Registered: ‎12-26-2018

Re: kudu compaction did not run

Thanks for reply, but that is a sad answer :-(


I reed the doc, indeed i trigger the 'Trickling inserts' 

Small cruds are happenning to the table all the time, with low pressure

According to the doc's suggestion, i should do the below actions:

1. set --flush_threshold_secs long enough, like 1 day

2. create a new table, and copy all data to new table

3. drop the old table

4. rename the new table to the old table name


Is it right?


And, I see the cdh6.1 just release, when will 6.2 release?


Cloudera Employee
Posts: 25
Registered: ‎02-22-2017

Re: kudu compaction did not run

Yes, but notice that raising --flush_threshold_secs too high can affect tablet server restart time.


Currently, CDH 6.2 is scheduled for March.


Posts: 7
Registered: ‎12-26-2018

Re: kudu compaction did not run

Just done it, and problem solved


Posts: 7
Registered: ‎12-26-2018

Re: kudu compaction did not run

Problem not solved

After a few hours, i check the rowset of that table, there are many small rowsets again!


And run this command, i see 137 rowsets

kudu fs list -fs_data_dirs=/data/1/kudu/data -fs_wal_dir=/data/1/kudu/wal -tablet_id=b110f90092b647e1bd5cd3de05b40aff

Here is my config in cm, and i have restarted kudu services after change it


Why was that? what can i du?

Posts: 7
Registered: ‎12-26-2018

Re: kudu compaction did not run

Sorry, i make a mistake

flush_threshold_secs flag is for both master and tabletserver

I did not set it on tabletserver, it's ok now after setting it