Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

kudu compaction did not run

Solved Go to solution

kudu compaction did not run

Explorer

kudu1.7.0 in cdh 5.15

3 master nodes, 4c32g, ubuntu16.04 
3 data nodes, 8c64g, 1.8T ssd, ubuntu16.04

 

Here is a table : project_construction_record, 62 columns, 170k records, no partition

The table has many crud operations every day

I run a simple sql on it (using impala): 

 

SELECT * FROM project_construction_record ORDER BY id LIMIT 1

it takes 7 seconds!

 

By checking the profile, I found this:

KUDU_SCAN_NODE (id=0) (6.06s)
  • BytesRead: 0 byte
  • CollectionItemsRead: 0
  • InactiveTotalTime: 0 ns
  • KuduRemoteScanTokens: 0
  • NumScannerThreadsStarted: 1
  • PeakMemoryUsage: 3.4 MB
  • RowsRead: 177,007
  • RowsReturned: 177,007
  • RowsReturnedRate: 29188/s
  • ScanRangesComplete: 1
  • ScannerThreadsInvoluntaryContextSwitches: 0
  • ScannerThreadsTotalWallClockTime: 6.09s
    • MaterializeTupleTime6.06s
    • ScannerThreadsSysTime: 48ms
    • ScannerThreadsUserTime: 172ms

So i check the scan of this sql, and found this:

columncells readbytes readblocks read
id176.92k1.91M19.96k
org_id176.92k1.91M19.96k
work_date176.92k2.03M19.96k
description176.92k1.21M19.96k
user_name176.92k775.9K19.96k
spot_name176.92k825.8K19.96k
spot_start_pile176.92k778.7K19.96k
spot_end_pile176.92k780.4K19.96k
........................

There are so many blocks read.

 

Then I run the kudu fs list command, and I got a 70M report data, here is the bottom:

 

0b6ac30b449043a68905e02b797144fc | 25024 | 40310988 | column
 0b6ac30b449043a68905e02b797144fc | 25024 | 40310989 | column
 0b6ac30b449043a68905e02b797144fc | 25024 | 40310990 | column
 0b6ac30b449043a68905e02b797144fc | 25024 | 40310991 | column
 0b6ac30b449043a68905e02b797144fc | 25024 | 40310992 | column
 0b6ac30b449043a68905e02b797144fc | 25024 | 40310993 | column
 0b6ac30b449043a68905e02b797144fc | 25024 | 40310996 | undo
 0b6ac30b449043a68905e02b797144fc | 25024 | 40310994 | bloom
 0b6ac30b449043a68905e02b797144fc | 25024 | 40310995 | adhoc-index

 

there are 25024 rowsets, and more than 1m blocks in the tablet

I left the maintenance and the compact flags by default, only change the tablet_history_max_age_sec to one day:

 

--maintenance_manager_history_size=8
--maintenance_manager_num_threads=1
--maintenance_manager_polling_interval_ms=250
--budgeted_compaction_target_rowset_size=33554432
--compaction_approximation_ratio=1.0499999523162842
--compaction_minimum_improvement=0.0099999997764825821
--deltafile_default_block_size=32768
--deltafile_default_compression_codec=lz4
--default_composite_key_index_block_size_bytes=4096
--tablet_delta_store_major_compact_min_ratio=0.10000000149011612
--tablet_delta_store_minor_compact_max=1000
--mrs_use_codegen=true
--compaction_policy_dump_svgs_pattern=
--enable_undo_delta_block_gc=true
--fault_crash_before_flush_tablet_meta_after_compaction=0
--fault_crash_before_flush_tablet_meta_after_flush_mrs=0
--max_cell_size_bytes=65536
--max_encoded_key_size_bytes=16384
--tablet_bloom_block_size=4096
--tablet_bloom_target_fp_rate=9.9999997473787516e-05
--tablet_compaction_budget_mb=128
--tablet_history_max_age_sec=86400

It is a production enviroment, and many other tables have same issue, the performance is getting slower and slower.

 

So my question is: 

why the compaction does not run? is it a bug? and can i do compact manually?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: kudu compaction did not run

Contributor

Hi huaj,

 

It looks like you are hitting KUDU-1400, which before the fix Kudu cmpacts rowsets based on overlap and not based on other criteria like on-disk size.

 

Unfortunately, there is no way to fix the small rowsets have been flushed. On the other hand, You can rebuild the affected tables, create new tables from this existing tables and see if that helps. Before doing that, please check this doc to see which patten usage pattern caused you to hit this issue and try to prevent that by following the recommendations.

 

FYI, this fix for KUDU-1400 should land in the next release CDH6.2.

 

 

11 REPLIES 11

Re: kudu compaction did not run

Explorer

I found this for that tablet in the tablet server metrics:

 

            {
                "name": "compact_rs_duration",
                "total_count": 0,
                "min": 0,
                "mean": 0,
                "percentile_75": 0,
                "percentile_95": 0,
                "percentile_99": 0,
                "percentile_99_9": 0,
                "percentile_99_99": 0,
                "max": 0,
                "total_sum": 0
            },

compaction has never run!

Re: kudu compaction did not run

Contributor

Hi huaj,

 

It looks like you are hitting KUDU-1400, which before the fix Kudu cmpacts rowsets based on overlap and not based on other criteria like on-disk size.

 

Unfortunately, there is no way to fix the small rowsets have been flushed. On the other hand, You can rebuild the affected tables, create new tables from this existing tables and see if that helps. Before doing that, please check this doc to see which patten usage pattern caused you to hit this issue and try to prevent that by following the recommendations.

 

FYI, this fix for KUDU-1400 should land in the next release CDH6.2.

 

 

Re: kudu compaction did not run

Explorer

Thanks for reply, but that is a sad answer :-(

 

I reed the doc, indeed i trigger the 'Trickling inserts' 

Small cruds are happenning to the table all the time, with low pressure

According to the doc's suggestion, i should do the below actions:

1. set --flush_threshold_secs long enough, like 1 day

2. create a new table, and copy all data to new table

3. drop the old table

4. rename the new table to the old table name

 

Is it right?

 

And, I see the cdh6.1 just release, when will 6.2 release?

 

Re: kudu compaction did not run

Contributor

Yes, but notice that raising --flush_threshold_secs too high can affect tablet server restart time.

 

Currently, CDH 6.2 is scheduled for March.

 

Re: kudu compaction did not run

Explorer

Just done it, and problem solved

Thanks!

Re: kudu compaction did not run

Explorer

Problem not solved

After a few hours, i check the rowset of that table, there are many small rowsets again!

rowsets.png

And run this command, i see 137 rowsets

kudu fs list -fs_data_dirs=/data/1/kudu/data -fs_wal_dir=/data/1/kudu/wal -tablet_id=b110f90092b647e1bd5cd3de05b40aff

Here is my config in cm, and i have restarted kudu services after change it

config.png

Why was that? what can i du?

Re: kudu compaction did not run

Explorer

Sorry, i make a mistake

flush_threshold_secs flag is for both master and tabletserver

I did not set it on tabletserver, it's ok now after setting it

Re: kudu compaction did not run

New Contributor

we hit exact same issue which was totally unexpected and a month before we would go LIVE. @huaj how is the fix working for you so far? 

Re: kudu compaction did not run

Explorer

waiting for kudu 1.9 in the coming cdh 6.2, hoping they would solve the problem

flush_threshold_secs flag sucks

when make it long, the query performance on the records in MRS (not yet flushing) sucks