kudu1.7.0 in cdh 5.15
3 master nodes, 4c32g, ubuntu16.04
3 data nodes, 8c64g, 1.8T ssd, ubuntu16.04
Here is a table : project_construction_record, 62 columns, 170k records, no partition
The table has many crud operations every day
I run a simple sql on it (using impala):
SELECT * FROM project_construction_record ORDER BY id LIMIT 1
it takes 7 seconds!
By checking the profile, I found this:
KUDU_SCAN_NODE (id=0) (6.06s)
- BytesRead: 0 byte
- CollectionItemsRead: 0
- InactiveTotalTime: 0 ns
- KuduRemoteScanTokens: 0
- NumScannerThreadsStarted: 1
- PeakMemoryUsage: 3.4 MB
- RowsRead: 177,007
- RowsReturned: 177,007
- RowsReturnedRate: 29188/s
- ScanRangesComplete: 1
- ScannerThreadsInvoluntaryContextSwitches: 0
- ScannerThreadsTotalWallClockTime: 6.09s
- MaterializeTupleTime
: 6.06s
- ScannerThreadsSysTime: 48ms
- ScannerThreadsUserTime: 172ms
So i check the scan of this sql, and found this:
| column | cells read | bytes read | blocks read |
| id | 176.92k | 1.91M | 19.96k |
| org_id | 176.92k | 1.91M | 19.96k |
| work_date | 176.92k | 2.03M | 19.96k |
| description | 176.92k | 1.21M | 19.96k |
| user_name | 176.92k | 775.9K | 19.96k |
| spot_name | 176.92k | 825.8K | 19.96k |
| spot_start_pile | 176.92k | 778.7K | 19.96k |
| spot_end_pile | 176.92k | 780.4K | 19.96k |
| ...... | ...... | ...... | ...... |
There are so many blocks read.
Then I run the kudu fs list command, and I got a 70M report data, here is the bottom:
0b6ac30b449043a68905e02b797144fc | 25024 | 40310988 | column 0b6ac30b449043a68905e02b797144fc | 25024 | 40310989 | column 0b6ac30b449043a68905e02b797144fc | 25024 | 40310990 | column 0b6ac30b449043a68905e02b797144fc | 25024 | 40310991 | column 0b6ac30b449043a68905e02b797144fc | 25024 | 40310992 | column 0b6ac30b449043a68905e02b797144fc | 25024 | 40310993 | column 0b6ac30b449043a68905e02b797144fc | 25024 | 40310996 | undo 0b6ac30b449043a68905e02b797144fc | 25024 | 40310994 | bloom 0b6ac30b449043a68905e02b797144fc | 25024 | 40310995 | adhoc-index
there are 25024 rowsets, and more than 1m blocks in the tablet
I left the maintenance and the compact flags by default, only change the tablet_history_max_age_sec to one day:
--maintenance_manager_history_size=8 --maintenance_manager_num_threads=1 --maintenance_manager_polling_interval_ms=250 --budgeted_compaction_target_rowset_size=33554432 --compaction_approximation_ratio=1.0499999523162842 --compaction_minimum_improvement=0.0099999997764825821 --deltafile_default_block_size=32768 --deltafile_default_compression_codec=lz4 --default_composite_key_index_block_size_bytes=4096 --tablet_delta_store_major_compact_min_ratio=0.10000000149011612 --tablet_delta_store_minor_compact_max=1000 --mrs_use_codegen=true --compaction_policy_dump_svgs_pattern= --enable_undo_delta_block_gc=true --fault_crash_before_flush_tablet_meta_after_compaction=0 --fault_crash_before_flush_tablet_meta_after_flush_mrs=0 --max_cell_size_bytes=65536 --max_encoded_key_size_bytes=16384 --tablet_bloom_block_size=4096 --tablet_bloom_target_fp_rate=9.9999997473787516e-05 --tablet_compaction_budget_mb=128 --tablet_history_max_age_sec=86400
It is a production enviroment, and many other tables have same issue, the performance is getting slower and slower.
So my question is:
why the compaction does not run? is it a bug? and can i do compact manually?
