Efficient time-series applications in HBase

[#1]

General optimizations

1. Do not run HDFS balancer. It breaks data locality and data locality is important for latency-sensitive applications
2. For the very same reason disable HBase auto region balancing: balance_switch false
3. Disable periodic automatic major compactions for time-series data. Time-series data is immutable (means no update/deletes usually). The only reason remaining for major compaction is decreasing number of store files, but we will apply different compaction policy, which limits number of files and does not require major compaction (see below)
4. Presplit table(s) with time-series data in advance.
5. Disable region splits completely (set DisabledRegionSplitPolicy). Region splitting results in major compaction and we do not run major compactions because it usually decrease performance, stability and increase operation latencies.
6. Enable WAL Compression - decrease write IO.

Table design

1. Do not store data in a raw format - use time-series specific compression (refer to OpenTSDB row key design)
2. Create coprocessor which will run periodically and compress raw data
3. Have separate column families for raw and compressed data
4. Increase hbase.hstore.blockingStoreFiles for both column families
5. Use FIFOCompactionPolicy for raw data (see below)
6. Use standard exploring compaction with limit on a maximum selection size for compressed data (see below)
7. Use gzip block compression for raw data (GZ) – decrease write IO.
8. Disable block cache for raw data (you will reduce block cache churn significantly)

FIFO compaction

1. First-In-First-Out
2. No compaction at all
3. TTL expired data just get archived
4. Ideal for raw data storage (minimum IO overhead)
5. No compaction – no block cache trashing
6. Sustains 100s MB/s write throughput per RS
7. Available 0.98.17, 1.2+, HDP-2.4+
8. Refer to https://issues.apache.org/jira/browse/HBASE-14468 for usage and configuration

Exploring Compaction + Max Size

1. Set hbase.hstore.compaction.max.size to some appropriate value (say 500MB). With default region size of 10GB this results in maximum 20 store files per region.
2. This helps in preserving temporal locality of data – data points which are close will be stored in a same file, distant ones – in a separate files.
3. This compaction works better with block cache
4. More efficient caching of recent data is possible
5. Good for most-recent-most-valuable data access pattern.
6. Use it for compressed and aggregated data
7. Helps to keep recent data in a block cache.