Hi @venkateswarlu prudhvi,
Loading too many hfiles into a single ColumnFamily(CF) might:
1. put too much pressure on compactions (disk IO, network IO could be bottleneck).
2. read(get/scan) performance could also be affected.
3. write operations could be blocked too.
Increase ‘hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily’ and ‘hbase.hstore.blockingStoreFiles’ would let your pass the EXCEPTION but they only mitigate the 3rd problem above.
Usually, the main reason for this issue is that ‘hbase.hregion.max.filesize’ is too small for your data. Ideally, we should generate one HFile for each CF before bulk loading. To do this, you could add the following line in your program after setting zookeeper hosts/client port: configuration.setLong(‘hbase.hregion.max.filesize’, 9223372036854775807);
One more thing, you should pre-split your HTable into 100 (or maybe 1000) regions evenly according to your data size in advance and set ‘hbase.hregion.max.filesize’ to ‘9223372036854775807’ in the table description by hbase shell, which means the table will not split automatically. To split region manually, you could write a small script(e.g. split.rb): split ‘<table_name>’, ‘002’
split ‘<table_name>’, ‘004’
split ‘<table_name>’, ‘006’
split ‘<table_name>’, ‘008’
and execute it by: $ echo split.rb | hbase shell
... View more
