Thanks. The export/import idea is very interesting. We will further investigate this.   Regarding the max compaction size, do you think this could be the reason that explain why our scheduled Major compactions are not taking place (i.e. it is just ignoring the files larger than 500Mb so it thinks there is nothing to compact)? ... View more

Thank you for your reply. It is much appreciated. @ben.hemphill wrote: Why do they stay there/ don't trigger a major compaction? I think we'd need to know file sizes/timestamps to speculate further. Minor compactions are still happening so given our advanced configuration the store files are mostly slightly larger than 500Mb. Our region size is configured as 15 Gb so there is one large store file of a few Gb. Some of the store files have a timestamp several months old (clearly showing that Major compactions were not happening).   According to the documentation "hbase.hstore.compaction.max.size" should only affect minor compactions. Do you have any experience to confirm this fact?   @ben.hemphill wrote:   1) How can we understand why certain regions are not being properly performing Major compactions (is there any indication on the logs, etc)? The best thing I've found so far is doing checks on the number of files in HDFS. we run compactions nightly during off peak, looking for regions with over X amount of hfiles and trigger a major compact for those regions. weekly we run the same full major compact. This helps keep on top of the trouble regions.    Is this a manual process or do you use some time of automation (configuration in HBase?)?   ben.hemphill wrote:  3) We have a few regions which are more problematic since they contain more than 1000 store files (which account for a few TBs of data in a single region). Is it safe to manually perform Major compaction of such regions and if so do we need to have a large enough buffer in terms of storage to ensure the compaction is successful. Depending on the filesizes you can go oom from this. To maximize chances of success, put the region on a regionserver all by it self and give it enough heap to store the resulting store file in memory. you will then likely trigger a few splits after compaction based on your max region size configuration. From our experience it seems the region split only happens after the Major compaction. Regarding the going oom, do you know if it tries to keep the full temporary store file (created during the compaction) in memory or just the store file being merged at each moment? Given the sum of the store files in the most problematic regions (> TB) and given our region size configuration (15GB), the outcome of the major compaction would be something like 100 new regions. Is this something that a Major compaction can handle?   We are currently considering performing pre-split of the problematic regions before doing the compaction but we have no experience with such procedure. Any recommendation? ... View more