Immediately what comes to mind is that there are many safety valves at different scopes of the hbase service. hbase.coprocessor.region.classes would be scoped to the regionserver safety valve, hbase.coprocessor.master.classes would be scoped to the master safety valve, etc.   additionally you can typically get away with putting things in the service wide safety valve since regionserver won't throw any errors on having hmaster specific config and vice versa. There are obviously cases where you want the same config item to be different for the different services.   you may already know this, but here is the cloudera article on setting this up: https://www.cloudera.com/documentation/enterprise/5-6-x/topics/cdh_ig_hbase_new_features_and_changes.html#concept_eyh_xqz_bp ... View more

very interesting! So hbck says it's in Hbase META but not in HDFS?  perhaps there is a HDFS permissions issue for the hbase user? (assumption being that hbase is able to start, but not write the data it needs to HDFS, yet somehow still lives enough to stay running in that weird state.)     ... View more

Here's what i suspect happened, will call out supposition. when you start the hbase service, first the master starts up, finds the meta region in hdfs, waits for the regionserver list to settle, and then assigns meta.   then it onlines all regions defined in meta (including hbase:namespace) in the case of initial startup it would make sense to me that it would online with the namespace configured and then flush to disk. (supposition)   if you do a hbck after meta is online, but before it can flush the namespace it will find them as holes. This is because hbck can only do guesswork based on the current state of HDFS or regionserver or zookeeper.   ALL hbck operations except for hbck and hbck -fixAssignments are dangerous. and fixAssignments isn't always perfect at fixing assignments but unless there is another bug encountered, it is not destructive. what -repairHoles does is create an EMPTY region in the place of the region that was now gone. This is so that you can at least salvage what is left in the case of a disaster.   It's possible that hbase then sees that the namespace regionfile exists and then will not flush the namespace table. (supposition) I'd suggest just removing and then re-adding the hbase service (and delete the remnants in hdfs and zookeeper in between those two steps if need be)  ... View more

Kinit needed? Maybe? Seems like a kerberos heavy stack trace. ... View more

Glad to help!    Max compaction size config(hbase.hstore.compaction.max.size), *edit* looks like instead of the default you are setting that to 512MB. Yes that certainly is at least part of the issue. that effectively means that compaction will ignore any storefile larger than 512MB. I'm unsure what that will do to the ability to split when necessary. It's not something we set on our clusters.    Leaving here for others: If you are relying on hbase to do the weekly major compact(hbase.hregion.majorcompaction), there is a difference in behavior between a externally initiated compaction and a internal system one. The system initiated compaction(hbase.hregion.majorcompaction) seems to trigger only a minor compaction when over the max number of regions a minor will consider (hbase.hstore.compaction.max). I am guessing this is due to a desire to not impact the system with a very long running major compaction. In your case, you will be constantly triggering only a minor compaction of that many stores every time Hbase considers that region for compaction. (hbase.server.compactchecker.interval.multiplier multiplied by hbase.server.thread.wakefrequency)  This is especially true if you generate more hfiles than (hbase.hstore.compaction.max) in the time it takes to do (hbase.server.compactchecker.interval.multiplier multiplied by hbase.server.thread.wakefrequency + compact time). Externally initiated compaction, either through hbase shell or through the API, sets the compaction priority to high and does not consider (hbase.hstore.compaction.max).  ... View more

For the first two questions:   hbase.hstore.compaction.max.size is just a limit on the size of a file that hbase will consider for compaction.   The compaction script we wrote is run by cron on the masters. the script will check if the machine that it is running on is the current active master, if so, it continues, if not, it exits. this ensures we only get 1 thing scheduling compactions. this is NOT built into hbase. I will check if my company will allow us to opensource it.  for the last one: From our experience it seems the region split only happens after the Major compaction.  {This is correct, i was indicating that i would expect the region to spilt many times after you compacted} Regarding the going oom, do you know if it tries to keep the full temporary store file (created during the compaction) in memory or just the store file being merged at each moment?  {stored in memory, compaction has to take care of all the deletes and merge all of the versions. Since you can't modify in HDFS (only append), this has to be stored in memory} Given the sum of the store files in the most problematic regions (> TB) and given our region size configuration (15GB), the outcome of the major compaction would be something like 100 new regions. Is this something that a Major compaction can handle? {it would first write out the >TB region, then splits would be triggered as you point out, that is problematic since you likely don't have a machine with >TB of memory hanging around}   We are currently considering performing pre-split of the problematic regions before doing the compaction but we have no experience with such procedure. Any recommendation? {pre-split implies you create the appropriate amount of regions before loading data. now that you have all of this data in one region, there isn't any great, built in way to split the region}   what you could do to fix it at this point is export the table -> presplit a new table -> import into the new table.      ... View more

Well if you do a lot of tiny bulkloads, that could help explain the large amounts of files, especially if you don't pre-split tables prior to bulkloading.   Why do they stay there/ don't trigger a major compaction? I think we'd need to know file sizes/timestamps to speculate further. trying to answer your questions: 1) How can we understand why certain regions are not being properly performing Major compactions (is there any indication on the logs, etc)? The best thing I've found so far is doing checks on the number of files in HDFS. we run compactions nightly during off peak, looking for regions with over X amount of hfiles and trigger a major compact for those regions. weekly we run the same full major compact. This helps keep on top of the trouble regions.    2) Which type of attributes should we monitor to ensure good HBase health (since currently the situation does not seem very healthy however HBase is not reporting anything in Cloudera Manager)? Read/Write performance would seem to be the ultimate measure of health. We've had the best luck in monitoring this at the application side. This is really a giant topic and worthy of many blog posts. I'd recommend Lars Hofhansl's blog. It's almost a must read for HBase admins IMHO. http://hadoop-hbase.blogspot.com/   3) We have a few regions which are more problematic since they contain more than 1000 store files (which account for a few TBs of data in a single region). Is it safe to manually perform Major compaction of such regions and if so do we need to have a large enough buffer in terms of storage to ensure the compaction is successful. Depending on the filesizes you can go oom from this. To maximize chances of success, put the region on a regionserver all by it self and give it enough heap to store the resulting store file in memory. you will then likely trigger a few splits after compaction based on your max region size configuration.   4) We assume data insertion can continue to happen while these large Major compaction activities are ongoing. Is this correct? Yes, however, if there is ever more than hbase.hstore.blockingStoreFiles then writes are severly limited. I suspect that is the case for your 1000 storefile region, so you are already impacted. read here for more on compactions: http://hadoop-hbase.blogspot.com/2014/07/about-hbase-flushes-and-compactions.html ... View more