HBASE Inconsistencies

[#1]

Hi Team ,

i ran hbase hbck command today and found many inconsistencies,please help how to fix those inconsistencies.

Hbase version =1.1.1.2.3

Thanks.

75 inconsistencies detected.
Status: INCONSISTENT

[#2]

Sure, let me try that way by -repair argument.

Many Thanks.

[#3]

Hi ,

i ran hbase hbck -repair command ,seems like it didn't fix the issue.

got below output at the end @sjiang

Exception in thread "main" java.lang.NullPointerException: <strong>Table 'prod:testj' not present!
        at </strong>com.google.common.base.Preconditions.checkNotNull(Preconditions.java:208)
        at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:869)
        at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:843)
        at org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:964)
        at org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:602)
        at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:695)
        at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:4688)
        at org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:4491)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
        at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:4479)

[#4]

This NPE shows that .tableinfo file does not exist. As suggested by Predrag below, run the repair tool multople times might solve the problem. (of course, sometime it might not solve the problem. We have to manually fix the issue)

Another thing is that please check whether any file/directory exist for this 'prod:testj' table (via hdfs commands). I suspected that the entire table directory disappear (as I mentioned earlier, someone might make a mistake by removing this directory). In this case, another solution is to drop this table and recreate the table and re-populate data.

[#5]

You may need to run "hbck -repair" several times. It's good to disable writes during repair. After each "repair", check again HBase Web UI, or run again just hbck to find out has something changed, like less RITs, or less inconsistencies. Regarding 2 RITs, you can also check their hdfs directories, to confirm they are healthy (no missing blocks). Finally, you can try to restart Region servers hosting RITs. So, your first target should be to get rid of RITs, and then other inconsistencies.