Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HBASE Inconsistencies

Highlighted

Re: HBASE Inconsistencies

Explorer

Sure, let me try that way by -repair argument.

Many Thanks.

Re: HBASE Inconsistencies

Explorer

Hi ,

i ran hbase hbck -repair command ,seems like it didn't fix the issue.

got below output at the end @sjiang

Exception in thread "main" java.lang.NullPointerException: <strong>Table 'prod:testj' not present!
        at </strong>com.google.common.base.Preconditions.checkNotNull(Preconditions.java:208)
        at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:869)
        at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:843)
        at org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:964)
        at org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:602)
        at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:695)
        at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:4688)
        at org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:4491)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
        at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:4479)
Highlighted

Re: HBASE Inconsistencies

Explorer

This NPE shows that .tableinfo file does not exist. As suggested by Predrag below, run the repair tool multople times might solve the problem. (of course, sometime it might not solve the problem. We have to manually fix the issue)

Another thing is that please check whether any file/directory exist for this 'prod:testj' table (via hdfs commands). I suspected that the entire table directory disappear (as I mentioned earlier, someone might make a mistake by removing this directory). In this case, another solution is to drop this table and recreate the table and re-populate data.

Highlighted

Re: HBASE Inconsistencies

You may need to run "hbck -repair" several times. It's good to disable writes during repair. After each "repair", check again HBase Web UI, or run again just hbck to find out has something changed, like less RITs, or less inconsistencies. Regarding 2 RITs, you can also check their hdfs directories, to confirm they are healthy (no missing blocks). Finally, you can try to restart Region servers hosting RITs. So, your first target should be to get rid of RITs, and then other inconsistencies.

Don't have an account?
Coming from Hortonworks? Activate your account here