HBASE NotServingRegionException

[#1]

Hi, i have an issue that had also time ago and dont remember how i fixed it.
Basically when try to connect to some hbase tables, it gives error : "org.apache.hadoop.hbase.NotServingRegionException:  table XXX is not online on worker04"

My hbase is 2.0.
Another thing is that if i restart hbase, i have like 200 Regions in transit, but they disappear if i do "restart all region servers" . But it doesnt fix the problem.

I recall that i did something with zookeeper, with the ZkCli, but cant remember what.
I think the problem is with the Meta table. But since i'm in a Production env. i must be very careful.

Thanks for your help!

[#2]

Hi @loridigia,

Based on the current error you provided "org.apache.hadoop.hbase.NotServingRegionException:  table XXX is not online on worker04" maybe some regions are not deployed on any RegionServers yet.

please check this result to see is there any inconsistencies on this table:

1. sudo -u hbase hbase hbck -details > /tmp/hbck.txt

2. If you see inconsistencies please grep ERROR from hbck.txt you will see which region has problem.

3. Then you need to check if this region's directory is complete in this result:

hdfs dfs -ls -R /hbase

4. Then need to check in hbase shell : scan 'hbase:meta', if this region's info are updated in hbase:meta table.

5. Based on type of the issue we need to use hbck2 jar to fix the inconsistencies.

https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2

These are general steps to deal with this kind of problem, there could be more complex issues behind it. We suggest you to file a case with Cloudera support. 

Thanks,

Will