So, the crazy thought you had resolved the days long mistery!   An expension for any poor soul that might encounter a similar issue:   In our setup, we have three machines reserved for master roles m[1-3] and 40 worker machines w[1-40]. I assigned 2 HBase Masters, HBase Thrift server and HBase REST server on m2 and m3, Region Servers on w[20-40].   I ran hbck from m1 that has no HBase roles on it. Normally, this would fail with ConnectException as it does from all other machines that don't have HBase roles on them w[1-19], because they don't have hbase-site.xml on them and don't know where to look for the zookeeper, so they fall back to default - localhost. However, As m1 is also Zookeeper leader, the localhost is actually an OK default and hbase hbck will work. Sort of. As, m1 doesn't have any HDFS roles either.   So, m1 was LITERALLY the only machine in the cluster that would report these inconsistencies. The check would either fail with connection exception or report everything being normal.   Many thanks for your time, it saved a bunch of mine (although I also already lost a lot of it :)). ... View more

Hm. I'll dig into the permissions issue. However, I'm I doubt that this is the reason behind this weirdness, because, not only that hmaster lives, but on the surface, it appears to be functioning normally. I created a table and filled it with 2 million rows. Did hbase hbck after this. It reported 37 inconsistencies of the same type. hbase:namespace still among them.   EDIT:   Additional info: When i scan 'hbase:namespace'   I get: ROW COLUMN+CELL default column=info:d, timestamp=1486140313224, value=\x0A\x07default hbase column=info:d, timestamp=1486140313283, value=\x0A\x05hbase 2 row(s) in 0.3760 seconds ...as I should.   More additional info (don't know if relevant), before inconsistency error, in log I get  No HDFS region dir found. Looks like this:   No HDFS region dir found: { meta => hbase:namespace,,1486140310904.86d0405303ed58995e1507e33cbf66a2., hdfs => null, deployed => hadoop-38.xxxx.xxxxxxxxxxx.de,60020,1486140300341;hbase:namespace,,1486140310904.86d0405303ed58995e1507e33cbf66a2., replicaId => 0 } meta={ENCODED => 86d0405303ed58995e1507e33cbf66a2, NAME => 'hbase:namespace,,1486140310904.86d0405303ed58995e1507e33cbf66a2.', STARTKEY => '', ENDKEY => ''} It says basically the same thing as the error above, just with the additional hint of No HDFS region dir found and it's marked as warning. The deployed part also contains deployment info that I found in /hbase/WALs folder, namely:   hdfs dfs -ls /hbase/WALs Found 16 items ... drwxr-xr-x - hbase hbase 0 2017-02-06 11:11 /hbase/WALs/hadoop-38.xxxx.xxxxxxxxxxx.de,60020,1486140300341 ... My next desperate idea is to try to read whatever it is in /hbase/data/hbase/namespace/86d0405303ed58995e1507e33cbf66a2/.regioninfo (following the No HDFS region dir found hint) as soon as I find some command line protobuf reader.   Again, thanks for taking the time to look into this, and as always ANY feedback is much appreciated!   Regards!   ... View more

Hi Ben,   Thanks for your response, much appreciated!   Actually, that is exactly what I did. I messed it up so bad that I had to delete the service. All I described above actually happened after I: 1. Stopped service 2. Deleted the service 3. hdfs -rm -r /hbase 4. echo "rmr /hbase" | zookeeper-client 5. added the service again   At this time incosistencies are piling up, I have 34 of them, and the one described above, found in the namespace table is still there.   ... View more

Just a quick update, the issue is still present after upgrading to CDH 5.10.0, so... if you sort of had an idea, but were kind of shy or thought "naaaaah, he probably already thought of that", I strongly encourage you to step forward 🙂 ... View more