Problem Statement: Whenever HBase is running, the "hdfs fsck /" reports four hbase-related files in the path "hbase/data/WALs/" as CORRUPT. Even after letting the cluster sit idle for a couple hours, it is still in the corrupt state. If HBase is shut down, the problem goes away. If HBase is then restarted, the problem recurs.
ERROR:
hdfs@test1:>$ hdfs fsck /
Connecting to namenode via http://test1.example.com:50070/fsck?ugi=hdfs&path=%2F
FSCK started by hdfs (auth:SIMPLE) from /39.0.8.2 for path / at Wed Jun 24 20:40:17 GMT 2015
...
/apps/hbase/data/WALs/test2.example.com,16020,1435168292684/test2.example.com%2C16020%2C1435168292684.default.1435175500556: MISSING 1 blocks of total size 83 B.
/apps/hbase/data/WALs/test3.example.com,16020,1435168290466/test3.example.com%2C16020%2C1435168290466..meta.1435175562144.meta: MISSING 1 blocks of total size 83 B.
/apps/hbase/data/WALs/test3.example.com
,16020,1435168290466/test3.example.com%2C16020%2C1435168290466.default.1435175498500: MISSING 1 blocks of total size 83 B.
/apps/hbase/data/WALs/est4.example.com,16020,1435168292373/test4.example.com%2C16020%2C1435168292373.default.1435175500301: MISSING 1 blocks of total size 83 B..................................................................................................
....................................................................................................
....................................................................................................
........................................................................................Status: CORRUPT
Total size: 723977553 B (Total open files size: 332 B)
Total dirs: 79
Total files: 388
Total symlinks: 0 (Files currently being written: 5)
Total blocks (validated): 387 (avg. block size 1870743 B) (Total open file blocks (not validated): 4)
********************************
UNDER MIN REPL'D BLOCKS: 4 (1.0335917 %)
dfs.namenode.replication.min: 1
CORRUPT FILES: 4
MISSING BLOCKS: 4
MISSING SIZE: 332 B
********************************
Minimally replicated blocks: 387 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Wed Jun 24 20:40:17 GMT 2015 in 7 milliseconds
The filesystem under path '/' is CORRUPT
hdfs@test1:>