Options
- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Printer Friendly Page
- Report Inappropriate Content
Super Guru
Created on 01-06-2017 12:43 PM
Problem Statement: Whenever HBase is running, the "hdfs fsck /" reports four hbase-related files in the path "hbase/data/WALs/" as CORRUPT. Even after letting the cluster sit idle for a couple hours, it is still in the corrupt state. If HBase is shut down, the problem goes away. If HBase is then restarted, the problem recurs.
ERROR:
hdfs@test1:>$ hdfs fsck / Connecting to namenode via http://test1.example.com:50070/fsck?ugi=hdfs&path=%2F FSCK started by hdfs (auth:SIMPLE) from /39.0.8.2 for path / at Wed Jun 24 20:40:17 GMT 2015 ... /apps/hbase/data/WALs/test2.example.com,16020,1435168292684/test2.example.com%2C16020%2C1435168292684.default.1435175500556: MISSING 1 blocks of total size 83 B. /apps/hbase/data/WALs/test3.example.com,16020,1435168290466/test3.example.com%2C16020%2C1435168290466..meta.1435175562144.meta: MISSING 1 blocks of total size 83 B. /apps/hbase/data/WALs/test3.example.com ,16020,1435168290466/test3.example.com%2C16020%2C1435168290466.default.1435175498500: MISSING 1 blocks of total size 83 B. /apps/hbase/data/WALs/est4.example.com,16020,1435168292373/test4.example.com%2C16020%2C1435168292373.default.1435175500301: MISSING 1 blocks of total size 83 B.................................................................................................. .................................................................................................... .................................................................................................... ........................................................................................Status: CORRUPT Total size: 723977553 B (Total open files size: 332 B) Total dirs: 79 Total files: 388 Total symlinks: 0 (Files currently being written: 5) Total blocks (validated): 387 (avg. block size 1870743 B) (Total open file blocks (not validated): 4) ******************************** UNDER MIN REPL'D BLOCKS: 4 (1.0335917 %) dfs.namenode.replication.min: 1 CORRUPT FILES: 4 MISSING BLOCKS: 4 MISSING SIZE: 332 B ******************************** Minimally replicated blocks: 387 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 3.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 3 Number of racks: 1 FSCK ended at Wed Jun 24 20:40:17 GMT 2015 in 7 milliseconds The filesystem under path '/' is CORRUPT hdfs@test1:>
ROOT CAUSE: This is a BUG with HDFS - https://issues.apache.org/jira/browse/HDFS-8809
There is a patch provided for resolving this issue. This will be soon fixed in upcoming HDP releases.