Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]

Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Share your experience with Cloudera on G2 and get a $25 Amazon Gift card.
Hi, I'm CLEO! Something exciting is coming to the Community. Stay Tuned!
Labels (1)
avatar
Super Guru

Problem Statement: Whenever HBase is running, the "hdfs fsck /" reports four hbase-related files in the path "hbase/data/WALs/" as CORRUPT. Even after letting the cluster sit idle for a couple hours, it is still in the corrupt state. If HBase is shut down, the problem goes away. If HBase is then restarted, the problem recurs.

ERROR:

hdfs@test1:>$ hdfs fsck /
Connecting to namenode via http://test1.example.com:50070/fsck?ugi=hdfs&path=%2F
FSCK started by hdfs (auth:SIMPLE) from /39.0.8.2 for path / at Wed Jun 24 20:40:17 GMT 2015
...
/apps/hbase/data/WALs/test2.example.com,16020,1435168292684/test2.example.com%2C16020%2C1435168292684.default.1435175500556: MISSING 1 blocks of total size 83 B.
/apps/hbase/data/WALs/test3.example.com,16020,1435168290466/test3.example.com%2C16020%2C1435168290466..meta.1435175562144.meta: MISSING 1 blocks of total size 83 B.
/apps/hbase/data/WALs/test3.example.com
,16020,1435168290466/test3.example.com%2C16020%2C1435168290466.default.1435175498500: MISSING 1 blocks of total size 83 B.
/apps/hbase/data/WALs/est4.example.com,16020,1435168292373/test4.example.com%2C16020%2C1435168292373.default.1435175500301: MISSING 1 blocks of total size 83 B..................................................................................................
....................................................................................................
....................................................................................................
........................................................................................Status: CORRUPT
 Total size:    723977553 B (Total open files size: 332 B)
 Total dirs:    79
 Total files:   388
 Total symlinks:                0 (Files currently being written: 5)
 Total blocks (validated):      387 (avg. block size 1870743 B) (Total open file blocks (not validated): 4)
  ********************************
  UNDER MIN REPL'D BLOCKS:      4 (1.0335917 %)
  dfs.namenode.replication.min: 1
  CORRUPT FILES:        4
  MISSING BLOCKS:       4
  MISSING SIZE:         332 B
  ********************************
 Minimally replicated blocks:   387 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          3
 Number of racks:               1
FSCK ended at Wed Jun 24 20:40:17 GMT 2015 in 7 milliseconds


The filesystem under path '/' is CORRUPT
hdfs@test1:>










ROOT CAUSE: This is a BUG with HDFS - https://issues.apache.org/jira/browse/HDFS-8809

There is a patch provided for resolving this issue. This will be soon fixed in upcoming HDP releases.

2,370 Views
0 Kudos
Version history
Last update:
‎01-06-2017 12:43 PM
Updated by:
Contributors