28730
DISCUSSIONS
101733
MEMBERS
3157
ARTICLES
Created 09-02-2016 07:29 AM
Hi there,
My Cloudera 5.8 cluster not working after a power trip, it show "Canary test failed to create file in directory /tmp/.cloudera_health_monitoring_canary_files." under HDFS summary, and it also says namenode is bad health, "NameNode summary: master.xxxxhadoop.local (Availability: Active, Health: Bad). This health test reflects the health of the active NameNode."
Under namenode, it says "This NameNode was in safe mode".
I tried to run sudo -u hdfs hdfs fsck / -files -blocks -locations, the output as follow:
Status: CORRUPT
Total size: 31013902258 B (Total open files size: 14594 B)
Total dirs: 3715
Total files: 6169
Total symlinks: 0 (Files currently being written: 1)
Total blocks (validated): 6268 (avg. block size 4947974 B) (Total open file blocks (not validated): 1)
********************************
CORRUPT FILES: 25
MISSING BLOCKS: 25
MISSING SIZE: 35928390 B
CORRUPT BLOCKS: 25
********************************
Minimally replicated blocks: 6243 (99.60115 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 2.9880345
Corrupt blocks: 25
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Fri Sep 02 22:16:09 MYT 2016 in 701 milliseconds
The filesystem under path '/' is CORRUPT
Anyone has any idea how to fix it?
Thanks!