Reply
Highlighted
Contributor
Posts: 48
Registered: ‎05-30-2015

Canary test failed to create file in directory /tmp/.cloudera_health_monitoring_canary_files.

Hi there,

 

My Cloudera 5.8 cluster not working after a power trip, it show "Canary test failed to create file in directory /tmp/.cloudera_health_monitoring_canary_files." under HDFS summary, and it also says namenode is bad health, "NameNode summary: master.xxxxhadoop.local (Availability: Active, Health: Bad). This health test reflects the health of the active NameNode."

 

Under namenode, it says "This NameNode was in safe mode".

 

I tried to run sudo -u hdfs hdfs fsck / -files -blocks -locations, the output as follow:

 

Status: CORRUPT
Total size: 31013902258 B (Total open files size: 14594 B)
Total dirs: 3715
Total files: 6169
Total symlinks: 0 (Files currently being written: 1)
Total blocks (validated): 6268 (avg. block size 4947974 B) (Total open file blocks (not validated): 1)
********************************
CORRUPT FILES: 25
MISSING BLOCKS: 25
MISSING SIZE: 35928390 B
CORRUPT BLOCKS: 25
********************************
Minimally replicated blocks: 6243 (99.60115 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 2.9880345
Corrupt blocks: 25
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Fri Sep 02 22:16:09 MYT 2016 in 701 milliseconds


The filesystem under path '/' is CORRUPT

 

Anyone has any idea how to fix it?

 

Thanks!

Announcements