@mike_bronson7
Regarding under replicated blocks, HDFS is supposed to recover them automatically (by creating missing copies to fulfill the replication factor) but in your case, your cluster-wide replication factor is 3 but the target is 10 It's suggesting have 5 data nodes while there are 10 replicas leading to the under replication alert!
According to the output you have 2 distinct problems
(a) Under replicated blocks, Target Replicas is 10 but found 5 live replica(s) [Last 2 lines]
(b) Corrupt blocks with 2 different solutions
Solution 1 under replicated
You could force the 2 blk to align with cluster-wide replication factor by adjusting using -setrep
$ hdfs dfs -setrep -w 3 [File_name]
Validate by
Now you should see 3 after the file permissions before the user:group like below
$ hdfs dfs -ls [File_name]
-rw-r--r-- 3 analyst hdfs 1068028 2019-10-27 12:30 /flighdata/airports.dat
And wait for the deletion to happen or run the below snippets sequentially
$ hdfs fsck / | grep 'Under replicated'
$ hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files
$ for hdfsfile in `cat /tmp/under_replicated_files`; do echo "Fixing $hdfsfile :" ; hadoop fs -setrep 3 $hdfsfile; done
For Corrupt files
$ hdfs fsck / | egrep -v '^\.+$' | grep -i corrupt
...............Example output............................
/user/analyst/test9: CORRUPT blockpool BP-762603225-192.168.1.2-1480061879099 block blk_1055741378
/user/analyst/data1: CORRUPT blockpool BP-762603225-192.168.1.2-1480061879099 block blk_1056741378
/user/analyst/data2: MISSING 3 blocks of total size 338192920 B.Status: CORRUPT
CORRUPT FILES: 9
CORRUPT BLOCKS: 18
Corrupt blocks: 18
The filesystem under path '/' is CORRUPT
Locate corrupted block
$ hdfs fsck / | egrep -v '^\.+$' | grep -i "corrupt blockpool"| awk '{print $1}' |sort |uniq |sed -e 's/://g' >corrupted.flst
Get the location in the above output corrupted.flst
$ hdfs fsck /user/analyst/xxxx -locations -blocks -files
Remove the corrupted files
hdfs dfs -rm /path/to/corrupted.flst
Skip the trash to permanently delete
$ hdfs dfs -rm -skipTrash /path/to/corrupt_filename.
You should give the cluster sometime to rebalance in the case of under-replicated files.
