Often this happens as there is a "hidden" character at the end of the file or folder name. For example a line break (\n, \r, etc).   If you list the files you can get a clue that is the case as usually the output will look strange with an extra line or something there.   You can try running a few commands like the following to see if it matches a file: hdfs dfs -ls $'/path/to/folder\r' hdfs dfs -ls $'/path/to/folder\n' hdfs dfs -ls $'/path/to/folder\r\n'   If any of those match, then you can delete the incorrect one with a similar command. If you get no luck with that, then pipe the ls output into "od -c" and it will show the special characters hdfs dfs -ls /path/to/folder | od -c ... View more

Without the stack trace, we are going to have a hard time to pin down what is going wrong. With 5.3.1 being pretty old, it could easily be a bug.   I wonder if this is (top answer) is causing the stack trace to be suppressed:   https://stackoverflow.com/questions/2295015/log4j-not-printing-the-stacktrace-for-exceptions   Could be worth a NN restart wth "-XX:-OmitStackTraceInFastThrow" added to the JVM options to see if we get a stack trace out, if you are able to schedule a restart.   It your key concern is getting the missing blocks back then you should be able to copy the block file and its corresponding meta file to another DN. The block will be in a given subfolder under its disk, eg: /data/hadoop-data/dn/current/BP-1308070615-172.22.131.23-1533215887051/current/finalized/subdir0/subdir0/blk_1073741869   In that example "subdir0/subdir0" - it does not matter which node you copy it to or which disk, but you must ensure the sub folders are maintained.   Then restart the target DN and see if the block moves from missing to not missing when it checks in.   I'd suggest trying this with one block to start with to ensure it works OK. ... View more