Often this happens as there is a "hidden" character at the end of the file or folder name. For example a line break (\n, \r, etc).   If you list the files you can get a clue that is the case as usually the output will look strange with an extra line or something there.   You can try running a few commands like the following to see if it matches a file: hdfs dfs -ls $'/path/to/folder\r' hdfs dfs -ls $'/path/to/folder\n' hdfs dfs -ls $'/path/to/folder\r\n'   If any of those match, then you can delete the incorrect one with a similar command. If you get no luck with that, then pipe the ls output into "od -c" and it will show the special characters hdfs dfs -ls /path/to/folder | od -c ... View more

Without the stack trace, we are going to have a hard time to pin down what is going wrong. With 5.3.1 being pretty old, it could easily be a bug.   I wonder if this is (top answer) is causing the stack trace to be suppressed:   https://stackoverflow.com/questions/2295015/log4j-not-printing-the-stacktrace-for-exceptions   Could be worth a NN restart wth "-XX:-OmitStackTraceInFastThrow" added to the JVM options to see if we get a stack trace out, if you are able to schedule a restart.   It your key concern is getting the missing blocks back then you should be able to copy the block file and its corresponding meta file to another DN. The block will be in a given subfolder under its disk, eg: /data/hadoop-data/dn/current/BP-1308070615-172.22.131.23-1533215887051/current/finalized/subdir0/subdir0/blk_1073741869   In that example "subdir0/subdir0" - it does not matter which node you copy it to or which disk, but you must ensure the sub folders are maintained.   Then restart the target DN and see if the block moves from missing to not missing when it checks in.   I'd suggest trying this with one block to start with to ensure it works OK. ... View more

Can you check your /etc/hadoop/conf/yarn-site.xml and ensure the following two parameters are set:   <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> ... View more

Hi,   It you set the balancer bandwidth in Cloudera manager, then when the datanodes are started, they will have that bandwidth seting for balancing operations. However, using the command line it is possible to change the bandwidth while the datanodes or balancer is running, without restarting them. If you do this, you just need to remember that if you restart any datanodes, the bandwidth setting will revert back to that set in Cloudera Manager and you will need to run the command again.   To set the balancer bandwidth from the command line and without a restart you can run the following:   sudo -u hdfs hdfs dfsadmin -setBalancerBandwidth 104857600   If you have HA setup for HDFS, the above command may fail, and you should check which is the active namenode and run the command like follows (substituting the correct hostname for activeNamenode below):   sudo -u hdfs hdfs dfsadmin -fs hdfs://activeNamenode:8020/ -setBalancerBandwidth 104857600 To check this command worked, the following log entries should appear within the Datanode log files: INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action: DNA_BALANCERBANDWIDTHUPDATE INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Updating balance throttler bandwidth from 10485760 bytes/s to: 104857600 bytes/s. ... View more