Hi Fawze,   This is not disk space issue. There is sufficient space on these large drives.   Thanks ... View more

"connection-pending remote" .. whatever the remote IP is, this system is not able to connect to it, masy be even to a specific port. This can happen if the remote system is busy or just the port to which the connection is being tried is busy or not open.   You need to check your network settings and/or the service state that is running on the unresponsive port. ... View more

We cannot be sure of the reasons for this message with the snippet that you have provided. If you notice, the connection is being successfuly set but there is not response from DN. ~~~ java.nio.channels.SocketChannel[connected local=/172.31.15.196:50010 remote=/172.31.1.81:57017] ~~~   It can happen due to various reasons, like, the pipeline is interrupted, there are network congestions at play, the DN disk is not performing well, DN host OS is having issues like kernel soft lockups or just that the DN is too heavily loaded to respond back. You'd have to dig in more into the logs and look for more information.   See the messages logged before the exception you're getting in the DN logs. ... View more

HDFS fsck only checks the files that are persisted on hdfs and not open files. Since you're seeing just one missing block in the UI warnings of CM and NN and no missing blocks in fsck output, this would indicate that the missing block alert is being generated from a file that is open in the memory and is most likely a false alarm.   This should go away when the NN role is restarted or the cluster is restarted, probably during your next maintainance window. ... View more