After disk space increased on one of the drive of a client node, its unable to start datanode. Even not connecting to the namenode. We increased space on /data/sbc1 .
We have started the node than run rebalance .But after a while we can see its again stopped.
Need your help.Below is the log.
Scheduling blk_1076562511_2824179 file /data/sdc1/dfs/dn/current/BP-926926485-10.25.176.190-1423244145752/current/finalized/subdir43/subdir10/blk_1076562511 for deletion
Thanks for your quick response. I have restarted Namenode than started datanode, Still same.Even did rebalance .No luck. Look like the Datanode is cresh. Since we did not decommission it while upgrading drive space..
Look like datanode is crashed, Can take backup of data/sdc1/dfs/dn directory and than clean it on that particular Node, and try to start. Or decomission and recomission , or reconfigure it??
We have replication factor value 2.Does deletion of that particular directory on that particular node will impact on data lost ?
If you remove your dataNode location file, you will lose your data.
If you have replication factor 2, you can delete this node and reconfigure newly another one(in another location patch for example). But you need to know that the replication action in this case would be very slow. Be patient.