03-26-2019 06:42 AM
After disk space increased on one of the drive of a client node, its unable to start datanode. Even not connecting to the namenode. We increased space on /data/sbc1 .
We have started the node than run rebalance .But after a while we can see its again stopped.
Need your help.Below is the log.
Scheduling blk_1076562511_2824179 file /data/sdc1/dfs/dn/current/BP-926926485-10.25.176.190-1423244145752/current/finalized/subdir43/subdir10/blk_1076562511 for deletion
03-27-2019 02:45 AM
03-27-2019 03:12 AM - edited 03-27-2019 03:16 AM
Thanks for your quick response. I have restarted Namenode than started datanode, Still same.Even did rebalance .No luck. Look like the Datanode is cresh. Since we did not decommission it while upgrading drive space..
03-27-2019 03:18 AM
You are using CM to start the services? In this case, what ERROR is show when you click on start Service?
Can you revise the nameNode log? Thanks
03-27-2019 05:43 PM
Look like datanode is crashed, Can take backup of data/sdc1/dfs/dn directory and than clean it on that particular Node, and try to start. Or decomission and recomission , or reconfigure it??
We have replication factor value 2.Does deletion of that particular directory on that particular node will impact on data lost ?
03-28-2019 01:01 AM
If you remove your dataNode location file, you will lose your data.
If you have replication factor 2, you can delete this node and reconfigure newly another one(in another location patch for example). But you need to know that the replication action in this case would be very slow. Be patient.