How to remove risk disks from Hadoop cluster ?

[#1]

Some disks failed in my HDFS cluster. and nodemanager cannot start in these nodes.

How to fix it ?

[#2]

Is multiple disks on the same node?

if yes, I think you can decommission the node from Ambari->hosts->data node -> you will find decommission from drop down.

[#3]

Thanks for quick reply.

[#4]

Take a look at this questions, maybe it is helpful => https://community.hortonworks.com/questions/3012/what-are-the-steps-an-operator-should-take-to-repl....

[#5 · Accepted solution]

In Namenode UI check and ensure that there are no missing and corrupt blocks. If this is true then you can successfully remove failed disk from DataNode.

Refer this for details

[#6]

Thanks for quick reply.

[#7]

Can you check the setting for

dfs.datanode.failed.volumes.tolerated

in your environment. Default is 0 which is a bit restrictive. Normally 1 or even 2 ( on datanodes with high disc density ) make more operational sense.

Then your datanode will start and you can take care of the discs.

[#8]

Iif you put this machine in a separate config group and remove referencw to the directories used you can keep the machine up. Removing disk and not replacing will mean your data will be writing to OS filesystem. Also do what Benjamin siggests and increase tolerance.