question Re: mass removal of datanodes impact! in Archives of Support Questions (Read Only)

mass removal of datanodes impact!

srinivas_kar96 — Mon, 12 Jun 2017 17:38:56 GMT

Hi All,

what is the impact on existing cluster if we remove 50 of 200 datanodes from existing cluster . space is not an issue as the system is 30% of the hdfs filesystem usage . how long does it takes to rebalance the complete cluster ?

Regards

Srinivas S

Re: mass removal of datanodes impact!

ledel — Mon, 12 Jun 2017 18:37:08 GMT

Hi @srinivas s if you're using rack awareness you should probably get rid of 50 datanodes by decomissioning them without loosing some blocks, otherwise you'll probably will.

rebalance time depends on your network and cluster utilization, you can adjust some parameters to make it fastest if necessary, basically

hdfs dfsadmin -setBalancerBandwidth <bandwidth (kb/s)>

or within your HDFS params (example) :

dfs.balance.bandwidthPerSec=100000000
dfs.datanode.max.transfer.threads=16384
dfs.datanode.balance.max.concurrent.moves=500

please check Accept if you're satisfied with my answer

Re: mass removal of datanodes impact!

ArpitAgarwal — Tue, 13 Jun 2017 00:06:28 GMT

Don't just remove the DataNodes. Even with rack awareness, removing >2 nodes from different racks will lead to data loss. Instead, you should decommission them first as described here:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_administration/content/ref-a179736c-eb7c-4dda-b3b4-6f3a778bd8c8.1.html

You may know this already, but I want to make it clear for others who read this discussion in the future.

Re: mass removal of datanodes impact!

ArpitAgarwal — Tue, 13 Jun 2017 00:12:09 GMT

@Laurent Edel this answer is incorrect. Please consider editing it to mention decommissioning. Else someone may assume it's OK to just remove nodes if they have rack awareness.