Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Permanently remove disk from all Data Nodes

Permanently remove disk from all Data Nodes

New Contributor

Hi everyone,

 

currently we have 5 data nodes managed by Cloudera Manager Enterprise. Each data node has 12 disks used for HDFS. We now want to remove one disk on all data nodes to be used by the operating system and therefore excluded by HDFS completely. So the goal would be to have 11 instead of 12 disks in HDFS on each data node.

I found a documentation describing disk hot swapping for data nodes: http://www.cloudera.com/documentation/enterprise/latest/topics/admin_dn_swap.html

 

However I cannot find a documenation how to permanently remove a disk from all data nodes.

 

What steps are necessary to permanently remove a disk? Any ideas how to do this without having data loss or downtime?

3 REPLIES 3

Re: Permanently remove disk from all Data Nodes

Champion
The config you need to find and change is dfs.data.dir. This is a comma separated list of paths that are used by HDFS to store data. Remove the path on the OS drive.

Re: Permanently remove disk from all Data Nodes

New Contributor

What happens when I do this on all Data Nodes (lets say remove the mount point for drive 12).

What happens with the data stored on this mount points? Does Cloudera automatically do a rebelancing of the data?

Highlighted

Re: Permanently remove disk from all Data Nodes

Champion
HDFS will not long consider those disk as being available to write or read data to. They are effectively dead to HDFS. The data on them will remain in place but be useless as the NN will already make the blocks on those drives as bad and mark them to be replicated elsewhere. This replication of the missing blocks will happen immediately since the drives are considered dead.

You can monitor it with hdfs fsck / or in CM.

It is worth mentioning that the balancer does not rebalance between disks just nodes (there is a new feature were this is done that was first added to CDH 5.8.x I believe but I haven't played with it yet). All of the nodes should be balanced though as they are each loosing roughly the same amount of data.

Five datanodes is small and I am assuming they are all in the same rack. That means there is a chance that multiple blocks replicas will exist across those same drives. I don't know if anyway to check this out. A safe approach would be to remove no more than one at a time to ensure that there is not chance that all three replicas are erased. This is more challenging but doable. You will need to create another template/role group were the drives is remove. Move one the datanodes into the new role group and push the update configs. Check that the data nodes is no longer using the drive (the CM DN page should list out the drives) and ensure that there are not missing or under-replicated blocks before doing the next.