Support Questions
Find answers, ask questions, and share your expertise

Intra Disk Balancing

Solved Go to solution

Intra Disk Balancing

Rising Star

Hello,

 

How do we balance data which are stored on individual disk on a particular datanode, we have 5 disks on a single node and one of the disk is 90% full, running balancer is not fixing the issue.

Would like to get some suggestions/ comments to fix this issue.

I was going through this article and it says it is not possible to balance disks within a single node, so what other options can we use to fix this until we upgrade to CDH 6.3

(https://community.cloudera.com/t5/Community-Articles/HDFS-Balancer-Balancing-Data-Between-Disks-on-a...)

 

CM & CDH - 5.16.3

 

Appreciate all inputs.

 

Thanks

Wert

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Intra Disk Balancing

Cloudera Employee

Hi @wert_1311 I see you have mentioned you are currently running on CM & CDH - 5.16.3. If this is correct then you can follow below article and fix the disk usage.

 

https://blog.cloudera.com/how-to-use-the-new-hdfs-intra-datanode-disk-balancer-in-apache-hadoop/

 

Let me know if any query or comment.

View solution in original post

4 REPLIES 4

Re: Intra Disk Balancing

Rising Star

Any assistance on this please ...

 

Re: Intra Disk Balancing

Cloudera Employee

Hi @wert_1311 I see you have mentioned you are currently running on CM & CDH - 5.16.3. If this is correct then you can follow below article and fix the disk usage.

 

https://blog.cloudera.com/how-to-use-the-new-hdfs-intra-datanode-disk-balancer-in-apache-hadoop/

 

Let me know if any query or comment.

View solution in original post

Re: Intra Disk Balancing

Contributor

Hello @wert_1311 You can balance the disk usage of the DN storage volumes using "intra-disk balancer" feature available in CDH starting release 5.8.2 and later.

 

You need to enable the feature by adding the  "dfs.disk.balancer.enabled" configuration to HDFS via the HDFS safety valve snippet in Cloudera Manager following the blog here - https://blog.cloudera.com/how-to-use-the-new-hdfs-intra-datanode-disk-balancer-in-apache-hadoop/

 

A typical disk-balancer task involves three steps (implemented via the "hdfs diskbalancer" command): plan, execute, and query.

The steps are as follows:

1. Enable intra disk balancer config in HDFS

2. "Plan" the intra disk balancer 

3. Execute the created plan

4. Query the running/executed plan

5. Verify the balancer report  

 

For more info refer the apache doc here - https://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html

 

Thanks and Regards,

Pabitra Das 

 

 

Re: Intra Disk Balancing

Rising Star

@abagal  / @PabitraDas  Appreciate all your assistance / inputs on this.

 

Thanks

Wert