Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Please see the Cloudera blog for information on the Cloudera Response to CVE-2021-4428

HDFS balancer doesn't work

New Contributor

I have 3 name nodes and 12 data nodes.

Disk usage become unbalanced, so I tried to run balancer.

I set threshold 5 but I think it doesn't work.

 

this is rebalancing log:

 

 

 

Sun May  9 02:05:55 UTC 2021
JAVA_HOME=/usr/lib/jvm/java-openjdk
using /usr/lib/jvm/java-openjdk as JAVA_HOME
using 6 as CDH_VERSION
using /var/run/cloudera-scm-agent/process/5888-hdfs-BALANCER as CONF_DIR
using  as SECURE_USER
using  as SECURE_GROUP
CONF_DIR=/var/run/cloudera-scm-agent/process/5888-hdfs-BALANCER
CMF_CONF_DIR=
unlimited
Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved
May 9, 2021 2:23:09 AM            0             10.73 GB           263.62 GB              20 GB
May 9, 2021 2:38:11 AM            1             20.02 GB           251.89 GB              20 GB
May 9, 2021 2:55:27 AM            2             30.62 GB           243.23 GB              20 GB
May 9, 2021 3:12:41 AM            3             40.63 GB           232.93 GB              20 GB
May 9, 2021 3:13:56 AM            4             41.26 GB           231.92 GB              10 GB
May 9, 2021 3:15:08 AM            5             41.88 GB           231.90 GB              10 GB
May 9, 2021 3:16:12 AM            6             42.41 GB           232.12 GB              10 GB
....
....
May 10, 2021 12:23:26 AM       1124            674.81 GB           229.43 GB              10 GB
May 10, 2021 12:24:39 AM       1125            675.43 GB           227.84 GB              10 GB
May 10, 2021 12:25:53 AM       1126            676.05 GB           228.45 GB              10 GB
May 10, 2021 12:26:06 AM       1127            676.05 GB           228.87 GB              10 GB
May 10, 2021 12:27:19 AM       1128            676.67 GB           228.89 GB              10 GB
May 10, 2021 12:28:32 AM       1129            677.28 GB           229.03 GB              10 GB
May 10, 2021 12:29:45 AM       1130            677.90 GB           230.30 GB              10 GB
May 10, 2021 12:31:01 AM       1131            678.55 GB           230.81 GB              10 GB
May 10, 2021 12:32:15 AM       1132            679.19 GB           228.35 GB              10 GB
May 10, 2021 12:33:27 AM       1133            679.80 GB           228.43 GB              10 GB
May 10, 2021 12:34:42 AM       1134            680.44 GB           228.60 GB              10 GB
May 10, 2021 12:35:54 AM       1135            681.06 GB           228.78 GB              10 GB
May 10, 2021 12:37:08 AM       1136            681.67 GB           230.25 GB              10 GB
May 10, 2021 12:38:20 AM       1137            682.29 GB           230.41 GB              10 GB
May 10, 2021 12:39:35 AM       1138            682.94 GB           231.95 GB              10 GB
May 10, 2021 12:40:48 AM       1139            683.56 GB           229.74 GB              10 GB
May 10, 2021 12:42:02 AM       1140            684.18 GB           229.97 GB              10 GB
May 10, 2021 12:42:15 AM       1141            684.18 GB           230.10 GB              10 GB
May 10, 2021 12:42:28 AM       1142            684.18 GB           230.13 GB              10 GB
May 10, 2021 12:42:42 AM       1143            684.18 GB           230.16 GB              10 GB
May 10, 2021 12:42:55 AM       1144            684.18 GB           230.20 GB              10 GB
No block has been moved for 5 iterations. Exiting...
May 10, 2021 12:43:08 AM       1145            684.18 GB           230.23 GB              10 GB
May 10, 2021 12:43:08 AM Balancing took 22.619939166666665 hours

 

 

 

 

 

and this is result of dfsadmin -report after rebalancing:

 

 

 

Configured Capacity: 24575333941248 (22.35 TB)
Present Capacity: 22011047845407 (20.02 TB)
DFS Remaining: 4812048093685 (4.38 TB)
DFS Used: 17198999751722 (15.64 TB)
DFS Used%: 78.14%
Replicated Blocks:
	Under replicated blocks: 0
	Blocks with corrupt replicas: 0
	Missing blocks: 0
	Missing blocks (with replication factor 1): 0
	Low redundancy blocks with highest priority to recover: 0
	Pending deletion blocks: 0
Erasure Coded Block Groups: 
	Low redundancy block groups: 0
	Block groups with corrupt internal blocks: 0
	Missing block groups: 0
	Low redundancy blocks with highest priority to recover: 0
	Pending deletion blocks: 0

-------------------------------------------------
Live datanodes (12):

Name: IPADDRESS (hadoop-data-2)
Hostname: hadoop-data-2
Rack: /default
Decommission Status : Normal
Configured Capacity: 2047944495104 (1.86 TB)
DFS Used: 1380502876160 (1.26 TB)
Non DFS Used: 116539301888 (108.54 GB)
DFS Remaining: 445611842446 (415.01 GB)
DFS Used%: 67.41%
DFS Remaining%: 21.76%
Configured Cache Capacity: 4294967296 (4 GB)
Cache Used: 0 (0 B)
Cache Remaining: 4294967296 (4 GB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Xceivers: 13
Last contact: Mon May 10 07:50:54 UTC 2021
Last Block Report: Mon May 10 02:04:44 UTC 2021


Name: IPADDRESS (hadoop-data-12)
Hostname: hadoop-data-12
Rack: /default
Decommission Status : Normal
Configured Capacity: 2047944495104 (1.86 TB)
DFS Used: 1363147811625 (1.24 TB)
Non DFS Used: 163468649687 (152.24 GB)
DFS Remaining: 416686426352 (388.07 GB)
DFS Used%: 66.56%
DFS Remaining%: 20.35%
Configured Cache Capacity: 4294967296 (4 GB)
Cache Used: 0 (0 B)
Cache Remaining: 4294967296 (4 GB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Xceivers: 16
Last contact: Mon May 10 07:50:51 UTC 2021
Last Block Report: Mon May 10 07:50:45 UTC 2021


Name: IPADDRESS (hadoop-data-1)
Hostname: hadoop-data-1
Rack: /default
Decommission Status : Normal
Configured Capacity: 2047944495104 (1.86 TB)
DFS Used: 1785170588379 (1.62 TB)
Non DFS Used: 60132982053 (56.00 GB)
DFS Remaining: 98041905152 (91.31 GB)
DFS Used%: 87.17%
DFS Remaining%: 4.79%
Configured Cache Capacity: 4294967296 (4 GB)
Cache Used: 0 (0 B)
Cache Remaining: 4294967296 (4 GB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Xceivers: 15
Last contact: Mon May 10 07:50:54 UTC 2021
Last Block Report: Mon May 10 07:49:21 UTC 2021


Name: IPADDRESS (hadoop-data-10)
Hostname: hadoop-data-10
Rack: /default
Decommission Status : Normal
Configured Capacity: 2047944495104 (1.86 TB)
DFS Used: 1346956799034 (1.23 TB)
Non DFS Used: 107139281862 (99.78 GB)
DFS Remaining: 489095795823 (455.51 GB)
DFS Used%: 65.77%
DFS Remaining%: 23.88%
Configured Cache Capacity: 4294967296 (4 GB)
Cache Used: 0 (0 B)
Cache Remaining: 4294967296 (4 GB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Xceivers: 11
Last contact: Mon May 10 07:50:53 UTC 2021
Last Block Report: Mon May 10 07:48:29 UTC 2021


Name: IPADDRESS (hadoop-data-5)
Hostname: hadoop-data-5
Rack: /default
Decommission Status : Normal
Configured Capacity: 2047944495104 (1.86 TB)
DFS Used: 1531371686750 (1.39 TB)
Non DFS Used: 46158320802 (42.99 GB)
DFS Remaining: 365381136627 (340.29 GB)
DFS Used%: 74.78%
DFS Remaining%: 17.84%
Configured Cache Capacity: 4294967296 (4 GB)
Cache Used: 0 (0 B)
Cache Remaining: 4294967296 (4 GB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Xceivers: 18
Last contact: Mon May 10 07:50:52 UTC 2021
Last Block Report: Mon May 10 06:28:19 UTC 2021


Name: IPADDRESS (hadoop-data-4)
Hostname: hadoop-data-4
Rack: /default
Decommission Status : Normal
Configured Capacity: 2047944495104 (1.86 TB)
DFS Used: 1526244233216 (1.39 TB)
Non DFS Used: 117792731136 (109.70 GB)
DFS Remaining: 298102558313 (277.63 GB)
DFS Used%: 74.53%
DFS Remaining%: 14.56%
Configured Cache Capacity: 4294967296 (4 GB)
Cache Used: 0 (0 B)
Cache Remaining: 4294967296 (4 GB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Xceivers: 17
Last contact: Mon May 10 07:50:54 UTC 2021
Last Block Report: Mon May 10 05:34:47 UTC 2021


Name: IPADDRESS (hadoop-data-9)
Hostname: hadoop-data-9
Rack: /default
Decommission Status : Normal
Configured Capacity: 2047944495104 (1.86 TB)
DFS Used: 1408694836518 (1.28 TB)
Non DFS Used: 233493926618 (217.46 GB)
DFS Remaining: 301132052190 (280.45 GB)
DFS Used%: 68.79%
DFS Remaining%: 14.70%
Configured Cache Capacity: 4294967296 (4 GB)
Cache Used: 0 (0 B)
Cache Remaining: 4294967296 (4 GB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Xceivers: 8
Last contact: Mon May 10 07:50:53 UTC 2021
Last Block Report: Mon May 10 06:13:11 UTC 2021


Name: IPADDRESS (hadoop-data-7)
Hostname: hadoop-data-7
Rack: /default
Decommission Status : Normal
Configured Capacity: 2047944495104 (1.86 TB)
DFS Used: 1330605393851 (1.21 TB)
Non DFS Used: 33949784133 (31.62 GB)
DFS Remaining: 578349216774 (538.63 GB)
DFS Used%: 64.97%
DFS Remaining%: 28.24%
Configured Cache Capacity: 4294967296 (4 GB)
Cache Used: 0 (0 B)
Cache Remaining: 4294967296 (4 GB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Xceivers: 7
Last contact: Mon May 10 07:50:53 UTC 2021
Last Block Report: Mon May 10 05:13:11 UTC 2021


Name: IPADDRESS (hadoop-data-8)
Hostname: hadoop-data-8
Rack: /default
Decommission Status : Normal
Configured Capacity: 2047944495104 (1.86 TB)
DFS Used: 1360429714279 (1.24 TB)
Non DFS Used: 38801142937 (36.14 GB)
DFS Remaining: 544081741622 (506.72 GB)
DFS Used%: 66.43%
DFS Remaining%: 26.57%
Configured Cache Capacity: 4294967296 (4 GB)
Cache Used: 0 (0 B)
Cache Remaining: 4294967296 (4 GB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Xceivers: 15
Last contact: Mon May 10 07:50:54 UTC 2021
Last Block Report: Mon May 10 07:17:45 UTC 2021


Name: IPADDRESS (hadoop-data-6)
Hostname: hadoop-data-6
Rack: /default
Decommission Status : Normal
Configured Capacity: 2047944495104 (1.86 TB)
DFS Used: 1393230476870 (1.27 TB)
Non DFS Used: 121177790906 (112.86 GB)
DFS Remaining: 427503318074 (398.14 GB)
DFS Used%: 68.03%
DFS Remaining%: 20.87%
Configured Cache Capacity: 4294967296 (4 GB)
Cache Used: 0 (0 B)
Cache Remaining: 4294967296 (4 GB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Xceivers: 22
Last contact: Mon May 10 07:50:52 UTC 2021
Last Block Report: Mon May 10 01:54:06 UTC 2021


Name: IPADDRESS (hadoop-data-11)
Hostname: hadoop-data-11
Rack: /default
Decommission Status : Normal
Configured Capacity: 2047944495104 (1.86 TB)
DFS Used: 1379660042240 (1.25 TB)
Non DFS Used: 232947793920 (216.95 GB)
DFS Remaining: 329600470630 (306.96 GB)
DFS Used%: 67.37%
DFS Remaining%: 16.09%
Configured Cache Capacity: 4294967296 (4 GB)
Cache Used: 0 (0 B)
Cache Remaining: 4294967296 (4 GB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Xceivers: 18
Last contact: Mon May 10 07:50:51 UTC 2021
Last Block Report: Mon May 10 05:30:09 UTC 2021


Name: IPADDRESS(hadoop-data-3)
Hostname: hadoop-data-3
Rack: /default
Decommission Status : Normal
Configured Capacity: 2047944495104 (1.86 TB)
DFS Used: 1392985292800 (1.27 TB)
Non DFS Used: 30939951104 (28.82 GB)
DFS Remaining: 518461629682 (482.86 GB)
DFS Used%: 68.02%
DFS Remaining%: 25.32%
Configured Cache Capacity: 4294967296 (4 GB)
Cache Used: 0 (0 B)
Cache Remaining: 4294967296 (4 GB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Xceivers: 17
Last contact: Mon May 10 07:50:52 UTC 2021
Last Block Report: Mon May 10 03:09:15 UTC 2021

 

 

Because I set 5% as threshold, I think difference between maximum and minimum usage is 5, but it isn't. (hadoop-data-1: 87%, hadoop-data-2: 67%)

I searched some solution but it doesn't work on me

What should I check to solve this problem? or do I misunderstand threshold of threshold or balancer?

 

0 REPLIES 0