- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Balancer: number of nodes to be included = 0
- Labels:
-
HDFS
Created ‎11-05-2015 10:49 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Using Enterprise CDH 5.2.0, we just added a new data node. When attempting to run the rebalancer, it succeeds right away but does not balance the cluster.
The existing data nodes still have 10x the data as the new node.
Am I reading the log wrong, or is it not including any nodes when rebalancing? Do I need to configure which nodes should be included? Here is the log:
2015-11-05 18:29:19,766 INFO [main] balancer.Balancer (Balancer.java:parse(686)) - Using a threshold of 10.0 2015-11-05 18:29:19,768 INFO [main] balancer.Balancer (Balancer.java:run(544)) - namenodes = [hdfs://nameservice1] 2015-11-05 18:29:19,768 INFO [main] balancer.Balancer (Balancer.java:run(545)) - parameters = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0, number of nodes to be excluded = 0, number of nodes to be included = 0] Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved 2015-11-05 18:29:20,506 INFO [main] net.NetworkTopology (NetworkTopology.java:add(419)) - Adding a new node: /default/10.81.128.198:50010 2015-11-05 18:29:20,506 INFO [main] net.NetworkTopology (NetworkTopology.java:add(419)) - Adding a new node: /default/10.81.128.197:50010 2015-11-05 18:29:20,507 INFO [main] net.NetworkTopology (NetworkTopology.java:add(419)) - Adding a new node: /default/10.81.128.195:50010 2015-11-05 18:29:20,507 INFO [main] net.NetworkTopology (NetworkTopology.java:add(419)) - Adding a new node: /default/10.81.128.196:50010 2015-11-05 18:29:20,508 INFO [main] balancer.Balancer (Balancer.java:logUtilizationCollection(349)) - 0 over-utilized: [] 2015-11-05 18:29:20,508 INFO [main] balancer.Balancer (Balancer.java:logUtilizationCollection(349)) - 0 underutilized: [] The cluster is balanced. Exiting... 05-Nov-2015 18:29:20 Balancing took 1.054 seconds
Created ‎11-08-2015 07:30 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Per the reported log, the Balancer seems to think all of the 'DFS Used%' values of each DN are well within 10% of each another, and it considers the DNs balanced. (P.s. Yes, it works on % used, not raw byte balancing).
Created ‎11-08-2015 07:30 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Per the reported log, the Balancer seems to think all of the 'DFS Used%' values of each DN are well within 10% of each another, and it considers the DNs balanced. (P.s. Yes, it works on % used, not raw byte balancing).
Created ‎11-09-2015 08:20 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks. That was my issue. The nodes are balanced in terms of DFS Used%, even though the amount raw bytes are varied.
