Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]

Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Share your experience with Cloudera on G2 and get a $25 Amazon Gift card.
Hi, I'm CLEO! Something exciting is coming to the Community. Stay Tuned!
Labels (1)
avatar

PROBLEM: Balancer fails in few minutes without any block movement.

SYMPTOMS: Following are the messages balancer exits with:-

16/11/22 07:08:29 DEBUG ipc.Client: IPC Client (280134559) connection to xxx.corp.example.com/0.0.0.0:8020 from [email protected] got value #1193
16/11/22 07:08:29 DEBUG ipc.ProtobufRpcEngine: Call: getBlocks took 2486ms
No block has been moved for 5 iterations. Exiting...Nov 22, 2016 7:08:29 AM           
4                  0 B            35.86 TB             200 GB

ROOT CAUSE: The rack distribution looked like below:-

/default-rack : 91 
/Example1 : 18 
/Example2 : 2 

The 100% utilized nodes which we were trying to balance to create space were those 20 nodes registered with racks /Example1 and /Example2.Thus based on following rack awareness rules in balancer (rule#3 for this issue) for block placement, it was not at all possible for even a single block to move compromising fault tolerance.

  /**   * Decide if the block is a good candidate to be moved from source to target.   
* A block is a good candidate if   
* 1. the block is not in the process of being moved/has not been moved;   
* 2. the block does not have a replica on the target;   
* 3. doing the move does not reduce the number of racks that the block has   */

 

SOLUTION: Distribute nodes evenly across all racks.If this is not possible add additional storage to respective nodes OR add new datanodes to the respective racks.

2,695 Views
0 Kudos
Comments
Version history
Last update:
‎02-17-2020 08:51 AM
Updated by:
Contributors