Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Who agreed with this solution

Re: Disk Balancer is not enabled

Cloudera Employee

Hi Gareeb,

The error messages that are written to the logs is a known issue in CDH 5.8. These messages are actually harmless, though it is filling up the DataNode logs.
The error message is related to a new feature HDFS-1312 "Re-balance disks within a Datanode". This new feature which allows balancing of data on disks within an individual datanode is not implemented in CDH 5.8 as yet.


The only way forward at this point in time is to suppress this error message and ensure it is not filling the datanode logs. The loglevel of the "org.apache.hadoop.hdfs.server.datanode.DiskBalancer" can be changed to FATAL.

There are two ways to do this. The first requires a restart of the cluster. The second does not require restart but the loglevel is not persisted after restart of datanode.


Option 1) modify log4j settings.

a) navigate to CM -> HDFS -> Configuraion ->DataNode -:> Advanced -> DataNode Logging Advanced Configuration Snippet (Safety Valve)
b) add: = FATAL
c) restart DataNodes (rolling restart recommended) .

Option 2) Modify setting on command line.

This will not persist on restarts of the DataNode and you need to run the following command for each DataNode (as 'hdfs' user):


a) Check current logging level:

      # hadoop daemonlog -getlevel org.apache.hadoop.hdfs.server.datanode.DiskBalancer


b) Set logging level for disk balancer to FATAL

      # hadoop daemonlog -setlevel org.apache.hadoop.hdfs.server.datanode.DiskBalancer FATAL

I believe this information will explain why this error message is displayed.




Who agreed with this solution