We have CDH5.9.2 hadoop 13-node cluster and from one week, I am exeperiencing block count concerning warning notification in cloudera manager for all 6 datanodes.
We have multitenancy implemented in our cluster. So I deleted the older files via hdfs command line and block count is below the threshold value.
But still the block count concerning warning appears in the cloudera manager.
Get the value of
a. CM -> HDFS -> Configuration -> DataNode Block Count Thresholds
b. CM -> HDFS -> WebUI -> Namenode Web UI -> Click on datanode menu -> Get the block count of your node
if b > a then you will get block count warning
also cloudera advice says "presence of many small files" also create this warning
1. if it is not disturbing anything then you can ignore this warning but just keep an on eye on block pool usage percentage from 'b'
2. you can increase block count thresholds in 'a'
3. you can cleanup unwanted data, but if your trash folder maintains old data (for ex: 24 hrs) then you will see the result after 24 hours
4. add additional data nodes and apply rebalance
b>a for all the datanodes.
I have cleaned up the unwanted data and in command line block count is showing below threshold value however it's not reflected in the cloudera manager console.