block count warning still shows in cloudera manager even after deleting files from command line

[#1]

Hello All,

We have CDH5.9.2 hadoop 13-node  cluster and from one week, I am exeperiencing block count concerning warning notification in cloudera manager for all 6 datanodes.

We have multitenancy implemented in our cluster. So I deleted the older files via hdfs command line and block count is below the threshold value.

But still the block count concerning warning appears in the cloudera manager.

Please suggest.

Thanks,

Priya

[#2]

@cdhhadoop

Get the value of
a. CM -> HDFS -> Configuration -> DataNode Block Count Thresholds
b. CM -> HDFS -> WebUI -> Namenode Web UI -> Click on datanode menu -> Get the block count of your node

if b > a then you will get block count warning

also cloudera advice says "presence of many small files" also create this warning

action: 
1. if it is not disturbing anything then you can ignore this warning but just keep an on eye on block pool usage percentage from 'b'

2. you can increase block count thresholds in 'a'
3. you can cleanup unwanted data, but if your trash folder maintains old data (for ex: 24 hrs) then you will see the result after 24 hours
4. add additional data nodes and apply rebalance
etc

[#3]

@saranvisa,

 b>a for all the datanodes.

I have cleaned up the unwanted data and in command line block count is showing below threshold value however it's not reflected in the cloudera manager console.

Please suggest.

Thanks,

[#4]

@cdhhadoop

as mentioned, you will get warning if b > a 

[#5]

@saranvisa,

It was b>a earlier.

I removed files using command line and block count is below threshold value. However it's not reflected in cloudera manager console.

Please suggest.

Thanks,

[#6]

What command are you using to check the block replica count of each DataNode? That information is present in either the DN metrics (SELECT blocks_total WHERE roleType = DATANODE) or in the NameNode Web UI as a column under its live DataNodes page. The block replica count is not shown as part of 'hdfs dfsadmin -report' and if you are relying on FSCK then ensure you're not counting just block IDs but actual replicas instead.

Do the information in these sources still indicate that each of the DataNodes have way lesser replicas than its alert threshold?

[#7]

@Harsh J
Thanks for the reply.
I am checking the block count by using the command hdfs fsck path and referring to total blocks column in the output.
Please suggest whether this is the right way or not.

Thanks,
Priya

[#8 · Accepted solution]

That's not enitirely right - The block count warning per DataNode stems
from the fact that the DataNode's carrying too many replicas (of blocks).
The FSCK total block count is a number that does not include the replica
multiplier (x3, typically) and is global (across all DNs).

Have you looked at the Live DataNodes page on your NameNode UI as indicated
in the earlier post? It should have the reported replica count of every DN
alive in the cluster.

[#9]

@Harsh J,
Thanks for quick reply.
I thought the ouptut of fsck command includes replica multiplier and gives final total block count. Thanks for the clarification.
I checked Datanodes page on namenode WebUI and block count for each datanode is more than threshold value.

Thanks,
Priya