Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

block count warning still shows in cloudera manager even after deleting files from command line

avatar
Rising Star

Hello All,

 

We have CDH5.9.2 hadoop 13-node  cluster and from one week, I am exeperiencing block count concerning warning notification in cloudera manager for all 6 datanodes.

 

We have multitenancy implemented in our cluster. So I deleted the older files via hdfs command line and block count is below the threshold value.

 

But still the block count concerning warning appears in the cloudera manager.

 

Please suggest.

 

Thanks,

Priya

1 ACCEPTED SOLUTION

avatar
Mentor
That's not enitirely right - The block count warning per DataNode stems
from the fact that the DataNode's carrying too many replicas (of blocks).
The FSCK total block count is a number that does not include the replica
multiplier (x3, typically) and is global (across all DNs).

Have you looked at the Live DataNodes page on your NameNode UI as indicated
in the earlier post? It should have the reported replica count of every DN
alive in the cluster.

View solution in original post

8 REPLIES 8

avatar
Champion

@cdhhadoop

 

Get the value of
a. CM -> HDFS -> Configuration -> DataNode Block Count Thresholds
b. CM -> HDFS -> WebUI -> Namenode Web UI -> Click on datanode menu -> Get the block count of your node

 

if b > a then you will get block count warning

 

also cloudera advice says "presence of many small files" also create this warning

 

action: 
1. if it is not disturbing anything then you can ignore this warning but just keep an on eye on block pool usage percentage from 'b'

2. you can increase block count thresholds in 'a'
3. you can cleanup unwanted data, but if your trash folder maintains old data (for ex: 24 hrs) then you will see the result after 24 hours
4. add additional data nodes and apply rebalance
etc

 

 

avatar
Rising Star

@saranvisa,

 

 b>a for all the datanodes.

 

I have cleaned up the unwanted data and in command line block count is showing below threshold value however it's not reflected in the cloudera manager console.

 

Please suggest.

 

Thanks,

 

avatar
Champion

@cdhhadoop

 

as mentioned, you will get warning if b > a 

avatar
Rising Star
@saranvisa,

It was b>a earlier.

I removed files using command line and block count is below threshold value. However it's not reflected in cloudera manager console.

Please suggest.

Thanks,

avatar
Mentor
What command are you using to check the block replica count of each DataNode? That information is present in either the DN metrics (SELECT blocks_total WHERE roleType = DATANODE) or in the NameNode Web UI as a column under its live DataNodes page. The block replica count is not shown as part of 'hdfs dfsadmin -report' and if you are relying on FSCK then ensure you're not counting just block IDs but actual replicas instead.

Do the information in these sources still indicate that each of the DataNodes have way lesser replicas than its alert threshold?

avatar
Rising Star
@Harsh J
Thanks for the reply.
I am checking the block count by using the command hdfs fsck path and referring to total blocks column in the output.
Please suggest whether this is the right way or not.

Thanks,
Priya

avatar
Mentor
That's not enitirely right - The block count warning per DataNode stems
from the fact that the DataNode's carrying too many replicas (of blocks).
The FSCK total block count is a number that does not include the replica
multiplier (x3, typically) and is global (across all DNs).

Have you looked at the Live DataNodes page on your NameNode UI as indicated
in the earlier post? It should have the reported replica count of every DN
alive in the cluster.

avatar
Rising Star
@Harsh J,
Thanks for quick reply.
I thought the ouptut of fsck command includes replica multiplier and gives final total block count. Thanks for the clarification.
I checked Datanodes page on namenode WebUI and block count for each datanode is more than threshold value.

Thanks,
Priya