Reply
Highlighted
Explorer
Posts: 64
Registered: ‎08-07-2017
Accepted Solution

block count warning still shows in cloudera manager even after deleting files from command line

Hello All,

 

We have CDH5.9.2 hadoop 13-node  cluster and from one week, I am exeperiencing block count concerning warning notification in cloudera manager for all 6 datanodes.

 

We have multitenancy implemented in our cluster. So I deleted the older files via hdfs command line and block count is below the threshold value.

 

But still the block count concerning warning appears in the cloudera manager.

 

Please suggest.

 

Thanks,

Priya

Posts: 394
Topics: 11
Kudos: 60
Solutions: 35
Registered: ‎09-02-2016

Re: block count warning still shows in cloudera manager even after deleting files from command line

@cdhhadoop

 

Get the value of
a. CM -> HDFS -> Configuration -> DataNode Block Count Thresholds
b. CM -> HDFS -> WebUI -> Namenode Web UI -> Click on datanode menu -> Get the block count of your node

 

if b > a then you will get block count warning

 

also cloudera advice says "presence of many small files" also create this warning

 

action: 
1. if it is not disturbing anything then you can ignore this warning but just keep an on eye on block pool usage percentage from 'b'

2. you can increase block count thresholds in 'a'
3. you can cleanup unwanted data, but if your trash folder maintains old data (for ex: 24 hrs) then you will see the result after 24 hours
4. add additional data nodes and apply rebalance
etc

 

 

Explorer
Posts: 64
Registered: ‎08-07-2017

Re: block count warning still shows in cloudera manager even after deleting files from command line

@saranvisa,

 

 b>a for all the datanodes.

 

I have cleaned up the unwanted data and in command line block count is showing below threshold value however it's not reflected in the cloudera manager console.

 

Please suggest.

 

Thanks,

 

Posts: 394
Topics: 11
Kudos: 60
Solutions: 35
Registered: ‎09-02-2016

Re: block count warning still shows in cloudera manager even after deleting files from command line

@cdhhadoop

 

as mentioned, you will get warning if b > a 

Explorer
Posts: 64
Registered: ‎08-07-2017

Re: block count warning still shows in cloudera manager even after deleting files from command line

@saranvisa,

It was b>a earlier.

I removed files using command line and block count is below threshold value. However it's not reflected in cloudera manager console.

Please suggest.

Thanks,
Posts: 1,567
Kudos: 289
Solutions: 240
Registered: ‎07-31-2013

Re: block count warning still shows in cloudera manager even after deleting files from command line

What command are you using to check the block replica count of each DataNode? That information is present in either the DN metrics (SELECT blocks_total WHERE roleType = DATANODE) or in the NameNode Web UI as a column under its live DataNodes page. The block replica count is not shown as part of 'hdfs dfsadmin -report' and if you are relying on FSCK then ensure you're not counting just block IDs but actual replicas instead.

Do the information in these sources still indicate that each of the DataNodes have way lesser replicas than its alert threshold?
Backline Customer Operations Engineer
Explorer
Posts: 64
Registered: ‎08-07-2017

Re: block count warning still shows in cloudera manager even after deleting files from command line

@Harsh J
Thanks for the reply.
I am checking the block count by using the command hdfs fsck path and referring to total blocks column in the output.
Please suggest whether this is the right way or not.

Thanks,
Priya
Posts: 1,567
Kudos: 289
Solutions: 240
Registered: ‎07-31-2013

Re: block count warning still shows in cloudera manager even after deleting files from command line

That's not enitirely right - The block count warning per DataNode stems
from the fact that the DataNode's carrying too many replicas (of blocks).
The FSCK total block count is a number that does not include the replica
multiplier (x3, typically) and is global (across all DNs).

Have you looked at the Live DataNodes page on your NameNode UI as indicated
in the earlier post? It should have the reported replica count of every DN
alive in the cluster.
Backline Customer Operations Engineer
Explorer
Posts: 64
Registered: ‎08-07-2017

Re: block count warning still shows in cloudera manager even after deleting files from command line

@Harsh J,
Thanks for quick reply.
I thought the ouptut of fsck command includes replica multiplier and gives final total block count. Thanks for the clarification.
I checked Datanodes page on namenode WebUI and block count for each datanode is more than threshold value.

Thanks,
Priya
Announcements