Reply
New Contributor
Posts: 1
Registered: ‎02-24-2016

Datanodes report block count more than the threshold

Hi ,

 

In on of our cluster , datanodes report block counts more than the threshold. We have cleared large number of small files from /hbase/oldWALS , but still the block counts hasn't reduced.

 

Also we are seeing lot of small files in the /tmp/logs/ folder , could you please advice if it is safe to delete these files from this folder and will the issue get resolved ?

 

Thanks,

Sumanth 

Posts: 215
Topics: 0
Kudos: 24
Solutions: 17
Registered: ‎08-16-2016

Re: Datanodes report block count more than the threshold

The logs in /tmp/logs are the application logs if aggregation is turned on. These can be removed depending on your requirement to retain for debugging/monitoring. It is best to allow the cluster to handle removal. Set the yarn.log-aggregation.retain-seconds to the number of seconds you need to keep on hand or can keep on hand.

Use the hdfs fsck / (any path) to view the number of files, directories, and blocks. This should help you hunt down were they are the large number of blocks are coming from.

I'll add that the threshold isn't terrible serious. It generally means that you may need to increase the Datanode heap size. I typically end up increase the threshold once I have boosted the heap.

With that said, small files can be an issue for the Namenode, so keep it in check.
Announcements