Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Datanodes report block count more than threshold on datanode and Namenode

Re: Datanodes report block count more than threshold on datanode and Namenode

Expert Contributor
Ok. Thanks.

Re: Datanodes report block count more than threshold on datanode and Namenode

Expert Contributor

What all I did:

 

1. Increased the memory of NN

2. Increased he disk of overall cluster

3. Increased dfs blocksize from 64MB to 128MB

4. Increased the block count threshold.

Re: Datanodes report block count more than threshold on datanode and Namenode

Champion

if you have Cloudera manager , you could easily find the problem as to which job is creating lot of stress on the storage . Please take a peek in to the below link 

 

https://www.cloudera.com/documentation/enterprise/latest/topics/cm_dg_disk_usage_reports.html

 

https://www.cloudera.com/documentation/enterprise/latest/topics/admin_directory_usage.html#concept_l...

Re: Datanodes report block count more than threshold on datanode and Namenode

New Contributor

Hi All,

 

I recoment to check which application team is causing it by using #hdfs dfs -count -v -h /project/*
If the FILE_COUNT is more than 10M, then its problem for mid size of cluster.

 

Please check the below link to reduce the block count.

 

https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ht_datanode.html#concept_uet_9pn_...

 

Reg,

Sandeep Kolli

 

Highlighted

Re: Datanodes report block count more than threshold on datanode and Namenode

New Contributor

To add,

 

For sizing a datanode heap it's similar to namenode heap, its recommend 1GB per 1M blocks. As a block could be as small a 1byte or as large as 128MB, the requirement of heap space is the same.