Reply
Expert Contributor
Posts: 338
Registered: ‎01-25-2017

Re: Datanodes report block count more than threshold on datanode and Namenode

all looks fine,

you can even reduce

yarn.nodemanager.resource.memory-mb
Expert Contributor
Posts: 63
Registered: ‎11-17-2016

Re: Datanodes report block count more than threshold on datanode and Namenode

Ok. Thanks.
Expert Contributor
Posts: 63
Registered: ‎11-17-2016

Re: Datanodes report block count more than threshold on datanode and Namenode

What all I did:

 

1. Increased the memory of NN

2. Increased he disk of overall cluster

3. Increased dfs blocksize from 64MB to 128MB

4. Increased the block count threshold.

New Contributor
Posts: 3
Registered: ‎06-29-2018

Re: Datanodes report block count more than threshold on datanode and Namenode

Hi All,

 

I recoment to check which application team is causing it by using #hdfs dfs -count -v -h /project/*
If the FILE_COUNT is more than 10M, then its problem for mid size of cluster.

 

Please check the below link to reduce the block count.

 

https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ht_datanode.html#concept_uet_9pn_...

 

Reg,

Sandeep Kolli

 

Highlighted
New Contributor
Posts: 3
Registered: ‎06-29-2018

Re: Datanodes report block count more than threshold on datanode and Namenode

To add,

 

For sizing a datanode heap it's similar to namenode heap, its recommend 1GB per 1M blocks. As a block could be as small a 1byte or as large as 128MB, the requirement of heap space is the same.

Announcements