Skip to main content
Amn_468
Amn_468
Silver ⭐️⭐️⭐️
October 27, 2020
Solved

Data Node Pause Duration

Hello,


On our data node, we are increasing getting alerts related to Data Node Pause Duration. So far, this is happening on a single data node out of nine data nodes.

Following is the error captured from DN logs 

2020-10-27 16:20:05,140 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1821ms GC pool 'ParNew' had collection(s): count=1 time=2075ms)

 

Current Java Heap Size of Data Node in Bytes is at 6GB

 

CM / CDH – 5.16.x

 

Any help is appreciated.

 

Regards

Amn

PabitraDas
PabitraDasAnswer
Gold ⭐️
November 11, 2020

Hello @Amn_468 

Since you reported the DN Pause time, I spoke/referred about DN heap only.  The block counts on most of the DN seems >6Millions, hence would suggest to increase the DN heap to 8GB (from current value of 6GB) and perorm a rolling restart to bring the new heap size into effect. 

 

There is no straight forward way to say you hit the small file problem but if your average block size is few MB or less than a MB in size, it is an indication that you are storing/accumulating small files in HDFS.  Simplest way to determine small files in cluster is to run fsck.

 

Fsck should show the average block size. If it's too low a value (eg ~ 1MB ), you might be hitting the problems of small files which would be worth looking at, otherwise, there is no need to review the number of blocks.

 

[..]

$ hdfs fsck /

..

...

Total blocks (validated): 2899 (avg. block size 11475601 B) <<<<<

[..]

 

You may refer belwo links for your help on dealing with small files.

- https://blog.cloudera.com/small-files-big-foils-addressing-the-associated-metadata-and-application-challenges/

- https://community.cloudera.com/t5/Community-Articles/Identify-where-most-of-the-small-file-are-located-in-a-large/ta-p/247253 

View in thread

9 replies

GangWar
Platinum ⭐️⭐️
GangWar
Platinum ⭐️⭐️
October 27, 2020

@Amn_468 This is due to the Java Heap Size. 

Let's say the default setting for the namenode_java_heapsize is 1GB. Cloudera recommends having 1GB of heap space for every 1M blocks in a cluster.
 

If the data in your cluster is growing rapidly, factor in the potential future number of blocks your cluster will require when determining the size setting, so you can avoid having to restart the namenode.  It is only possible to change the setting by restarting the namenode.

Calculating the Required Heap Size

  1. Determine the number of blocks in the cluster. This information is available on the namenode web UI under the Summary section, with information like the following:

    117,387 files and directories, 56,875 blocks = 174,262 total filesystem object(s).

    Alternatively, the information is available from the output of the fsck command:

    • Total size: 9958827546 B (Total open files size: 93 B)
      Total dirs: 20397
      Total files: 57993
      Total symlinks: 0 (Files currently being written: 1)
      Total blocks (validated): 56874 (avg. block size 175103 B) (Total open file blocks (not validated): 1)
      ...

Given the number of blocks, allocate 1GB of heap space for each 1M blocks, plus some additional memory for growth. For example, if there are 6,543,567 blocks, you need 6.5GB of heap to cover the current cluster size, but 8GB would be a sensible setting to allow for growth of the cluster.
 
After that you can adjust the Java Heap Size for NN. Hope this helps. 
Translate
Cheers! Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
Shelton
Shelton
Gold ⭐️⭐️⭐️
October 27, 2020

@Amn_468 

Increasing the Java Heap Size for the NameNode and Secondary NameNode Services,you could be using the default 1GB setting for heap size

As a general rule of thumb take a look at the configuration of your Heap Sizes for every 1 Million Blocks in your cluster should have at least 1GB of Heap Size.

  • 2 Million Blocks 2GB heap size
  • 3 Million Blocks 3GB heap size
    .....
  • n Million Blocks n GB heap size

After increasing the Java Heap Size and restart the HDFS Services that should resolve the issue.

 

Please revert

Translate
Amn_468
Amn_468Author
Silver ⭐️⭐️⭐️
October 27, 2020

Hello @GangWar @Shelton 

 

Appericate your assistance,

Following is the information available from NN WebUI- (23,326,719 files and directories, 22,735,340 blocks = 46,062,059 total filesystem object(s).

Heap Memory used 5.47 GB of 10.6 GB Heap Memory. Max Heap Memory is 10.6 GB.

Non Heap Memory used 120.51 MB of 122.7 MB Commited Non Heap Memory. Max Non Heap Memory is <unbounded>.)

 

Could you please re-confirm whether I need to adjust the NN Heap Memory OR DN heap memory, as the issue is seen on data Node and that too only one data node other 8 seem to be running without any issues.

 

Thanks 

Amn

Translate
Shelton
Shelton
Gold ⭐️⭐️⭐️
October 27, 2020

@Amn_468 

 

The NameNode is solely responsible for the Cluster Metadata so please increase the NN heap size and restart the services.
Please revert 

 

Translate
PabitraDas
Gold ⭐️
PabitraDas
Gold ⭐️
November 9, 2020

Hello @Amn_468 The DN Pause alert you see for 1/9 DataNodes are indication of growing blocks on it.

Compared to other DNs, possibly this DN in question have stored more number of blocks than other nodes. You may compare the block counts of each DN in HDFS > HDFS > WebUI > Active NN Web UI > DataNodes > Cehck the blocks column under section "In Operation".

 

The log snippet you shared indicates a pause of 2sec only, which is not sign of worry. However, with proper JVM heap size allocated for DN, you may avoid these frequent pause alerts.

 

As a thumb rule you may need 1GB heap for 1Million blocks and since you have 6GB allocated for DN heap, please verify the block counts on the DNs and ensure they are not too high (> 6Millions) in count which may explain why there are so many pause alerts. 

 

In case the block count is too high than expected, it means you need to increase the heap size to accomodate the block objects in JVM heap memeory.  

 

On a. side note, growing block counts also an early warning/indication of small files problem in cluster. You need to be vigilant about that. Verify the average block size and that would help you to understand, if you are having small files problem in your cluster. 

 

Regards,

Pabitra Das

 

 

 

Translate
Amn_468
Amn_468Author
Silver ⭐️⭐️⭐️
November 11, 2020

Hello @PabitraDas,

Appreciate your assistance, below is the block count on our DNs, as mentioned earlier we have allocated 6 GB JVM Heap for DN's and 10 GB Heap for NN & SNN.

Do you suggest to increase DN Heap, or NN / SNN Heap as suggested by Shelton.

Block Count:

Node 1 = 7421379 
Node 2 = 5569699
Node 3 = 6003009
Node 4 = 7444205
Node 5 = 8770674
Node 6 = 8849641
Node 7 = 8232779
Node 8 = 8354714
Node 9 = 8860602

Also, would greatly appreciate if you have any pointers / suggestions (scripts etc. ) to identify small file issue and possible remediation.

 

Thanks 

Amn

Translate
PabitraDas
Gold ⭐️
PabitraDasAnswer
Gold ⭐️
November 11, 2020

Hello @Amn_468 

Since you reported the DN Pause time, I spoke/referred about DN heap only.  The block counts on most of the DN seems >6Millions, hence would suggest to increase the DN heap to 8GB (from current value of 6GB) and perorm a rolling restart to bring the new heap size into effect. 

 

There is no straight forward way to say you hit the small file problem but if your average block size is few MB or less than a MB in size, it is an indication that you are storing/accumulating small files in HDFS.  Simplest way to determine small files in cluster is to run fsck.

 

Fsck should show the average block size. If it's too low a value (eg ~ 1MB ), you might be hitting the problems of small files which would be worth looking at, otherwise, there is no need to review the number of blocks.

 

[..]

$ hdfs fsck /

..

...

Total blocks (validated): 2899 (avg. block size 11475601 B) <<<<<

[..]

 

You may refer belwo links for your help on dealing with small files.

- https://blog.cloudera.com/small-files-big-foils-addressing-the-associated-metadata-and-application-challenges/

- https://community.cloudera.com/t5/Community-Articles/Identify-where-most-of-the-small-file-are-located-in-a-large/ta-p/247253 

Translate
Terms & ConditionsAccessibility statement